Hey everyone,

I assuming if you are reading this, you also noticed that the instance has been down for a few hours. We have been struggling with some sort of traffic issue from our mlmym self-hosted instance at https://old.literature.com/. It had been flooding the instance with so many requests that it could not keep up. I went ahead and shut it down and we should be good to go for now. I’m not totally sure what that issue was but if I figure it out I will throw it in the comments.

I also learned today that mlmym has been abandoned by the developer. :(

Anyway hope everything is smooth sailing from here. I will post a more detailed “state of the server” later this week.

Thanks all!

EDIT:

I deployed Anubis in front of mlmym so https://old.literature.com/ is back up for anyone that uses that front end (me included). If this is successful, I mostly likely will deploy Anubis against the main frontend as well. Stay tuned for any announcements about that.

  • flamingos-cant (hopepunk arc)@feddit.uk
    link
    fedilink
    English
    arrow-up
    9
    ·
    edit-2
    12 days ago

    The issue with mlmym is that it serves static HTML which AI scrapers love, so they just hit it until your rate limits are exhausted. The .world people have actually forked it. Ignore the readme though, you should run the image ghcr.io/fedihosting-foundation-forks/mlmym:main and to get around bots degrading your server, you can do what I did with old.feddit.uk and redirect requests to the login page unless they’re logged in:

    Conf snippet
        if ($http_user_agent ~ Chrome) {
            set $old_chrome N;
        }
    
        # block request if user agent indicates it's less than Chrome 120
        if ($http_user_agent !~ "Chrome\/1[234]") {
            set $old_chrome "${old_chrome}O";
        }
    
        if ($old_chrome = NO) {
           return 444;
        }
    
        # block users that aren't logged in from mymlm
        set $logged_in N;
    
        if ($http_cookie) {
           set $logged_in '';
        }
    
        if ($http_accept !~ 'text\/css') {
    	set $logged_in "${logged_in}O";
        }
        
        location / {
    	    # limit_req zone=old burst=5 nodelay;
    	    if ($logged_in = NO) {
    	       return 302 /login;
    	    }
    
    
    	    proxy_pass http://localhost:8280/;
    
    	    proxy_set_header HOST $host;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    
        # Redirect bots away from Mlmym as it saturates the RateLimits too easily
        location /robots.txt {
            return 301 https://feddit.uk/robots.txt;
        }
    
        location /login {
        	proxy_pass http://localhost:8280/;
    
    	proxy_set_header HOST $host;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;   
        }
    
    • Arthur@literature.cafeOPM
      link
      fedilink
      arrow-up
      5
      ·
      edit-2
      12 days ago

      What a wonderful world that has been created for us. Thanks for the help I’ll get this deployed! I’ll probably just put anubis in front of mlmym though I think.