Fuck AI

I highly recommend using Anubis as a proxy for your entire instance. It's a little complicated to get going, but it stops any and all AI scrapers with a denial of access. Having a robots.txt works, but only so much, because some of these bots do not respect it. And, honestly, with the way Sam Altman talks about the people he's stolen and scraped from, I don't think anyone should be surprised.

But, I have Anubis running on my personal website and I've tested to see if ChatGPT can see it, and it cannot. Good enough for me

Patience, AI crash bubble burst will be soon.

So I just had a look at your robots.txt:

User-Agent: *
  Disallow: /login
  Disallow: /login_reset
  Disallow: /settings
  Disallow: /create_community
  Disallow: /create_post
  Disallow: /create_private_message
  Disallow: /inbox
  Disallow: /setup
  Disallow: /admin
  Disallow: /password_change
  Disallow: /search/
  Disallow: /modlog
  Crawl-delay: 60

You explicitly allow searching your content by bots... That's likely one of the reasons why you get bot traffic.

Just cache. Read only traffic should add negligible load to your server. Or you're doing something horribly wrong

You can enable Private Instance in your admin settings, this will mean only logged in users can see content. This will prevent AI scrapers from slowing down your instance as all they'll see is an empty homepage, so no DB calls. As long as you're on 0.19.11, federation will still work.

At some point they're going to try to evade detection to continue scraping the web. The cat and mouse game continues except now the "pirates" are big tech.

Use Anubis. That's pretty much the only thing you can do against bots that they have no way of circumventing.

You can either use Cloudflare(proprietary) or anubis (Foss)

Cloudflare has pretty good protection against this, but I totally understand not wanting to use Cloudflare

Haha, just wait when you get ddosed by anonymous user agents. I have been there.

I'm talking 40k requests per 5 seconds.