this post was submitted on 21 Mar 2025
1264 points (99.5% liked)
Technology
67151 readers
4132 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
It's worked alright for me. Your mileage may vary.
If someone is scraping my site at a low crawl rate I honestly don't care so long as it doesn't impact my performance for everyone else. If I hosted anything that was not just public knowledge or copy regurgitated verbatim from the bumf provided by the vendors of the brands I sell, I might oppose to it ideologically. But I don't. So I don't.
If parallel crawling from multiple organizations legitimately becomes a concern for us I will have to get more creative. But thus far it hasn't, and honestly just wholesale blocking Amazon from our shit instantly solved 90% of the problem.