this post was submitted on 19 Aug 2025
866 points (99.3% liked)

Technology

76337 readers
1541 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots

founded 2 years ago
MODERATORS
L3s@lemmy.world
enu@lemmy.world
technopagan@lemmy.world
L4s@lemmy.world
L3s@hackingne.ws
L4s@hackingne.ws
866
The AI company Perplexity is complaining their bots can't bypass Cloudflare's firewall (www.searchenginejournal.com)
submitted 2 months ago* (last edited 2 months ago) by Davriellelouna@lemmy.world to c/technology@lemmy.world
248 comments fedilink hide all child comments
(page 4) 50 comments
sorted by: hot top controversial new old
[–] Electricd@lemmybefree.net 2 points 2 months ago* (last edited 2 months ago) (1 children)

They do have a point though. It would be great to let per-prompt searches go through, but not mass scrapping

I believe a lot of websites don't want both though

[–] threeganzi@sh.itjust.works 2 points 2 months ago (1 children)

Does it not need to be scraped to be indexed, assuming it’s semi-typical RAG stuff?

[–] Electricd@lemmybefree.net 1 points 2 months ago

I assume their script does some search engine stuff like query google or bing and then "scrap" the links they go on

Some selenium stuff

[–] tarknassus@lemmy.world 2 points 2 months ago

I don't see a problem here. Maybe Perplexity should consider the reasons WHY Cloudflare have a firewall...?

[–] josefo@leminal.space 1 points 2 months ago

I really hope Cloudflare doesn't eventually evolve into a shitty ass company, so far I like them very much, and all this massive L for AI only improves my opinion on them.

load more comments
view more: ‹ prev next ›