Lemmy.one
  • Communities
  • Create Post
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
Lemmit.Online bot@lemmit.onlineMB to /r/Technology@lemmit.onlineEnglish · 25 days ago

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

blog.cloudflare.com

external-link
message-square
0
fedilink
  • cross-posted to:
  • cybersecurity@sh.itjust.works
  • technology@lemmy.zip
  • hackernews@lemmy.bestiver.se
1
external-link

Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives

blog.cloudflare.com

Lemmit.Online bot@lemmit.onlineMB to /r/Technology@lemmit.onlineEnglish · 25 days ago
message-square
0
fedilink
  • cross-posted to:
  • cybersecurity@sh.itjust.works
  • technology@lemmy.zip
  • hackernews@lemmy.bestiver.se
Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites.
This is an automated archive made by the Lemmit Bot.

The original was posted on /r/technology by /u/memloh on 2025-08-04 14:07:42+00:00.

alert-triangle
You must log in or # to comment.

/r/Technology@lemmit.online

technology@lemmit.online

Subscribe from Remote Instance

You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !technology@lemmit.online
lock
Community locked: only moderators can create posts. You can still comment on posts.

Subreddit dedicated to the news and discussions about the creation and use of technology and its surrounding issues.

Visibility: Public
globe

This community can be federated to other instances and be posted/commented in by their users.

  • 1 user / day
  • 15 users / week
  • 45 users / month
  • 154 users / 6 months
  • 5 local subscribers
  • 198 subscribers
  • 20.3K Posts
  • 169 Comments
  • Modlog
  • mods:
  • Lemmit.Online bot@lemmit.online
  • BE: 0.19.7
  • Modlog
  • Legal
  • Instances
  • Docs
  • Code
  • join-lemmy.org