![](/static/253f0d9b/assets/icons/icon-96x96.png)
![](https://lemmy.world/pictrs/image/ea4c7d39-bb1c-4f59-b46b-c795c3ee0536.jpeg)
11·
3 months agoRobots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
Nor evidence, for that matter