Hubi@feddit.de to Reddit@lemmy.worldEnglish · 3 个月前Reddit shares plunge almost 25% in two days, finish the week below first day closewww.cnbc.comexternal-linkmessage-square77fedilinkarrow-up1639
arrow-up1625external-linkReddit shares plunge almost 25% in two days, finish the week below first day closewww.cnbc.comHubi@feddit.de to Reddit@lemmy.worldEnglish · 3 个月前message-square77fedilink
minus-squareSacrificedBeans@lemmy.worldlinkfedilinkarrow-up11·3 个月前Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.
Robots.txt is a compliance standard afaik, more like an advisory guide. It cannot enforce bans on scrapers, it sets some polite boundaries for the website.