What is it about?
The Robots Exclusion Protocol is used across the internet by website owners to outline privacy directives to automated crawlers, but is not a method of enforcement. This paper explores how effective the protocol is in preventing unwanted web traffic towards websites, and whether there are directives in the protocol that are more likely to be complied with.
Featured Image
Photo by julien Tromeur on Unsplash
Why is it important?
Our findings show that stricter directives in the robots.txt directives of the Robots Exclusion Protocol are less likely to be followed. Consequently, more enforceable alternatives to the protocol are necessary to protect user data in websites across the web.
Read the Original
This page is a summary of: Scrapers Selectively Respect robots.txt Directives: Evidence From a Large-Scale Empirical Study, October 2025, ACM (Association for Computing Machinery),
DOI: 10.1145/3730567.3764471.
You can read the full text:
Contributors
The following have contributed to this page
