Crawler information
Our crawler reads the public pages of school and academy trust websites in England to check what they publish online. This page explains what it does and how to block it.
How to recognise it
It identifies itself with this user agent:
website-compliance-audit/0.2.0 (UK schools website audit; public pages only; robots.txt honoured; 1 req/s; https://evidentschools.com/crawler/)The version number changes when we update the crawler. Some pages are loaded in a standard headless browser so that content built by scripts can be read. The headless browser sends the same user agent.
What it does
- Reads public pages and the documents they link to, such as PDFs.
- Honours robots.txt in full, including rules for document paths and any Crawl-delay.
- Makes no more than one request a second to a site, or fewer if your Crawl-delay asks for it.
- Stops at a fixed page limit for each site.
What it never does
- Logs in, or tries to reach anything behind a login.
- Fills in or submits forms.
- Accepts cookie banners.
- Gets round bot protection or CAPTCHAs.
- Opens documents your robots.txt disallows.
How to block it
Add this to your robots.txt:
User-agent: website-compliance-audit
Disallow: /Or email hello@evidentschools.com with your website address and we'll stop checking it.