CourtSentencesBot

Last Updated: 26 October 2026

If you have seen CourtSentencesBot in your server logs, this page explains what it does, how it behaves, and how to block it or ask us to remove content.

What it does

CourtSentencesBot reads publicly available reports of sentencings in UK courts and records the outcome: who was sentenced, for what offence, the sentence, the court and the date. These records make up a public record of sentences on courtsentences.co.uk, organised by court.

Every record names the publication it came from, and links to the original report from the record's details, so readers can go to the source.

What it never does

• Log in, or use subscriber or member accounts.
• Get past bot protection, CAPTCHAs or login pages. A bot challenge is treated as a refusal, not as an obstacle to work around.
• Use content a publisher has marked as subscriber-only, even when the full text is present in the page source. It recognises this through the markup publishers already provide for search engines: isAccessibleForFree set to false in schema.org data, or a content_tier meta tag of locked, premium or subscriber.
• Disguise itself. Every request it makes, including when it loads a page in a browser, carries the user-agent above.

How it behaves

• It checks robots.txt before every page it fetches, following the robots.txt standard (RFC 9309). If your robots.txt cannot be read because your server returns an error or does not respond, it assumes it may not crawl your site and checks again later.
• It caches each site's robots.txt for up to 12 hours, so changes you make take effect within that time.
• It fetches no more than one page every 0.5 seconds from any one site. If your robots.txt sets a Crawl-delay, it waits at least that long between pages, up to a maximum of two minutes.
• It treats a 401, 403 or 429 response, or a bot challenge, as a refusal of that page, and does not try to get round it. Refusals are handled page by page, so other pages on your site may still be requested, and a refused page may be checked again after 30 days in case your policy has changed. To stop all requests, use robots.txt as shown below, which it always checks first.
• Some pages only show their text once JavaScript has run. For those pages, and only where robots.txt allows the page and your server has already served it normally, it loads the page once more in a browser that also identifies as CourtSentencesBot. As with any browser, loading the page this way also loads the scripts, styles and images the page itself asks for.

How to block it

To stop CourtSentencesBot crawling your whole site, add this to your robots.txt:

User-agent: CourtSentencesBot
Disallow: /

To block only part of your site, name the path instead. For example:

User-agent: CourtSentencesBot
Disallow: /news/courts/

Changes take effect within 12 hours. Blocking stops future crawling, but does not remove records we already hold. To have those removed, contact us as described below.

Removing content

If you are a publisher and would like records based on your reporting removed, or would like us to stop collecting from your publication altogether, email [email protected]. When we stop collecting from a publication, we also remove the records we already hold from it.

If you are named in a record and want it corrected or removed, use the flag icon on the record itself, which opens our removal request form, or contact us. Every request is reviewed by a person. Our Editorial and Safeguards Policy explains how requests are handled and what we never publish.