About our nightly re-check
You are probably here because requests from our server appeared in your logs. This page says exactly what they are.
What it does
Every fee, deadline, exemption and penalty CharityFile publishes cites a public statute, regulation or agency page, and keeps a verbatim quote from it. Once a night our production server fetches each cited page and checks that the quote is still there. Nothing behind a login or a paywall is requested.
How it behaves
- One GET per distinct cited URL per run, from one server. Rows that share a URL reuse that response. A host listed in ops/check-quotes-skip-hosts.txt is not fetched.
- Plain page fetches. The requests identify as a desktop browser and do not carry a bot token; they do not send conditional headers and do not read robots.txt. We say so here because a page for webmasters should not claim more than the script does.
- Nothing is stored except the check result. A missing quote flags the row for a person to re-verify; the page itself is not kept.
How to ask us to stop
Email support@charityfile.com with the host name and a log excerpt. We add that host to ops/check-quotes-skip-hosts.txt and re-verify its rows by hand instead.