GuthBot

What this bot does

GuthBot is Guth Labs’ direct-operated collection bot. The current collector checks a curated list of public documentation and release sources, retains source references and text snapshots, and prepares Markdown for private editorial review and research. This registration covers reference and retrieval use; it does not describe a model-training crawler.

The scheduler checks for due sources every five minutes, throughout the day. Healthy sources refresh about every 30 minutes; failures and server retry instructions defer collection. An authorized operator can also request a due-source run. Collection is sequential, bounded, and limited to configured sources; it is not an unrestricted web crawl.

Recognize GuthBot

HTTP user agent:

GuthBot/1.0 (+https://guth.news/bot)

GuthBot uses Ed25519 HTTP message signatures. Its public keys are available in the signed key directory. The signature agent is https://guth.news.

A user-agent string alone is not proof of identity. Publishing signing keys is not a claim of Cloudflare verification or approval.

Website owner controls

To disallow GuthBot collection, add this group to your site’s robots.txt:

User-agent: GuthBot
Disallow: /

The collector checks robots.txt before fetching a source. Unavailable or redirected robots files defer collection; a missing file (404) permits the next check. It also honors:

• Content signals search=no, ai-input=no and noai; ai-train=no is recorded and excludes the page from any training use.
• noindex, none, noai and nofollow in <meta name="robots">, <meta name="GuthBot"> or X-Robots-Tag: nothing from a noindex page is retained or indexed.
• TDM reservations (TDMRep): the tdm-reservation header or meta tag and /.well-known/tdmrep.json.
• AI-crawler blocks: if your robots.txt disallows a page to AI crawlers such as GPTBot, ClaudeBot, Google-Extended or CCBot, GuthBot treats it as an AI opt-out and does not retain or index the page.
• HTTP 402 (pay per crawl): the page is skipped and not retried soon; GuthBot does not pay for access.
• Crawl-delay: requests to one site are spaced by at least the declared delay.

Cross-origin redirects are refused. Pages on substack.com are never collected.

GuthBot does not log in to source websites or bypass access challenges. Changing crawl rules controls future collection; contact us about previously retained material.

Contact Guth Labs

For crawler questions, opt-out requests, or removal requests, email dev@getguth.com. Include the affected URL and, where available, the request timestamp.