RavenBot
How Raven reads public pages.
RavenBot reads public pages for people using Raven: a preview of a link or a find they're looking at, and pages they asked Raven to keep an eye on, to tell them when something changes. It also reads public posts where people describe their own experience with products, places and tools, so Raven can cite the person who wrote it.
How it identifies itself
RavenBot/1.0 (+https://www.tryraven.io/bot/)What it does
- Reads public RSS and Atom feeds and public pages. No logins, paywalls or private content.
- Follows robots.txt for every page it reads. If your robots.txt blocks major AI crawlers such as GPTBot, ClaudeBot, Google-Extended or PerplexityBot, RavenBot treats that as a block too, even if it isn't named.
- Goes slowly: one request at a time per site, with a pause between requests.
- Re-reads a page someone asked Raven to watch at most once a day. It keeps a fingerprint of each section and a little of its text with that person's thread, to tell what changed, and never republishes the page.
- For experiences, stores a short excerpt (at most 900 characters), the author's name as your site shows it, and a link to the original. Never the full text.
- Doesn't use your content to train AI models. Excerpts are only used to find and cite experiences in People search.
How to block it
Add this to your site's robots.txt:
User-agent: RavenBot
Disallow: /RavenBot checks robots.txt before reading any page, including every page a link redirects to, and reads it again at least every hour, so a block applies within the hour.
Removing your content
To have excerpts from your site or writing removed from Raven, email hello@useharbor.io with your site or profile address. We remove them and stop collecting from you.