Quick answer
What is the LLMWebSearch bot and how do I block it?
The LLMWebSearch page reader fetches public pages that appear in a customer's web search results, so an AI agent can read them. It sends the user agent LLMWebSearch/1.0 (+https://llmwebsearch.com/bot/) and obeys robots.txt rules for LLMWebSearch or for all robots. Two lines in robots.txt block it. It reads at most 10 pages per search, and never works around a refusal.
The page reader at a glance
| User-Agent header | LLMWebSearch/1.0 (+https://llmwebsearch.com/bot/) |
|---|---|
| robots.txt token | LLMWebSearch |
| When it visits | Only when your page is among the results a customer asked it to read, or a customer names your URL |
| Pages per search | Up to 10, usually 3 |
| robots.txt | Read before any page; rules cached for 24 hours at most |
| Content-Signal | ai-input=no keeps your content out of responses |
| Refusals | Accepted as final: no retry under another name, no CAPTCHA solving, no sign-in |
| Contact | [email protected] |
Why is it visiting my site?
Because a customer's search returned one of your pages near the top, and their agent asked for the text of the first results. Less often, a customer sent your URL to our /extract endpoint because they already had the link.
The reader does not crawl. It does not follow your links, walk your sitemap or build an index. Each visit is one page, requested because a person or an agent wanted to read it at that moment.
How do I block it completely?
Add a group for its token to your robots.txt. The rule takes effect at our next read of the file, within 24 hours. Pages it may not read still appear in search results with their title, URL and snippet, but without their content.
User-agent: LLMWebSearch
Disallow: /
Can I block only some sections?
Yes. List the paths you want kept out; everything else stays readable. The reader picks the group written for LLMWebSearch if there is one, and the * group otherwise, like most well-behaved robots.
User-agent: LLMWebSearch
Disallow: /account/
Disallow: /checkout/
Allow: /
Can I allow search but refuse AI use?
Yes, with a Content Signal. A site that sets Content-Signal: ai-input=no in its robots.txt keeps its pages out of your results' content. Your pages can still be listed as results, so people find them, but their text is not handed to an AI agent through our API.
User-agent: *
Content-Signal: search=yes, ai-input=no
Allow: /
What does the reader never do?
When a site turns our reader away, the result comes back without content and with the site's status: we never solve CAPTCHAs, sign in to accounts or work around a block or a paywall. It also does not pretend to be a regular browser, does not switch to another user agent after a refusal, and does not send cookies or credentials.
Some pages only show their text after JavaScript runs. For those, the reader can load the page in a headless browser, still under the same user agent and the same robots.txt rules. A 403, a 429 or a challenge page ends the attempt; the customer receives your answer, not your content.
How can I recognise its requests?
By the user agent LLMWebSearch/1.0 (+https://llmwebsearch.com/bot/), which links back to this page. We do not publish a list of IP addresses today. If a request claims to be our reader and behaves differently from what this page describes, tell us: it may be someone else using our name.
What happens to the content it reads?
It goes into one customer's API response and nowhere else. We keep no copy of your page, build no archive of it and do not train any model on it. Our logs record the customer's query excerpt and the number of pages read, not your page's text. The privacy policy describes the logs in full.
Questions from site owners
How fast does a robots.txt change apply?
Within 24 hours. The reader keeps a copy of your robots.txt for at most a day, then reads it again before the next page.
Will blocking the reader remove my pages from search results?
No. Results come from a live web search that we do not control. Blocking the reader means our customers see your title, URL and snippet, but not the text of the page.
Does it respect Crawl-delay?
The reader does not crawl, so it rarely sends more than a few requests to one site. If its visits ever weigh on your server, write to [email protected] and we will slow them down for your domain.
Does it read pages that need a login or a subscription?
No. It sends no credentials and no cookies, and customers are forbidden from asking for such pages by the acceptable use policy.
Who do I contact about the reader?
[email protected]. Include the URL, the time of the request and, if relevant, the lines of your robots.txt. A person reads every message.