# The LLMWebSearch page reader

> For site owners: how the LLMWebSearch page reader identifies itself, when it visits, how it follows robots.txt and Content-Signal, and how to block it.

- URL: https://llmwebsearch.com/bot/
- Last updated: 2026-10-22
- Publisher: LLMWebSearch (https://llmwebsearch.com)

Our page reader opens a public page only when a customer's search needs it. It says who it is, reads robots.txt first, and takes no for an answer. Here is everything a site owner needs to recognise it, limit it or block it.

**Quick answer — What is the LLMWebSearch bot and how do I block it?**

The **LLMWebSearch** page reader fetches public pages that appear in a customer's web search results, so an AI agent can read them. It sends the user agent `LLMWebSearch/1.0 (+https://llmwebsearch.com/bot/)` and obeys robots.txt rules for `LLMWebSearch` or for all robots. Two lines in robots.txt block it. It reads at most 10 pages per search, and never works around a refusal.

## The page reader at a glance

| Attribute | Value |
| --- | --- |
| User-Agent header | `LLMWebSearch/1.0 (+https://llmwebsearch.com/bot/)` |
| robots.txt token | `LLMWebSearch` |
| When it visits | Only when your page is among the results a customer asked it to read, or a customer names your URL |
| Pages per search | Up to 10, usually 3 |
| robots.txt | Read before any page; rules cached for 24 hours at most |
| Content-Signal | `ai-input=no` keeps your content out of responses |
| Refusals | Accepted as final: no retry under another name, no CAPTCHA solving, no sign-in |
| Contact | abuse@llmwebsearch.com |

## Why is it visiting my site?

Because a customer's search returned one of your pages near the top, and their agent asked for the text of the first results. Less often, a customer sent your URL to our `/extract` endpoint because they already had the link.

The reader does not crawl. It does not follow your links, walk your sitemap or build an index. Each visit is one page, requested because a person or an agent wanted to read it at that moment.

## How do I block it completely?

Add a group for its token to your robots.txt. The rule takes effect at our next read of the file, within 24 hours. Pages it may not read still appear in search results with their title, URL and snippet, but without their content.

*robots.txt: keep the LLMWebSearch reader out of the whole site*

```text
User-agent: LLMWebSearch
Disallow: /

```

## Can I block only some sections?

Yes. List the paths you want kept out; everything else stays readable. The reader picks the group written for `LLMWebSearch` if there is one, and the `*` group otherwise, like most well-behaved robots.

*robots.txt: keep account and checkout pages out*

```text
User-agent: LLMWebSearch
Disallow: /account/
Disallow: /checkout/
Allow: /

```

## Can I allow search but refuse AI use?

Yes, with a Content Signal. A site that sets Content-Signal: ai-input=no in its robots.txt keeps its pages out of your results' content. Your pages can still be listed as results, so people find them, but their text is not handed to an AI agent through our API.

*robots.txt: listed in results, no content for AI input*

```text
User-agent: *
Content-Signal: search=yes, ai-input=no
Allow: /

```

## What does the reader never do?

When a site turns our reader away, the result comes back without content and with the site's status: we never solve CAPTCHAs, sign in to accounts or work around a block or a paywall. It also does not pretend to be a regular browser, does not switch to another user agent after a refusal, and does not send cookies or credentials.

Some pages only show their text after JavaScript runs. For those, the reader can load the page in a headless browser, still under the same user agent and the same robots.txt rules. A 403, a 429 or a challenge page ends the attempt; the customer receives your answer, not your content.

## How can I recognise its requests?

By the user agent `LLMWebSearch/1.0 (+https://llmwebsearch.com/bot/)`, which links back to this page. We do not publish a list of IP addresses today. If a request claims to be our reader and behaves differently from what this page describes, tell us: it may be someone else using our name.

## What happens to the content it reads?

It goes into one customer's API response and nowhere else. We keep no copy of your page, build no archive of it and do not train any model on it. Our logs record the customer's query excerpt and the number of pages read, not your page's text. The [privacy policy](https://llmwebsearch.com/privacy/) describes the logs in full.

## Questions from site owners

### How fast does a robots.txt change apply?

Within 24 hours. The reader keeps a copy of your robots.txt for at most a day, then reads it again before the next page.

### Will blocking the reader remove my pages from search results?

No. Results come from a live web search that we do not control. Blocking the reader means our customers see your title, URL and snippet, but not the text of the page.

### Does it respect Crawl-delay?

The reader does not crawl, so it rarely sends more than a few requests to one site. If its visits ever weigh on your server, write to abuse@llmwebsearch.com and we will slow them down for your domain.

### Does it read pages that need a login or a subscription?

No. It sends no credentials and no cookies, and customers are forbidden from asking for such pages by the [acceptable use policy](https://llmwebsearch.com/acceptable-use/).

### Who do I contact about the reader?

abuse@llmwebsearch.com. Include the URL, the time of the request and, if relevant, the lines of your robots.txt. A person reads every message.

---

LLMWebSearch: LLMWebSearch is a web search API for AI agents that returns ranked results with the content of the top pages in Markdown, paid in crypto with no ID upload. Prepaid credits from $1.20/1K searches never expire. No KYC — no ID, no selfie, no documents. Pay in crypto.