Quick answer
How do I read web pages I already have the URLs for?
Send up to 20 URLs to POST https://llmwebsearch.com/api/v1/extract. LLMWebSearch reads each public page and returns its main content as Markdown or plain text, with a status per URL. Billing is 1 credit for each started group of 3 pages read, from $1.20/1K searches per search credit as of October 2026. robots.txt is respected. No KYC; pay in crypto.
/extract at a glance
| Method | POST with a JSON body |
|---|---|
| URL | https://llmwebsearch.com/api/v1/extract |
| Input | urls: 1 to 20 http or https addresses |
| Output | Main content per URL, in markdown (default) or text |
| Billing | Reading pages whose URLs you already have costs 1 credit for each started group of 3 pages returned with content, up to 20 URLs per call; refused or failed URLs are not counted |
| robots.txt | Our reader respects robots.txt: it introduces itself as LLMWebSearch, follows the rules a site writes for it or for all robots, and leaves out the content of any page it is not allowed to read |
| Page reader | LLMWebSearch/1.0 (+https://llmwebsearch.com/bot/) |
Read a list of URLs
POST/api/v1/extract
Fetches each URL in parallel and returns its main content, in the order you sent them. Each item has its own content_status, so one refused page does not fail the call. The example reads three pages; titles and values are illustrative.
Authentication: API key in the Authorization: Bearer API_KEY header.
Request body
| Field | Type | Required | Description |
|---|---|---|---|
urls | string[] | Yes | 1 to 20 http or https addresses. Private, local and cloud metadata addresses are refused. |
format | string | No | markdown (default) or text. |
max_chars | integer | No | Length cap of each page's content, 500 to 50,000 characters. Default 6,000. |
{
"urls": [
"https://docs.example.com/security/key-rotation",
"https://blog.example.org/key-rotation-checklist",
"https://example.com/changelog"
],
"max_chars": 20000
}Response 200
{
"credits_used": 1,
"results": [
{
"url": "https://docs.example.com/security/key-rotation",
"title": "Rotating API keys",
"content_status": "ok",
"content": "# Rotating API keys\n\n## Zero-downtime rotation\n\n1. Create a second key...",
"word_count": 948
},
{
"url": "https://blog.example.org/key-rotation-checklist",
"content_status": "blocked_by_robots"
},
{
"url": "https://example.com/changelog",
"title": "Changelog",
"content_status": "ok",
"content": "# Changelog\n\n## 2026-10-20\n\n- New export format...",
"word_count": 2204
}
]
}Code examples
curl -s "https://llmwebsearch.com/api/v1/extract" \
-H "Authorization: Bearer $LLMWEBSEARCH_API_KEY" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://docs.example.com/security/key-rotation", "https://example.com/changelog"], "max_chars": 20000}'import os
import requests
resp = requests.post(
"https://llmwebsearch.com/api/v1/extract",
headers={"Authorization": "Bearer " + os.environ["LLMWEBSEARCH_API_KEY"]},
json={"urls": ["https://docs.example.com/security/key-rotation", "https://example.com/changelog"]},
timeout=60,
)
for page in resp.json()["results"]:
print(page["url"], page["content_status"], page.get("word_count", 0))const res = await fetch('https://llmwebsearch.com/api/v1/extract', {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.LLMWEBSEARCH_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({ urls: ['https://docs.example.com/security/key-rotation', 'https://example.com/changelog'] }),
});
for (const page of (await res.json()).results) console.log(page.url, page.content_status);When should I use extract instead of read in a search?
Use extract when the URLs do not come from the search you are running. A user pastes a link, a document cites three sources, or a first search with read=0 let the model choose which pages deserve a full read.
Use read inside /search when you simply want the top results read in the same call. It is one request instead of two, and up to 3 pages are included in the single credit of the search.
How are extract calls billed?
One credit for each started group of 3 pages read. The count uses the pages that come back with content: a URL refused by robots.txt or by the site, or one that fails to load, is not counted.
| Pages read | Credits |
|---|---|
| 1 to 3 | 1 |
| 4 to 6 | 2 |
| 7 to 9 | 3 |
| 19 or 20 | 7 |
The response always states the total in credits_used. A call where no page could be read costs nothing.
What does each item of the response tell me?
Each item repeats the url you sent and gives a content_status. With ok, you also get the page title, its main content and its word_count. Otherwise the status names the reason: blocked_by_robots, blocked_by_site_policy, target_refused with the site's target_status, fetch_failed, fetch_timeout or unsupported_content.
Items keep the order of your urls list, so you can match them by index as well as by URL. The response format page describes each status.
Which URLs are refused before any fetch?
Addresses that point to private networks, to the local machine or to cloud metadata services are refused. This prevents anyone from using our reader to reach internal systems. A redirect that leads to such an address is refused too, and at most 5 redirects are followed.
The address must use http or https. A call where urls is empty, holds more than 20 entries or contains a malformed address returns 400 invalid_parameter, free of charge.
Does extract follow the same rules as search?
Yes. It is the same reader, introducing itself as LLMWebSearch/1.0 (+https://llmwebsearch.com/bot/). A site that sets Content-Signal: ai-input=no in its robots.txt keeps its pages out of your results' content. When a site turns our reader away, the result comes back without content and with the site's status: we never solve CAPTCHAs, sign in to accounts or work around a block or a paywall.
Pages that need a login, or sit behind a paywall, come back without content. Point the agent to another source, or to a page you are allowed to read with your own browser session.
Extract endpoint: common questions
Which kinds of files can it read?
Web pages are what it is built for. A link to a video, an archive or another file without readable text comes back as unsupported_content, and that URL is not counted in the price.
Is a URL sent twice read twice?
Send each URL once. If a list may hold duplicates, remove them on your side first, so the results match your list one for one.
Is JavaScript rendering extra?
No. When a page needs a browser to show its text, it is rendered at no added cost.
Can extract read a page I am logged in to?
No. The reader never signs in to accounts and sends no cookies of yours. When a site turns our reader away, the result comes back without content and with the site's status: we never solve CAPTCHAs, sign in to accounts or work around a block or a paywall.