Olostep / Crawl pages
Olostep tool olostep-crawl-pages on looot: input fields, $0 per call, output shape, and code to run it with curl, JavaScript or Python.
Fetches the list of crawled pages and content that have been processed for a specific crawl ID.
- Tool id:
olostep-crawl-pages - Provider: Olostep
- Job: Results web crawl (
web.crawl.results) - Price: $0 per call. A call that fails at the provider costs $0.
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
cursor |
integer | no | Optional integer representing the index to start fetching content from. Useful to paginate until all URLs are fetched. Start with 0, then provide response['cursor'] value of the last request. |
formats |
array | no | Deprecated: Use /retrieve endpoint with retrieve_id. Array of formats to fetch (e.g., [“html”, “markdown”]). |
limit |
integer | no | Optional integer to limit the number of results returned. Recommended 10-50 results at a time. Paginated using cursor. Maximum 10MB of content can be fetched in a single request. |
search_query |
string | no | An optional search query to sort the results by relevance. Uses the original search_query by default if provided. |
crawl_id |
string | yes | The ID of the crawl to retrieve the list of URLs for. |
Output
The run’s result holds the provider’s answer. Signed in, looot inspect olostep-crawl-pages prints its fields.
Run it
Every call needs your API token in LOOOT_TOKEN; Sign in shows how to get one. Each run also needs a new idempotency key, so a retry never pays twice. With wait: 30 the answer comes back inline when the run ends within 30 seconds. Otherwise you get the running run back: poll GET /v1/runs/<runId>.
Example input with placeholder values:
curl -X POST "https://api.looot.ai/v1/runs" \
-H "Authorization: Bearer $LOOOT_TOKEN" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{"endpointId":"olostep-crawl-pages","input":{"crawl_id":"<crawl_id>"},"wait":30}'const response = await fetch("https://api.looot.ai/v1/runs", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.LOOOT_TOKEN}`,
"Content-Type": "application/json",
"Idempotency-Key": crypto.randomUUID(),
},
body: JSON.stringify({
endpointId: "olostep-crawl-pages",
input: {
crawl_id: "<crawl_id>",
},
wait: 30,
}),
});
const run = await response.json();
console.log(run.status, run.result);import os
import uuid
import requests
response = requests.post(
"https://api.looot.ai/v1/runs",
headers={
"Authorization": f"Bearer {os.environ['LOOOT_TOKEN']}",
"Idempotency-Key": str(uuid.uuid4()),
},
json={
"endpointId": "olostep-crawl-pages",
"input": {
"crawl_id": "<crawl_id>",
},
"wait": 30,
},
timeout=90,
)
run = response.json()
print(run["status"], run.get("result"))To let looot pick among every provider of this job instead, send job:web.crawl.results as endpointId; see the job page.