Skip to content
looot docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

Olostep / Crawl pages

Olostep tool olostep-crawl-pages on looot: input fields, $0 per call, output shape, and code to run it with curl, JavaScript or Python.

Fetches the list of crawled pages and content that have been processed for a specific crawl ID.

  • Tool id: olostep-crawl-pages
  • Provider: Olostep
  • Job: Results web crawl (web.crawl.results)
  • Price: $0 per call. A call that fails at the provider costs $0.

Inputs

Name Type Required Description
cursor integer no Optional integer representing the index to start fetching content from. Useful to paginate until all URLs are fetched. Start with 0, then provide response['cursor'] value of the last request.
formats array no Deprecated: Use /retrieve endpoint with retrieve_id. Array of formats to fetch (e.g., [“html”, “markdown”]).
limit integer no Optional integer to limit the number of results returned. Recommended 10-50 results at a time. Paginated using cursor. Maximum 10MB of content can be fetched in a single request.
search_query string no An optional search query to sort the results by relevance. Uses the original search_query by default if provided.
crawl_id string yes The ID of the crawl to retrieve the list of URLs for.

Output

The run’s result holds the provider’s answer. Signed in, looot inspect olostep-crawl-pages prints its fields.

Run it

Every call needs your API token in LOOOT_TOKEN; Sign in shows how to get one. Each run also needs a new idempotency key, so a retry never pays twice. With wait: 30 the answer comes back inline when the run ends within 30 seconds. Otherwise you get the running run back: poll GET /v1/runs/<runId>.

Example input with placeholder values:

curl -X POST "https://api.looot.ai/v1/runs" \
  -H "Authorization: Bearer $LOOOT_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"endpointId":"olostep-crawl-pages","input":{"crawl_id":"<crawl_id>"},"wait":30}'
const response = await fetch("https://api.looot.ai/v1/runs", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.LOOOT_TOKEN}`,
    "Content-Type": "application/json",
    "Idempotency-Key": crypto.randomUUID(),
  },
  body: JSON.stringify({
    endpointId: "olostep-crawl-pages",
    input: {
      crawl_id: "<crawl_id>",
    },
    wait: 30,
  }),
});
const run = await response.json();
console.log(run.status, run.result);
import os
import uuid

import requests

response = requests.post(
    "https://api.looot.ai/v1/runs",
    headers={
        "Authorization": f"Bearer {os.environ['LOOOT_TOKEN']}",
        "Idempotency-Key": str(uuid.uuid4()),
    },
    json={
        "endpointId": "olostep-crawl-pages",
        "input": {
            "crawl_id": "<crawl_id>",
        },
        "wait": 30,
    },
    timeout=90,
)
run = response.json()
print(run["status"], run.get("result"))

To let looot pick among every provider of this job instead, send job:web.crawl.results as endpointId; see the job page.

Was this page helpful?