Skip to content
looot docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

Olostep / Crawl website (start)

Olostep tool olostep-crawl-start on looot: input fields, $0.0018 per result, output shape, and code to run it with curl, JavaScript or Python.

Initiates a new crawl process with the specified parameters.

  • Tool id: olostep-crawl-start
  • Provider: Olostep
  • Job: Crawl web (web.crawl)
  • Price: $0.0018 per result. A call that fails at the provider costs $0.

Inputs

Name Type Required Description
exclude_urls array no URL path names in glob pattern to exclude. For example: /careers/**. Excluded URLs will supersede included URLs.
follow_robots_txt boolean no Whether to respect robots.txt rules. If set to false, the crawler will scrape the website regardless of robots.txt disallow directives. true by default.
include_external boolean no Crawl first-degree external links.
include_subdomain boolean no Include subdomains of the website. false by default.
include_urls array no URL path patterns to include in the crawl using glob syntax. Defaults to /** which includes all URLs. Use patterns like /blog/** to crawl specific sections (e.g., only blog pages), /products/*.html for product pages, or multiple patterns for different sections. Supports standard glob features like * (any characters) and ** (recursive matching).
max_depth number no Maximum depth of the crawl. Useful to extract only up to n-degree of links.
max_pages number yes Maximum number of pages to crawl. Recommended for most use cases like crawling an entire website.
scrape_options object no Controls what each individual page scrape requests from the Olostep API. All fields are optional.
search_query string no An optional search query to find specific links and also sort the results by relevance.
start_url string yes The starting point of the crawl.
timeout number no End the crawl after n seconds with the pages completed until then. May take ~10s extra from provided timeout.
top_n number no An optional number to only crawl the top N most relevant links on every page as per search query.
webhook string no HTTPS URL to receive a POST request when the crawl completes. Must be a publicly accessible URL using http:// or https:// protocol. Cannot point to localhost or private IP addresses. See Webhooks for payload format and retry behavior.

Output

The run’s result holds the provider’s answer. Signed in, looot inspect olostep-crawl-start prints its fields.

Run it

Every call needs your API token in LOOOT_TOKEN; Sign in shows how to get one. Each run also needs a new idempotency key, so a retry never pays twice. With wait: 30 the answer comes back inline when the run ends within 30 seconds. Otherwise you get the running run back: poll GET /v1/runs/<runId>.

Example input with placeholder values:

curl -X POST "https://api.looot.ai/v1/runs" \
  -H "Authorization: Bearer $LOOOT_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"endpointId":"olostep-crawl-start","input":{"max_pages":1,"start_url":"https://example.com"},"wait":30}'
const response = await fetch("https://api.looot.ai/v1/runs", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.LOOOT_TOKEN}`,
    "Content-Type": "application/json",
    "Idempotency-Key": crypto.randomUUID(),
  },
  body: JSON.stringify({
    endpointId: "olostep-crawl-start",
    input: {
      max_pages: 1,
      start_url: "https://example.com",
    },
    wait: 30,
  }),
});
const run = await response.json();
console.log(run.status, run.result);
import os
import uuid

import requests

response = requests.post(
    "https://api.looot.ai/v1/runs",
    headers={
        "Authorization": f"Bearer {os.environ['LOOOT_TOKEN']}",
        "Idempotency-Key": str(uuid.uuid4()),
    },
    json={
        "endpointId": "olostep-crawl-start",
        "input": {
            "max_pages": 1,
            "start_url": "https://example.com",
        },
        "wait": 30,
    },
    timeout=90,
)
run = response.json()
print(run["status"], run.get("result"))

To let looot pick among every provider of this job instead, send job:web.crawl as endpointId; see the job page.

Was this page helpful?