Olostep / Crawl website (start)
Olostep tool olostep-crawl-start on looot: input fields, $0.0018 per result, output shape, and code to run it with curl, JavaScript or Python.
Initiates a new crawl process with the specified parameters.
- Tool id:
olostep-crawl-start - Provider: Olostep
- Job: Crawl web (
web.crawl) - Price: $0.0018 per result. A call that fails at the provider costs $0.
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
exclude_urls |
array | no | URL path names in glob pattern to exclude. For example: /careers/**. Excluded URLs will supersede included URLs. |
follow_robots_txt |
boolean | no | Whether to respect robots.txt rules. If set to false, the crawler will scrape the website regardless of robots.txt disallow directives. true by default. |
include_external |
boolean | no | Crawl first-degree external links. |
include_subdomain |
boolean | no | Include subdomains of the website. false by default. |
include_urls |
array | no | URL path patterns to include in the crawl using glob syntax. Defaults to /** which includes all URLs. Use patterns like /blog/** to crawl specific sections (e.g., only blog pages), /products/*.html for product pages, or multiple patterns for different sections. Supports standard glob features like * (any characters) and ** (recursive matching). |
max_depth |
number | no | Maximum depth of the crawl. Useful to extract only up to n-degree of links. |
max_pages |
number | yes | Maximum number of pages to crawl. Recommended for most use cases like crawling an entire website. |
scrape_options |
object | no | Controls what each individual page scrape requests from the Olostep API. All fields are optional. |
search_query |
string | no | An optional search query to find specific links and also sort the results by relevance. |
start_url |
string | yes | The starting point of the crawl. |
timeout |
number | no | End the crawl after n seconds with the pages completed until then. May take ~10s extra from provided timeout. |
top_n |
number | no | An optional number to only crawl the top N most relevant links on every page as per search query. |
webhook |
string | no | HTTPS URL to receive a POST request when the crawl completes. Must be a publicly accessible URL using http:// or https:// protocol. Cannot point to localhost or private IP addresses. See Webhooks for payload format and retry behavior. |
Output
The run’s result holds the provider’s answer. Signed in, looot inspect olostep-crawl-start prints its fields.
Run it
Every call needs your API token in LOOOT_TOKEN; Sign in shows how to get one. Each run also needs a new idempotency key, so a retry never pays twice. With wait: 30 the answer comes back inline when the run ends within 30 seconds. Otherwise you get the running run back: poll GET /v1/runs/<runId>.
Example input with placeholder values:
curl -X POST "https://api.looot.ai/v1/runs" \
-H "Authorization: Bearer $LOOOT_TOKEN" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $(uuidgen)" \
-d '{"endpointId":"olostep-crawl-start","input":{"max_pages":1,"start_url":"https://example.com"},"wait":30}'const response = await fetch("https://api.looot.ai/v1/runs", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.LOOOT_TOKEN}`,
"Content-Type": "application/json",
"Idempotency-Key": crypto.randomUUID(),
},
body: JSON.stringify({
endpointId: "olostep-crawl-start",
input: {
max_pages: 1,
start_url: "https://example.com",
},
wait: 30,
}),
});
const run = await response.json();
console.log(run.status, run.result);import os
import uuid
import requests
response = requests.post(
"https://api.looot.ai/v1/runs",
headers={
"Authorization": f"Bearer {os.environ['LOOOT_TOKEN']}",
"Idempotency-Key": str(uuid.uuid4()),
},
json={
"endpointId": "olostep-crawl-start",
"input": {
"max_pages": 1,
"start_url": "https://example.com",
},
"wait": 30,
},
timeout=90,
)
run = response.json()
print(run["status"], run.get("result"))To let looot pick among every provider of this job instead, send job:web.crawl as endpointId; see the job page.