---
title: "Olostep / Crawl website (start)"
description: "Olostep tool olostep-crawl-start on looot: input fields, $0.0018 per result, output shape, and code to run it with curl, JavaScript or Python."
sidebar:
  hidden: true
---

{/* Generated by scripts/generate-api-reference.mjs from data/api-reference.json. Do not edit. */}

Initiates a new crawl process with the specified parameters.

- **Tool id:** `olostep-crawl-start`
- **Provider:** [Olostep](/providers/olostep)
- **Job:** [Crawl web](/reference/jobs/web-crawl) (`web.crawl`)
- **Price:** $0.0018 per result. A call that fails at the provider costs $0.

## Inputs

| Name | Type | Required | Description |
| --- | --- | --- | --- |
| `exclude_urls` | array | no | URL path names in glob pattern to exclude. For example: `/careers/**`. Excluded URLs will supersede included URLs. |
| `follow_robots_txt` | boolean | no | Whether to respect robots.txt rules. If set to `false`, the crawler will scrape the website regardless of robots.txt disallow directives. `true` by default. |
| `include_external` | boolean | no | Crawl first-degree external links. |
| `include_subdomain` | boolean | no | Include subdomains of the website. `false` by default. |
| `include_urls` | array | no | URL path patterns to include in the crawl using glob syntax. Defaults to `/**` which includes all URLs. Use patterns like `/blog/**` to crawl specific sections (e.g., only blog pages), `/products/*.html` for product pages, or multiple patterns for different sections. Supports standard glob features like * (any characters) and ** (recursive matching). |
| `max_depth` | number | no | Maximum depth of the crawl. Useful to extract only up to n-degree of links. |
| `max_pages` | number | yes | Maximum number of pages to crawl. Recommended for most use cases like crawling an entire website. |
| `scrape_options` | object | no | Controls what each individual page scrape requests from the Olostep API. All fields are optional. |
| `search_query` | string | no | An optional search query to find specific links and also sort the results by relevance. |
| `start_url` | string | yes | The starting point of the crawl. |
| `timeout` | number | no | End the crawl after n seconds with the pages completed until then. May take ~10s extra from provided timeout. |
| `top_n` | number | no | An optional number to only crawl the top N most relevant links on every page as per search query. |
| `webhook` | string | no | HTTPS URL to receive a POST request when the crawl completes. Must be a publicly accessible URL using `http://` or `https://` protocol. Cannot point to localhost or private IP addresses. See Webhooks for payload format and retry behavior. |

## Output

The run's `result` holds the provider's answer. Signed in, `looot inspect olostep-crawl-start` prints its fields.

## Run it

Every call needs your API token in `LOOOT_TOKEN`; [Sign in](/get-started/sign-in#use-the-token-in-scripts-and-agents) shows how to get one. Each run also needs a new idempotency key, so a retry never pays twice. With `wait: 30` the answer comes back inline when the run ends within 30 seconds. Otherwise you get the running run back: poll `GET /v1/runs/<runId>`.

Example input with placeholder values:

<CodeGroup>

```bash curl
curl -X POST "https://api.looot.ai/v1/runs" \
  -H "Authorization: Bearer $LOOOT_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"endpointId":"olostep-crawl-start","input":{"max_pages":1,"start_url":"https://example.com"},"wait":30}'
```

```js JavaScript
const response = await fetch("https://api.looot.ai/v1/runs", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.LOOOT_TOKEN}`,
    "Content-Type": "application/json",
    "Idempotency-Key": crypto.randomUUID(),
  },
  body: JSON.stringify({
    endpointId: "olostep-crawl-start",
    input: {
      max_pages: 1,
      start_url: "https://example.com",
    },
    wait: 30,
  }),
});
const run = await response.json();
console.log(run.status, run.result);
```

```python Python
import os
import uuid

import requests

response = requests.post(
    "https://api.looot.ai/v1/runs",
    headers={
        "Authorization": f"Bearer {os.environ['LOOOT_TOKEN']}",
        "Idempotency-Key": str(uuid.uuid4()),
    },
    json={
        "endpointId": "olostep-crawl-start",
        "input": {
            "max_pages": 1,
            "start_url": "https://example.com",
        },
        "wait": 30,
    },
    timeout=90,
)
run = response.json()
print(run["status"], run.get("result"))
```

</CodeGroup>

To let looot pick among every provider of this job instead, send `job:web.crawl` as `endpointId`; see [the job page](/reference/jobs/web-crawl).
