Evaluate judge

Run job:judge.evaluate through the looot API: 1 provider, from $0.00029 per call. Inputs, prices, output shape and code for curl, JavaScript and Python.

Evaluate judge.

Job id judge.evaluate, in Generative AI (Judge). As of 2026-10-01, 1 provider serve it through 3 tools. Run job:judge.evaluate and looot picks one of them; with fallback on, a miss moves on to the next. See Jobs.

Inputs

Fields the job’s tools take, most shared first. Any key you send passes through unchanged.

Name	Type	Taken by	Required by	Description
`model`	string	3 of 3	0	Pinned TypeSafe System One model id.
`questions`	object	3 of 3	3	Exactly one noul question. The object’s single key is your own name for it (the announcement’s `name`); the answer comes back under the same key.
`state`	string	3 of 3	3	The text being judged (the announcement’s `text`).

Providers and prices

Provider	Tool	Price
TypeSafe (Jev)	`typesafe-noul`	$0.00029 per call
TypeSafe (Jev)	`typesafe-score`	$0.00033 per call
TypeSafe (Jev)	`typesafe-choice`	$0.00037 per call

A call that fails at the provider costs $0. See What’s free.

Output

Each provider answers in its own shape. Shape of the run’s result from typesafe-noul, checked against 2 real answers:

{ model, usage: { input_tokens, output_tokens }, answers: { spam: { noul, type } } }

Run it

Every call needs your API token in LOOOT_TOKEN; Sign in shows how to get one. Each run also needs a new idempotency key, so a retry never pays twice. With wait: 30 the answer comes back inline when the run ends within 30 seconds. Otherwise you get the running run back: poll GET /v1/runs/<runId>.

Example input with placeholder values:

curl -X POST "https://api.looot.ai/v1/runs" \
  -H "Authorization: Bearer $LOOOT_TOKEN" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"endpointId":"job:judge.evaluate","input":{"state":"<state>","questions":{}},"wait":30}'

const response = await fetch("https://api.looot.ai/v1/runs", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.LOOOT_TOKEN}`,
    "Content-Type": "application/json",
    "Idempotency-Key": crypto.randomUUID(),
  },
  body: JSON.stringify({
    endpointId: "job:judge.evaluate",
    input: {
      state: "<state>",
      questions: {},
    },
    wait: 30,
  }),
});
const run = await response.json();
console.log(run.status, run.result);

import os
import uuid

import requests

response = requests.post(
    "https://api.looot.ai/v1/runs",
    headers={
        "Authorization": f"Bearer {os.environ['LOOOT_TOKEN']}",
        "Idempotency-Key": str(uuid.uuid4()),
    },
    json={
        "endpointId": "job:judge.evaluate",
        "input": {
            "state": "<state>",
            "questions": {},
        },
        "wait": 30,
    },
    timeout=90,
)
run = response.json()
print(run["status"], run.get("result"))

To pin one provider, send its tool id as endpointId instead of job:judge.evaluate.