> ## Documentation Index
> Fetch the complete documentation index at: https://docs.danubeai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark tool calling per model

> Run the same task set through any chat model with the Danube tools and score which models call tools correctly

## What it measures

The same three Danube tools, the same ten tasks, one model at a time. A task passes when the agent executed a matching Danube tool successfully and, where the task says so, the final answer mentions the expected thing. Every task in the shipped set uses credential-free tools, so any Danube account can run it.

The scoring is in `danube.integrations.benchmark` and is framework-neutral: give it a LangChain-shaped message list and a `TaskSpec` and it returns a `RunResult`.

## Run it

```bash theme={null}
pip install "danube[langchain]" langgraph langchain langchain-openai langchain-anthropic langchain-google-genai
export DANUBE_API_KEY=...  OPENAI_API_KEY=...  ANTHROPIC_API_KEY=...  GOOGLE_API_KEY=...

python scripts/model_tool_benchmark.py \
  --model openai:gpt-4.1 \
  --model anthropic:claude-sonnet-4-5 \
  --model google_genai:gemini-2.5-pro
```

Model ids are `provider:model` as LangChain's `init_chat_model` understands them. Results go to `data/benchmarks/results/<date>.json` (every run, with the tools each model executed) and `<date>.md`, a table like:

| model                       | tasks passed | execute success | tool calls per task | latency per task | errors |
| --------------------------- | ------------ | --------------- | ------------------- | ---------------- | ------ |
| anthropic:claude-sonnet-4-5 | 9/10 (90%)   | 95%             | 3.1                 | 8.4s             | 0      |
| openai:gpt-4.1              | 8/10 (80%)   | 89%             | 3.6                 | 6.9s             | 0      |

The numbers above are illustrative; run it to get real ones.

## Your own tasks

```json theme={null}
{"tasks": [
  {"id": "hn_top", "prompt": "Top three Hacker News stories?", "expect_tool": ["Hacker News"]},
  {"id": "honest", "prompt": "Send an email to a@b.c", "expect_tool": [], "expect_answer": ["credential"]}
]}
```

`expect_tool` substrings are matched against the name of a tool the agent executed successfully; `expect_answer` against the final answer. Pass the file with `--tasks`.

## Scoring in your own harness

```python theme={null}
from danube.integrations.benchmark import TaskSpec, score_run, summarize, markdown_table

result = score_run(state["messages"], TaskSpec(id="hn", prompt="…", expect_tool=["Hacker News"]), "openai:gpt-4.1", latency_s=3.2)
print(markdown_table(summarize([result])))
```