Benchmark tool calling per model

What it measures

The same three Danube tools, the same ten tasks, one model at a time. A task passes when the agent executed a matching Danube tool successfully and, where the task says so, the final answer mentions the expected thing. Every task in the shipped set uses credential-free tools, so any Danube account can run it. The scoring is in danube.integrations.benchmark and is framework-neutral: give it a LangChain-shaped message list and a TaskSpec and it returns a RunResult.

Run it

pip install "danube[langchain]" langgraph langchain langchain-openai langchain-anthropic langchain-google-genai
export DANUBE_API_KEY=...  OPENAI_API_KEY=...  ANTHROPIC_API_KEY=...  GOOGLE_API_KEY=...

python scripts/model_tool_benchmark.py \
  --model openai:gpt-4.1 \
  --model anthropic:claude-sonnet-4-5 \
  --model google_genai:gemini-2.5-pro

Model ids are provider:model as LangChain’s init_chat_model understands them. Results go to data/benchmarks/results/<date>.json (every run, with the tools each model executed) and <date>.md, a table like:

model	tasks passed	execute success	tool calls per task	latency per task	errors
anthropic:claude-sonnet-4-5	9/10 (90%)	95%	3.1	8.4s	0
openai:gpt-4.1	8/10 (80%)	89%	3.6	6.9s	0

The numbers above are illustrative; run it to get real ones.

Your own tasks

{"tasks": [
  {"id": "hn_top", "prompt": "Top three Hacker News stories?", "expect_tool": ["Hacker News"]},
  {"id": "honest", "prompt": "Send an email to a@b.c", "expect_tool": [], "expect_answer": ["credential"]}
]}

expect_tool substrings are matched against the name of a tool the agent executed successfully; expect_answer against the final answer. Pass the file with --tasks.

Scoring in your own harness

from danube.integrations.benchmark import TaskSpec, score_run, summarize, markdown_table

result = score_run(state["messages"], TaskSpec(id="hn", prompt="…", expect_tool=["Hacker News"]), "openai:gpt-4.1", latency_s=3.2)
print(markdown_table(summarize([result])))

Get Started

Organizations & Governance

Account

CLI

MCP Clients

SDKs & Integrations

Benchmark tool calling per model

What it measures

Run it

Your own tasks

Scoring in your own harness

​What it measures

​Run it

​Your own tasks

​Scoring in your own harness

What it measures

Run it

Your own tasks

Scoring in your own harness