# Agent MCP benchmark More experiment-tracking reads now come from coding agents than from humans on many teams. This benchmark measures the part of that workflow you feel every day: whether an agent connected to your tracker's MCP server answers questions about your runs correctly, and how much work it takes to get there. The setup keeps everything identical except the server: the same deterministic dataset seeded to both trackers, the same agent (headless Claude Code, `claude-sonnet-5`), the same prompts, and the same graders. Each task runs as an isolated agent session that can see exactly one MCP server. ## Result Latest run: July 13, 2026. Eight read/analysis tasks, three trials per side. | Metric | InstantML MCP | W&B MCP | | --- | ---: | ---: | | Correct answers | 24 / 24 | 21 / 24 | | Average task score | 1.00 | 0.90 | | Tool calls | 80 | 106 | | Failed tool calls | 0 | 1 | | Agent output tokens | 19,099 | 24,054 | Protocol: 30 seeded runs plus a distractor project, 4 metric series per run, computed ground truth (run counts, best run, top-3 ranking, config lookups, per-optimizer averages, a series threshold crossing, and a config diff), regex/order graders with no LLM judge, and full transcripts retained. The dataset generator, seeders, runner, and grader are committed under `benchmarks/agent-mcp/` in the InstantML repository. ## Why the scores differ Every W&B miss was an in-context data processing error after a correct raw fetch — the W&B platform returned correct data, verified through the W&B public API after the runs: - Ranking the top 3 runs (missed in 1 of 3 trials): the agent fetched all 30 runs' summary blobs through W&B's general-purpose query tool and ranked them itself, dropping the true #3. - Finding where `train/loss` first crossed a threshold (missed in 2 of 3 trials): the agent pulled the full 120-row history and misread the table, answering 37 and 39 for a true 40. The InstantML agent answered the same questions from server-computed results: `tracker.list_runs` sorts by metric summaries server-side, `tracker.query_metrics` returns bounded series, and `tracker.compare_runs` returns computed diffs. When a task needed arithmetic across runs, the agent exported bounded JSON with `tracker.export_runs` and computed the answer in a script instead of estimating from a table in context. The difference is the tool contract, not the agent: when the server returns answers, the agent verifies; when the server returns raw data, the agent becomes the least reliable stage in the pipeline. ## The honest headline - On identical data with identical prompts, the InstantML MCP server produced a correct answer in every trial; the W&B MCP server missed 3 of 24 task-runs, all on in-context ranking or series reading. - The InstantML side also used 25% fewer tool calls and 21% fewer output tokens per benchmark pass. - This does not support a blanket "InstantML's MCP is better for every agent workflow" claim. W&B's server had essentially clean tool-call reliability, and surfaces InstantML does not offer (Weave trace debugging, sweeps, a docs search tool) were out of scope. ## Not measured yet - Latency. The InstantML server ran locally while W&B's ran hosted, so wall time is not comparable. A hosted-vs-hosted rerun is required before any latency claim. - Write-path tasks such as report creation and editing, which need an LLM-judged grader. - Other agent models, larger task sets, and more trials. Three trials of eight tasks is directional evidence, not a statistical result. - W&B Weave traces, sweeps, artifacts, and registry workflows. ## Fairness rules - Both sides are public, documented MCP servers used as shipped; no private request shapes. - Identical dataset, prompts, model, turn limits, and graders on both sides; only the MCP server differs. - Ground truth is computed from the generated dataset, and per-trial scores are published, including InstantML-unfavorable details such as tool-call counts that include retries. - Data correctness in the losing trials was verified against the vendor's own API before attributing failures to the tool layer. ## Reproduce it The harness lives in `benchmarks/agent-mcp/` with a README covering the exact commands: generate the dataset, seed both trackers, run each side with the `claude` CLI, and grade. Seeding W&B writes to an external SaaS, so use a scratch entity and review transcripts before sharing them. ## Next steps - [Use InstantML with agents](/docs/sdk/agent-mcp.md) - [Latency benchmarks](/docs/benchmarks.md) - [InstantML as a W&B alternative](/docs/guides/wandb-alternative.md) ## Agent navigation - [Docs index](/llms.txt) - [Full docs bundle](/llms-full.txt) ### Get Started - [Overview](/docs/index.md) - [Quickstart](/docs/quickstart.md) - [Core Concepts](/docs/concepts/core-concepts.md) - [Experiment Tracking](/docs/guides/experiment-tracking.md) - [Pricing](/docs/pricing.md) - [Benchmarks](/docs/benchmarks.md) - [Examples](/docs/guides/examples.md) - [Troubleshooting](/docs/troubleshooting.md) ### SDK - [Installation Auth](/docs/sdk/installation-auth.md) - [Logging](/docs/sdk/logging.md) - [Tracing](/docs/sdk/tracing.md) - [Metrics Steps](/docs/sdk/metrics-steps.md) - [Config Tags Notes](/docs/sdk/config-tags-notes.md) - [Artifacts Checkpoints](/docs/sdk/artifacts-checkpoints.md) - [Rich Objects](/docs/sdk/rich-objects.md) - [Distributed Training](/docs/sdk/distributed-training.md) - [Console System Integrations](/docs/sdk/console-system-integrations.md) - [Reliability](/docs/sdk/reliability.md) - [Cli Login](/docs/sdk/cli-login.md) - [Querying Data](/docs/sdk/querying-data.md) - [Agent Mcp](/docs/sdk/agent-mcp.md) - [Examples Patterns](/docs/sdk/examples-patterns.md) ### Integrations - [Overview](/docs/integrations/overview.md) - [Pytorch Lightning](/docs/integrations/pytorch-lightning.md) - [Huggingface Transformers](/docs/integrations/huggingface-transformers.md) - [Keras](/docs/integrations/keras.md) - [Wandb](/docs/integrations/wandb.md) ### Dashboard - [Tour](/docs/dashboard/tour.md) - [Organizations Workspaces](/docs/dashboard/organizations-workspaces.md) - [Runs Workspace](/docs/dashboard/runs-workspace.md) - [Metrics Charts](/docs/dashboard/metrics-charts.md) - [Run Detail](/docs/dashboard/run-detail.md) - [Traces](/docs/dashboard/traces.md) - [Compare Runs](/docs/dashboard/compare-runs.md) - [Research Dashboards](/docs/dashboard/research-dashboards.md) - [Artifacts Files](/docs/dashboard/artifacts-files.md) - [Run Health](/docs/dashboard/alerts.md) - [Datasets](/docs/dashboard/datasets.md) - [Checkpoints](/docs/dashboard/checkpoints.md) - [Reports](/docs/dashboard/reports.md) - [Settings Api Keys](/docs/dashboard/settings-api-keys.md) - [Api Tab](/docs/dashboard/api-tab.md) - [Onboarding Team Billing](/docs/dashboard/onboarding-team-billing.md) ### Data - [Imports](/docs/guides/imports.md) - [W&B alternative](/docs/guides/wandb-alternative.md) - [Agent Mcp Benchmark](/docs/guides/agent-mcp-benchmark.md) (current page) - [InstantML vs MLflow](/docs/guides/instantml-vs-mlflow.md) - [W&B import guide](/docs/guides/wandb-import-guide.md) - [W&B and Neptune imports](/docs/guides/wandb-neptune-imports.md) - [Export Usage Limits](/docs/guides/export-usage-limits.md) - [Pricing Limits Billing](/docs/guides/pricing-limits-billing.md) - [Auth Billing Storage](/docs/guides/auth-billing-storage.md) - [Customer Owned Clickhouse](/docs/guides/customer-owned-clickhouse.md) - [Observability](/docs/guides/observability.md) ### API **Practical API guides** - [Authentication](/docs/api/authentication.md) - [Errors And Limits](/docs/api/errors-and-limits.md) - [Health Observability](/docs/api/health-observability.md) - [Projects Runs](/docs/api/projects-runs.md) - [Metrics Series](/docs/api/metrics-series.md) - [Attributes Objects](/docs/api/attributes-objects.md) - [Artifacts](/docs/api/artifacts.md) - [Reports](/docs/api/reports.md) - [Iframe Embeds](/docs/api/iframe-embeds.md) - [Import Export Usage](/docs/api/import-export-usage.md) - [Dashboard Control State](/docs/api/dashboard-control-state.md) **Architecture** - [System Overview](/docs/architecture/system-overview.md) - [Service Planes](/docs/architecture/service-planes.md) - [Storage Model](/docs/architecture/storage-model.md) - [Google Clickhouse](/docs/architecture/google-clickhouse.md) - [Auth Tenancy](/docs/architecture/auth-tenancy.md) - [Schema Reference](/docs/architecture/schema-reference.md) ### API Reference - [API Reference](/docs/api-reference.md)