AgentToolDrop

Agent observability and eval tools

Tools from our catalog for seeing what an agent did and checking whether it was right: platforms that trace each LLM call and tool step and run evals on those traces, test frameworks that run evals in CI, and a framework for benchmark-style task suites. The table shows what each covers, whether you can self-host it, and the license.

Updated . 7 tools from the catalog, GitHub data fetched . How we chose

Several of these pair an open-source part with a hosted product. The self-hosting and license columns say where that line is, as each project documents it.

At a glance

Stars and last push come from GitHub; every other column restates the tool's own docs.

Tools in this guide, in the order of the lists below
Tool CoversSelf-hostingLicensePricingStarsLast push
Arize Phoenix OpenTelemetry traces, evals, datasets, experiments, prompt playgroundLocally, in a container or on KubernetesElastic License 2.0Free11,840
Opik Traces, datasets, experiments, LLM-as-a-judge metrics, monitoringDocker Compose or Helm; or Comet's cloudApache-2.0Open source22,615
Langfuse Traces, prompt management, datasets, evalsDocker Compose, a VM or Kubernetes; or Langfuse CloudMITOpen source35,838
LangSmith Tracing, evals and monitoring on a hosted platformEnterprise plan onlyMITFree tier + paid1,079
Promptfoo Evals, side-by-side provider comparison, red teaming, PR scanningRuns locallyMITOpen source26,000
DeepEval Pytest-style tests with LLM-as-a-judge and other metricsRuns locally; shared reports need a Confident AI accountApache-2.0Open source18,875
Inspect Task suites from datasets, solvers and scorers; 200+ pre-built evalsRuns locally; sandboxes in Docker, Kubernetes, Modal and othersMITOpen source3,005

Tracing, with evals on the same data

  1. Rank 21

    Arize Phoenix

    Evals

    Traces LLM and agent apps over OpenTelemetry and runs evals, datasets and experiments in a local UI

    Pick it if you want an observability and eval UI on your own machine, instrumented with OpenTelemetry. The Elastic License 2.0 is not OSI open source.

    • score 41
    • 11.8k stars
    • 8.2 stars/day since creation
    • pushed
    • arize-phoenix-v20.21.4 on
    • Elastic License 2.0

    Install with pip

    pip install arize-phoenix
phoenix serve
    Use it when, skip it if, stats for Arize Phoenix

    Why you'd want it

    Phoenix collects traces through OpenTelemetry-based OpenInference instrumentation, with integrations for frameworks such as the OpenAI Agents SDK, Claude Agent SDK, LangGraph and Vercel AI SDK across Python, JavaScript, Java and Go. On top of traces it offers LLM-based evals, versioned datasets, experiments, a prompt playground and prompt management. It runs locally with phoenix serve, in a container or on Kubernetes, and exposes a remote MCP endpoint so Claude Code, Cursor and other MCP clients can query traces, datasets and experiments.

    Best for

    • Local trace debugging during development
    • OpenTelemetry-based stacks
    • Self-hosted eval and experiment tracking

    Use it when

    You want an observability and eval UI on your own machine or infrastructure, instrumented with OpenTelemetry rather than a vendor-specific SDK.

    Skip it if

    You need an OSI open source license or want to offer it as a hosted service; the Elastic License 2.0 forbids that. For a managed platform Arize points to its separate Arize AX product.

    Stats

    Stars
    11,840
    Last push
    Release
    arize-phoenix-v20.21.4,
    License
    Elastic License 2.0
    Pricing
    Free
    Works with
    Docker
    Stars per day, since repo creation
    8.2
    New and rising score
    41 of 100

    Full stat block for Arize Phoenix

  2. Rank 26

    Opik

    Evals

    Traces agent runs and LLM calls, scores them with LLM-as-a-judge metrics and monitors production

    Pick it if you want tracing, evaluation and production monitoring under one Apache-2.0 license you can self-host.

    • score 38
    • 22.6k stars
    • 18 stars/day since creation
    • pushed
    • 2.2.100 on
    • Apache-2.0

    Install the Python SDK

    pip install opik
opik configure
    Use it when, skip it if, stats for Opik

    Why you'd want it

    Opik, built by Comet, logs nested traces through a @track decorator in Python, a TypeScript SDK or OpenTelemetry, so multi-step agent runs and tool calls show up as full trace trees. It adds datasets, experiments, LLM-as-a-judge metrics for hallucination, moderation and RAG, a PyTest integration for CI, online evaluation rules, dashboards and an Agent Optimizer SDK. The full platform, backend included, is Apache-2.0 and can be self-hosted with Docker Compose or Helm, or used on Comet's cloud.

    Best for

    • Teams that want a fully Apache-2.0 self-hosted platform
    • Agent tracing and evals in one tool
    • Running LLM evals in PyTest

    Use it when

    You want tracing, evaluation and production monitoring under one open source license you can self-host, or want your coding agent to read traces through Opik's MCP server.

    Skip it if

    You only need lightweight offline tests; self-hosting runs a multi-container stack with databases and caches, and the alternative is a Comet account.

    Stats

    Stars
    22,615
    Last push
    Release
    2.2.100,
    License
    Apache-2.0
    Pricing
    Open source
    Works with
    Linux, macOS, Windows, Docker
    Stars per day, since repo creation
    18
    New and rising score
    38 of 100

    Full stat block for Opik

  3. Langfuse

    Evals Standard

    Traces LLM calls and agent steps, then runs evals, datasets and prompt versioning on top of them

    Pick it if you need to see what your app did on each request and iterate on prompts against the same traces, self-hosted or in the cloud.

    • 35.8k stars
    • 28.8 stars/day since creation
    • pushed
    • v4.59.0 on
    • MIT

    Install the Python SDK

    pip install langfuse
    Use it when, skip it if, stats for Langfuse

    Why you'd want it

    Langfuse instruments an app through Python and JS/TS SDKs or integrations such as the OpenAI SDK, LangChain, LlamaIndex, Vercel AI SDK and Mastra, and records LLM calls, retrieval and agent actions as traces you can inspect. The same platform covers prompt management with server and client caching, datasets for pre-deployment tests, and evaluations via LLM-as-a-judge, code evaluators, user feedback and manual labeling. You can use Langfuse Cloud or self-host it with Docker Compose, on a VM or on Kubernetes via Helm.

    Best for

    • Teams that want to self-host LLM observability
    • Tracing, prompts and evals in one place
    • Python and JS/TS stacks

    Use it when

    You need to see what your LLM app or agent did on each request and want to evaluate and iterate on prompts against the same traces.

    Skip it if

    You do not want to run a server stack yourself and cannot use a hosted account; self-hosting means running the Docker Compose or Kubernetes deployment. Code under the ee/ directories is not MIT licensed.

    Stats

    Stars
    35,838
    Last push
    Release
    v4.59.0,
    License
    MIT
    Pricing
    Open source
    Works with
    Docker
    Stars per day, since repo creation
    28.8
    Tier
    Standard, not ranked

    Full stat block for Langfuse

  4. LangSmith

    Evals Standard

    Traces, evaluates and monitors LLM apps and agents on LangChain's hosted platform via Python and JS SDKs

    Pick it if you want managed tracing and evals with no servers to run, especially if your app already uses LangChain.

    • 1.1k stars
    • 0.9 stars/day since creation
    • pushed
    • v0.14.8 on
    • MIT

    Install the Python SDK

    pip install -U langsmith
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=ls_...
    Use it when, skip it if, stats for LangSmith

    Why you'd want it

    LangSmith is LangChain's platform for debugging, evaluating and monitoring language model apps and agents; the MIT-licensed Python and JavaScript SDKs send traces to it, for example by wrapping the OpenAI client. It works with any LLM application and has a native integration with LangChain Python and LangChain JS. There is a free Developer plan with a monthly trace allowance and paid per-seat plans, with pay-as-you-go usage beyond the included traces.

    Best for

    • Teams already building with LangChain
    • Hosted tracing without running infrastructure
    • Python and JS/TS apps

    Use it when

    You want managed tracing and evals with no servers to run, especially if your app already uses LangChain.

    Skip it if

    You want to self-host without an enterprise contract; self-hosted and hybrid deployment are Enterprise plan options. Usage beyond the included traces is billed pay-as-you-go.

    Stats

    Stars
    1,079
    Last push
    Release
    v0.14.8,
    License
    MIT
    Pricing
    Free tier + paid
    Stars per day, since repo creation
    0.9
    Tier
    Standard, not ranked

    Full stat block for LangSmith

Evals as tests in CI

  1. Rank 23

    Promptfoo

    Evals

    Runs evals and red-team scans against prompts, models and agents from a CLI, with a web results viewer

    Pick it if you want repeatable local eval runs and red-team scans that can gate changes in CI.

    • score 40
    • 26k stars
    • 20.5 stars/day since creation
    • pushed
    • code-scan-action-0.2.2 on
    • MIT

    Install with npm

    npm install -g promptfoo
promptfoo init --example getting-started
    Use it when, skip it if, stats for Promptfoo

    Why you'd want it

    Promptfoo is a CLI and library that runs automated evaluations of prompts and models, compares providers such as OpenAI, Anthropic, Azure, Bedrock and Ollama side by side, and shows results in a web viewer. It also does red teaming and vulnerability scanning, and can review pull requests for LLM-related security and compliance issues. Evals run locally, it works with any LLM API or programming language, and it plugs into CI/CD.

    Best for

    • Regression tests for prompts in CI
    • Comparing models side by side
    • Red teaming LLM apps before launch

    Use it when

    You want repeatable, local eval runs and security scans that can gate changes in CI.

    Skip it if

    You need tracing and monitoring of live production traffic; promptfoo is built around evals, red teaming and code scanning rather than observability.

    Stats

    Stars
    26,000
    Last push
    Release
    code-scan-action-0.2.2,
    License
    MIT
    Pricing
    Open source
    Stars per day, since repo creation
    20.5
    New and rising score
    40 of 100

    Full stat block for Promptfoo

  2. Rank 37

    DeepEval

    Evals

    Unit tests LLM apps and agents in a Pytest-style workflow with LLM-as-a-judge and other metrics

    Pick it if you want LLM quality checks written as tests that run locally or in any CI environment.

    • score 26
    • 18.9k stars
    • 16.2 stars/day since creation
    • pushed
    • python-v4.2.4 on
    • Apache-2.0

    Install with pip

    pip install -U deepeval
    Use it when, skip it if, stats for DeepEval

    Why you'd want it

    DeepEval works like Pytest for LLM apps: you write test cases and assert them against metrics such as G-Eval, task completion, tool correctness, answer relevancy, faithfulness and hallucination, then run them with deepeval test run. Metrics cover agents, RAG, multi-turn chat, MCP use and multimodal output, can be powered by any LLM, and support both end-to-end and component-level evaluation. It also generates synthetic datasets, runs benchmarks like MMLU and GSM8K, and integrates with LangChain, LangGraph, CrewAI, OpenAI Agents, Pydantic AI and others.

    Best for

    • Python teams that want evals in their test suite
    • RAG and agent metrics
    • CI checks on LLM output quality

    Use it when

    You want LLM quality checks written as tests that run locally or in any CI environment.

    Skip it if

    You are not working in Python (it requires Python 3.9+). Shared reports, iteration comparisons and production monitoring live in the separate Confident AI platform, which needs an account.

    Stats

    Stars
    18,875
    Last push
    Release
    python-v4.2.4,
    License
    Apache-2.0
    Pricing
    Open source
    Stars per day, since repo creation
    16.2
    New and rising score
    26 of 100

    Full stat block for DeepEval

Benchmark-style evaluation

  1. Rank 31

    Inspect

    Evals

    Defines and runs model and agent evaluations in Python from datasets, solvers and scorers

    Pick it if you want to measure a model or agent on a task suite, including tool use and sandboxed code, and compare providers.

    • score 31
    • 3k stars
    • 2.8 stars/day since creation
    • pushed
    • 0.3.280 on
    • MIT

    Install with pip

    pip install inspect-ai
    Use it when, skip it if, stats for Inspect

    Why you'd want it

    Inspect, from the UK AI Security Institute and Meridian Labs, builds evaluations from composable datasets, solvers (from a single generate call to multi-turn agents with tools) and scorers, and runs them with inspect eval. It supports custom and MCP tools plus built-in bash, python, web browsing and computer tools, can evaluate external agents like Claude Code, Codex CLI and Gemini CLI, and sandboxes untrusted model code in Docker, Kubernetes, Modal and other systems. It ships over 200 pre-built benchmark evals, support for over 20 model providers, the Inspect View log viewer and a VS Code extension.

    Best for

    • Benchmarking models and agents
    • Agentic evals that run code in sandboxes
    • Research and safety evaluation teams

    Use it when

    You want to measure how a model or agent performs on a task suite, including agentic tasks with tools and sandboxed code, and compare results across providers.

    Skip it if

    You need tracing of a live production app; Inspect is an evaluation framework, and evals are written in Python.

    Stats

    Stars
    3,005
    Last push
    Release
    0.3.280,
    License
    MIT
    Pricing
    Open source
    Works with
    VS Code
    Stars per day, since repo creation
    2.8
    New and rising score
    31 of 100

    Full stat block for Inspect

Newer options worth a look

New and rising tools from the catalog that this guide does not compare yet, by new and rising score. Younger and less proven than the picks above: read each one's skip-it-if first.

  1. Rank 2

    Reef

    Framework

    Turns feedback on an agent's live requests into versioned updates to its model weights or its harness

    • score 74
    • 7.6k stars
    • 168 stars/day since creation
    • pushed
    • v0.1.3 on
    • Apache-2.0

    Install from PyPI with uv

    uv venv && source .venv/bin/activate
uv pip install reef-infra
python3 -c "import reef; print(reef.__version__)"
    Use it when, skip it if, stats for Reef

    Why you'd want it

    Reef sits between an agent and its model as an OpenAI- and Anthropic-compatible inference endpoint, records each interaction and returns a receipt that a later report can score or comment on. Recipes turn eligible feedback into updates, either training model weights with Slime and SGLang on a GPU stack or refining the harness, meaning prompts, rules and skills, through a model API with no local GPUs. Accepted updates go into a version history and are served without restarting Reef. The built-in Reefine recipe refines a coding harness from plain-language asks, such as reproducing a bug with a failing test first.

    Best for

    • Agents that should learn from user feedback
    • Harness optimization without GPUs
    • Continual learning research

    Use it when

    You run an agent with a steady stream of tasks or feedback and want that signal to produce evaluated, versioned improvements instead of manual prompt edits.

    Skip it if

    You want a stable agent today: harness work needs a model endpoint, representative tasks and an evaluator, weight training needs a supported GPU stack, and it is at v0.1.3 with the repo created on 2026-08-31. The harness install pipes a script into bash and the pi adapter runs commands with no sandbox, so keep the install root outside the project the agent works in.

    Stats

    Stars
    7,602
    Last push
    Release
    v0.1.3,
    License
    Apache-2.0
    Pricing
    Open source
    Stars per day, since repo creation
    168
    New and rising score
    74 of 100

    Full stat block for Reef

  2. Rank 10

    SkillSpector

    Devtool

    Scans agent skills for prompt injection, data exfiltration and supply-chain risks before you install them

    • score 52
    • 20.2k stars
    • 97.1 stars/day since creation
    • pushed
    • v2.12.0 on
    • Apache-2.0

    Install with uv

    uv tool install git+https://github.com/NVIDIA/skillspector.git
    Use it when, skip it if, stats for SkillSpector

    Why you'd want it

    SkillSpector scans a skill from a Git repo, URL, zip file, directory or single SKILL.md against 71 patterns in 17 categories, including prompt injection, data exfiltration, privilege escalation, MCP tool poisoning and MCP least privilege. It runs static analysis first and an optional LLM evaluation second, gives a 0 to 100 risk score, and writes terminal, JSON, Markdown or SARIF reports. It can also run as an MCP server with a scan_skill tool, so Claude Code, Codex CLI or Gemini CLI can gate skill and MCP installs on the result. It is part of NVIDIA's Verified Skills pipeline.

    Best for

    • Vetting third-party skills before install
    • CI checks with SARIF output
    • Gating installs from inside an agent

    Use it when

    You install skills or MCP servers from public repositories and want a report on risky patterns before they reach your agent.

    Skip it if

    You only use skills you wrote yourself. The optional LLM analysis sends skill content to the configured provider with your API key; --no-llm runs static checks only. The README notes the stdio MCP transport still has an initialize hang reported in issue #199.

    Stats

    Stars
    20,224
    Last push
    Release
    v2.12.0,
    License
    Apache-2.0
    Pricing
    Open source
    Works with
    Claude Code, Codex, Gemini CLI, Docker
    Stars per day, since repo creation
    97.1
    New and rising score
    52 of 100

    Full stat block for SkillSpector

How we chose

Picked from the 95 tools in our catalog, each verified against its own repository and docs; 7 made this guide. Within each group, new and rising tools come first, sorted by the new and rising score (star velocity, recent pushes and releases, repo age and an editorial weight), then the standards, the well-known tools, by name. It is not a benchmark. The number beside a tool is its rank in the full new and rising ledger, not within this guide; standards are not ranked. Install commands are copied from each project's docs; every tool page links to the page they came from. GitHub signals (stars, last push, latest release) were fetched on 2026-10-15. How the new and rising score works.

Frameworks with tracing built in, such as the OpenAI Agents SDK, are covered in AI agent frameworks compared. The MCP Inspector tests MCP servers rather than agents and is left out.

For agents: this guide as markdown at /guides/agent-observability-and-evals-tools.md, and every guide in /tools.json.

More guides