Runs on: Windows WSL2 macOS Linux Docker
-
1 The trace
Langfuse
langfuse/langfuseSelf-hostable, which matters here more than anywhere: the traces contain your prompts and your customers' data. The most widely adopted of the open options, and two source lists agree.
The most widely adopted self-hostable LLM observability platform: traces every agent step, manages prompt versions, and runs evals in one tool. Preferred over cloud-only alternatives when data residency or cost control is a constraint
pip install langfuse openai -
2 The second opinion
Arize-Phoenix
Arize-ai/phoenixThree source lists carry it -- the strongest agreement in this category -- and it pairs a trace UI with an eval runtime, so you can grade the runs you are looking at.
Self-hostable trace UI and eval runtime for agent workflows. Lets harness engineers audit and replay every reasoning step and tool call offline, without sending data to a third-party cloud
npx @arizeai/phoenix-cli setup -
3 The test suite
promptfoo
promptfoo/promptfooTests as YAML, with assertions and LLM-as-judge. It is the lowest-ceremony way to turn "it seemed better" into something a pull request can fail on.
YAML-driven LLM testing framework with LLM-as-judge, assertion DSL, and native CI integration. The most practical tool for adding agent output regression tests to a PR pipeline without writing a test harness from scratch
npm install -g promptfoo -
4 The metrics
DeepEval
confident-ai/deepevalTwenty-plus built-in metrics for LLM and agent evaluation. Use it when you have stopped arguing about whether output is good and started arguing about which axis.
The most complete open-source LLM/agent eval framework: 20+ built-in metrics (hallucination, answer relevancy, RAGAs, tool correctness), pytest integration, and a CI-friendly runner. Removes the need to hand-roll eval infrastructure when you need structured, repeatable agent quality gates
pip install -U deepeval -
5 The standard
OpenLLMetry
traceloop/openllmetryOpenTelemetry instrumentation for LLM calls and agent steps. The one pick here that does not lock you in: the traces go wherever your existing observability already goes.
OpenTelemetry-based instrumentation for LLM calls and agent steps: adds trace spans to every inference and tool call without modifying business logic. The cleanest way to bring the existing OTEL ecosystem (Grafana, Datadog, Jaeger) to a harness
pip install traceloop-sdk -
6 The thing you install
SkillSpector
NVIDIA/SkillSpectorScans agent skills for vulnerabilities. Every other page on this site encourages you to install other people's instructions; this is the tool that reads them first.
by NVIDIA - Security scanner for AI agent skills. Detect vulnerabilities, malicious patterns, and security risks
uv tool install git+https://github.com/NVIDIA/skillspector.git
The whole set, in one paste
Every install command above, in the order they are listed. Read them before you run them — they are other people's one-liners, and none of this set pipes a download into a shell.
pip install langfuse openai npx @arizeai/phoenix-cli setup npm install -g promptfoo pip install -U deepeval pip install traceloop-sdk uv tool install git+https://github.com/NVIDIA/skillspector.git