How we make this list: we compare pricing pages, product documentation and public user reviews. Ratings come from G2, Capterra or Trustpilot, with the date we checked them.
Key takeaways
- Langfuse and Opik are the strongest open source starting points, with self hosting and generous free tiers.
- LangSmith is the natural pick for teams already building with LangChain, since tracing, evals and deployment share one platform.
- Helicone and Portkey track usage through a gateway, which makes cost, caching and routing control easier than log based tracing alone.
- Arize AX and Braintrust go furthest on evaluation and experiment management for teams shipping agents at scale.
- Traceloop suits teams that want OpenTelemetry based tracing with quality checks running automatically on real traffic.
Quick comparison
| Tool | Best for | Starting price | Free plan |
|---|
 Langfuse | Open source agent observability | $29 per month | Yes |
 LangSmith | LangChain agent engineering | $39 per seat/month | Yes |
 Arize AX | Enterprise observability and evals | $50 per month | Yes |
 Braintrust | Evals and experiment management | $249 per month | Yes |
 Helicone | Gateway based cost and usage tracking | See website | Yes |
 Opik | Open source tracing and evals | $19 per month | Yes |
 Portkey | AI gateway and governance | See website | Yes |
 Traceloop | OpenTelemetry tracing and quality checks | See website | Yes |
Is your product missing or out of date?
We review this list regularly. Vendors can request a listing, send updated pricing or ask about featured placement.
Starting price$29 per month
Free planYes
Langfuse has become the default open source choice for LLM observability, and its pitch is the full loop: trace production behavior, build datasets from real traffic, run experiments and evals, then ship prompt and model changes with evidence. It supports OpenTelemetry instrumentation plus more than 100 integrations, so it works across languages, frameworks and model providers without locking a team into one stack. The free tier covers 50,000 observations per month with no card required, which is enough for a prototype or a small production app.
The trade off is operational. Self hosting gives full data control but means running and scaling the platform yourself, while the cloud plans bill on usage that grows with trace volume. For teams that value open source, data portability and a single platform from tracing to evals, Langfuse is the most complete starting point in this list.
What we like- Open source tracing, evals, prompt management and datasets in one platform
- Self hosting option with full data control
- Free tier with 50,000 observations per month
Watch out for- Self hosting adds operational work
- Usage based pricing needs monitoring at high volume
Starting price$39 per seat/month
Free planYes
LangSmith is LangChain platform for agent engineering, covering observation, evaluation and deployment of agents. Tracing captures prompts, tool calls, retrievals and model outputs step by step, which makes debugging a failing agent far faster than reading raw logs. Datasets, evaluators and a prompt hub let teams test changes before release, and deployment tooling carries the same agent from prototype to production.
The free Developer tier covers one seat and a limited trace volume, while Plus adds seats and higher limits at $39 per seat per month. Teams outside LangChain can still use LangSmith, but the integration advantage is strongest for LangChain and LangGraph users. If your agents already run on LangChain, LangSmith removes the integration work that a separate tracing tool would add.
What we like- Deep integration with LangChain and LangGraph
- Tracing, evals, prompt hub and deployment in one place
- Free Developer tier for small teams
Watch out for- Best value inside the LangChain ecosystem
- Per seat pricing grows with team size
Starting price$50 per month
Free planYes
Arize AX is the enterprise platform for agent observability, evaluation and improvement, built around turning production signals into continual learning. It traces agent steps, scores outputs with evals, and surfaces patterns in failures so teams can fix root causes rather than individual bad outputs. The open source Phoenix project gives developers a free tracing starting point that feeds into the same workflow, and the platform reports processing trillions of spans per year, which signals scale for large deployments.
Pricing starts with a Pro tier around $50 per month, with enterprise terms negotiated beyond that. Smaller teams may find the evaluation design work heavier than a simple tracing tool. For organizations that need compliance, scale and a serious evaluation program around production agents, Arize AX is the most enterprise ready option here.
What we like- Observability plus evals on one platform at enterprise scale
- Open source Phoenix tracing for local development
- Strong compliance coverage including SOC 2 and HIPAA
Watch out for- Enterprise focus means higher cost at scale
- Setup and evaluation design take time to get right
#4Braintrust
Best for: Evals and experiment management4.3/5 on
G2 (58 reviews, checked Oct 2026)
Visit braintrust.dev
Starting price$249 per month
Free planYes
Braintrust positions itself as the active observability platform for agents, and its strength is the loop from observation to improvement. Traces are searchable across millions of logs, evals run against versioned datasets with LLM, code or human scoring, and experiments compare prompts and models side by side. A dedicated database for AI data, Brainstore, speeds investigation across long prompts and tool calls. Product and engineering teams can annotate, discuss and act on the same traces.
The free Starter tier covers small use, but Pro starts at $249 per month, which prices it above simpler tracing tools. Teams that only need basic tracing will pay for evaluation depth they do not use. Teams that run evals as part of every release will find Braintrust workflow the most disciplined in this list.
What we like- Strongest experiment and dataset workflow in this list
- Observability, evals and prompt management for whole teams
- Searchable trace database built for AI data
Watch out for- Pro pricing starts high for small teams
- Best value for teams already running structured evals
Starting priceSee website
Free planYes
Helicone works differently from log based tracers: it sits as a gateway proxy in front of model providers, so requests pass through it and usage, cost, latency and errors are recorded automatically. That design makes cost tracking, caching, rate limiting and provider routing simple, since the gateway sees every request regardless of framework. Teams use it to debug prompts, analyze spend by user or feature, and add guardrails without instrumenting every code path.
Published pricing has varied, with older sources showing lower paid tiers than newer ones, so the current plan should be confirmed on the pricing page before budgeting. The homepage also notes Helicone is no longer part of Mintlify, a change worth confirming in procurement. For API based LLM apps where cost and gateway control matter as much as tracing, Helicone is a practical and lightweight choice.
What we like- Gateway proxy gives cost, latency and usage data with minimal code change
- Caching and rate limiting built in
- Free tier for getting started
Watch out for- Pricing has changed across sources, so confirm current plans
- Gateway model suits API based apps best
Starting price$19 per month
Free planYes
Opik, built by Comet, is an open source AI observability and evaluation platform that logs every agent step, from user input to retrieval and tool calls, then scores outcomes with more than 30 metrics covering relevance, hallucination, task completion and more. Test suites define plain text assertions for what good looks like, an AI assistant suggests code fixes for failures, and production monitoring alerts when traces fail criteria. Guardrails can block policy violations and PII exposure before they reach users.
Cloud pricing starts with a free tier and a Pro plan around $19 per month, with self hosting available from the open source code. The Comet ownership gives it enterprise backing, but buyers comparing it with Langfuse should weigh community size and integration breadth. For teams that want open source tracing plus structured evals and automated fixes, Opik is a strong and affordable pick.
What we like- True open source core with a generous free tier
- Tracing, LLM as a judge evals, prompt management and monitoring in one place
- Automated fix suggestions and test suites
Watch out for- Newer brand under Comet, so some buyers will check roadmap stability
- Advanced features sit on paid tiers
Starting priceSee website
Free planYes
Portkey gives AI teams a production stack: an AI gateway, observability, guardrails, governance and prompt management in one platform. Routing requests through the gateway means usage, cost and quality are tracked across every model and provider, while caching, load balancing and fallbacks improve reliability and cut spend. Governance features cover access control, PII redaction, budget limits and audit logs, which matter for enterprises running many teams on shared models.
Pricing is not published as a simple starting figure, so budget on a sales conversation, and note that the homepage now presents Portkey as PRISMA AIRS AI Gateway, which signals a branding transition to confirm during procurement. For organizations standardizing many teams on one gateway with governance, Portkey is the most complete platform here. For a single app, a lighter tracer may be enough.
What we like- Gateway with observability, guardrails, governance and prompt management in one platform
- Access to 1,600+ models through one API
- Caching and routing cut model spend
Watch out for- Gateway and governance depth can exceed small team needs
- Homepage now presents Portkey as PRISMA AIRS AI Gateway, so branding is in transition
Starting priceSee website
Free planYes
Traceloop is an LLM reliability platform built on OpenTelemetry, with OpenLLMetry as its open source instrumentation SDK. A single line of setup starts capturing prompts, responses, latency and tool calls, and built in quality checks for faithfulness, relevance and safety run on real traffic without writing tests first. Custom evaluators can be trained from annotated examples, and quality gates can run on pull requests or in production to block regressions.
Deployment flexibility is a selling point, with cloud, on premises and air gapped options plus SOC 2 and HIPAA compliance. Pricing is not published, so teams should request it early. For teams that already standardize on OpenTelemetry and want quality monitoring with minimal setup, Traceloop is the most standards aligned choice in this list.
What we like- One line OpenTelemetry based setup with OpenLLMetry SDK
- Quality checks run automatically on real data
- SOC 2 and HIPAA compliant with cloud, on prem and air gapped options
Watch out for- Pricing is not published on the homepage
- Smaller ecosystem than Langfuse or LangSmith
How we chose
We compared vendor product pages, documentation and public comparisons, and we opened each homepage to confirm the product exists and matches its name. We favored tools that track LLM calls, traces, cost or quality in production, covering open source tracers, gateway platforms and evaluation focused tools, over generic analytics or logging products. Prices are taken from vendor and comparison sources where they agreed, and marked See website where sources conflicted or pricing is not published. Portkey homepage branding now presents it as PRISMA AIRS AI Gateway, and Helicone notes it is no longer part of Mintlify; both are flagged in the tool entries.
What to track first in an LLM app
Start with the four signals that catch most production problems: cost, latency, errors and quality. Cost tracking by model, user and feature prevents surprise bills when a new model or prompt increases token use. Latency tracking by step shows whether slowness comes from the model, retrieval or a tool call. Error tracking covers provider failures, timeouts and malformed outputs. Quality tracking needs evals, not just logs, since a response can be fast, cheap and wrong.
Choose gateway or tracer based on your stack
Gateway tools like Helicone and Portkey suit teams that call models through APIs and want cost, caching and routing control with little code change. Tracer tools like Langfuse, LangSmith, Opik and Traceloop suit teams that need step by step visibility into agents, retrieval and tool calls, plus evals tied to traces. Many production teams run both: a gateway for cost and reliability, and a tracer for debugging and evaluation. Decide which problem hurts more today, and add the other layer when the first is stable.
Keep evals tied to real traffic
Evals built only on hand written examples drift from production reality. The strongest workflow in this category is to sample real traces, annotate failures, build datasets from those cases, and run evals on every prompt or model change. That loop, supported best by Langfuse, Braintrust, Arize AX and Opik, turns tracking from a dashboard into a quality system. Start with a small set of checks that matter to users, such as factual accuracy, safety and task completion, and expand only when those checks run reliably.
Frequently asked questions
What is LLM tracking?
LLM tracking, also called LLM observability, means recording and analyzing what happens when an application calls a language model: prompts, responses, tool calls, retrieval steps, latency, token use, cost and errors. It gives teams the visibility to debug failures, control spend and measure quality. Without it, a bad model output is hard to reproduce or explain.
What is the difference between tracing and evaluation?
Tracing records what happened in a single run, step by step, so you can debug it. Evaluation scores whether outputs were good, using metrics, LLM judges, code checks or human review across many runs. Tracing answers what happened, evaluation answers whether it was good enough. Most tools in this list do both, with different strengths.
Do I need an LLM tracking tool for a small app?
A small prototype can survive on provider dashboards and logs, but any app with real users benefits from tracing once debugging gets painful or costs grow. Free tiers from Langfuse, LangSmith, Opik and Helicone cover early use at no cost, and open source self hosting removes the subscription entirely. Start with tracing, add evals when quality complaints appear.
How does gateway based tracking differ from SDK tracing?
Gateway tracking routes model requests through a proxy, so it captures cost, latency and usage with minimal code change and adds caching, routing and guardrails. SDK tracing instruments your code, so it captures agent steps, tool calls and retrieval in detail. Gateway tools are simpler to adopt, SDK tracers give deeper debugging. Many teams use both.
Can I self host an LLM tracking tool?
Yes. Langfuse and Opik both offer open source self hosting, and Arize Phoenix provides open source tracing. Self hosting gives full data control and can suit regulated environments, but it adds operational work for scaling, backups and upgrades. Cloud plans trade that work for a subscription and usage based pricing.
How much do LLM tracking tools cost?
Free tiers cover small use on most tools here. Paid plans start around $19 per month for Opik, $29 for Langfuse, $39 per seat for LangSmith and $50 for Arize AX, with Braintrust Pro at $249 per month. Gateway and enterprise tools often use negotiated pricing. Budget on trace volume and seats, since both drive cost at scale, and check whether the free tier covers your expected monthly observations.
Sources we checked (8)
- https://langfuse.com/pricing
- https://smith.langchain.com/pricing
- https://arize.com/pricing
- https://braintrust.dev/pricing
- https://helicone.ai/pricing
- https://opik.com/pricing
- https://portkey.ai/pricing
- https://traceloop.com/pricing
Is your product missing or out of date?
We review this list regularly. Vendors can request a listing, send updated pricing or ask about featured placement.
Shihab Mia · Founder, Rankite
Shihab has 5+ years in B2B and SaaS SEO. He writes Rankite's software buying guides from pricing pages, documentation and public user reviews. More about Shihab.