How we make this list: we compare pricing pages, product documentation and public user reviews. Ratings come from G2, Capterra or Trustpilot, with the date we checked them.
Key takeaways
- LangSmith, Langfuse and Arize AX cover the widest ground, combining traces, evals and production monitoring in one place.
- Usage based pricing can rise fast for agent workloads, so check how traces, spans and retention are counted before you commit.
- Open source options such as Langfuse, Opik and Traceloop give teams more control over data and deployment.
- Portkey stands out when routing, fallbacks and cost control matter as much as observability.
- Free plans are useful for a pilot, but retention limits and seat limits often decide the real cost.
Quick comparison
| Tool | Best for | Starting price | Free plan |
|---|
 LangSmith | LangChain and LangGraph teams | $39 per seat/month | Yes |
 Langfuse | Open source tracing and evals | $29 per month | Yes |
 Arize AI | Production agent evaluation | $50 per month | Yes |
 Portkey | Multi-model routing and guardrails | $49 per month | Yes |
 HoneyHive | Agent observability and evaluation | See website | Yes |
 Weights and Biases Weave | Teams already using Weights and Biases | $60 per month | Yes |
 Traceloop | OpenTelemetry based LLM tracking | See website | Yes |
 Opik | Open source evals and monitoring | $19 per month | Yes |
Is your product missing or out of date?
We review this list regularly. Vendors can request a listing, send updated pricing or ask about featured placement.
Starting price$39 per seat/month
Free planYes
LangSmith is the natural first choice for teams building with LangChain or LangGraph, because tracing, datasets, evals and deployment live in one workspace. A trace shows each model call, tool call and retrieval step in order, which makes it far easier to find why an agent looped, picked the wrong tool or returned a weak answer. The evaluation side lets teams score outputs against datasets, collect human feedback and compare prompt versions before a change reaches users.
The catch is billing. The Developer plan is free with a limited trace allowance, while Plus starts at 39 dollars per seat per month with a larger base trace allowance and pay-as-you-go use beyond it. Large teams should model seat count and trace volume together, and set retention and sampling rules early. Teams outside the LangChain world can still use LangSmith through its SDKs, but they may get less value from the framework specific parts.
What we like- Deep tracing for agents and chains
- Strong evaluation and dataset tools
- Prompt hub and deployment options
Watch out for- Per-seat pricing adds up for large teams
- Usage based trace charges need close watching
Starting price$29 per month
Free planYes
Langfuse suits teams that want full tracing and evaluation without paying per seat. Its Hobby plan is free, and Core starts at 29 dollars per month with a larger unit allowance and longer data access. Units cover traces, observations and scores, so agent runs with many tool calls consume units faster than simple chat calls. That model is fair for small teams, but it rewards careful sampling and retention settings.
Prompt versioning, datasets and experiments are central here, not add-ons. Teams can test a prompt change against a fixed dataset, review scores side by side and roll back when a change regresses. Self-hosting is a real option for teams with data residency needs, though running the database and storage stack adds operational work. For most small and mid-size teams, the hosted service is the simpler start.
What we like- Open source and self-hostable
- Usage based pricing with no per-seat charge on paid plans
- Prompt management and experiments included
Watch out for- Unit based pricing takes care to forecast
- Advanced compliance features sit on higher plans
Starting price$50 per month
Free planYes
Arize AX is built for teams that need to watch agents in production and keep improving them, not just log calls. Its free tier gives a modest span allowance, and AX Pro starts at 50 dollars per month with a larger span and ingestion allowance. Because pricing is based on span and data volume rather than seats, product managers, support staff and engineers can review traces without changing the bill. That matters once quality work spreads beyond engineering.
The platform also includes Phoenix, a free self-hosted option for tracing and evals, which is useful for a local pilot. In the hosted product, evaluation, experiments and issue detection help teams spot drift, compare agent versions and turn failures into test cases. The main watch point is span volume. Multi-step agents create many spans per conversation, so a pilot should measure real span counts before sizing a plan.
What we like- Unlimited users on hosted tiers
- Span based pricing with unlimited evaluations
- Strong agent tracing and issue detection
Watch out for- Span volume for agents can grow quickly
- Enterprise features require a custom quote
Starting price$49 per month
Free planYes
Portkey is part gateway and part observability tool, which sets it apart from pure tracing products. One OpenAI compatible API can route traffic across many providers and models, with automatic fallbacks and retries when a provider fails or slows down. Logs, traces, cost tracking and feedback sit on top of that routing layer, so teams can see spend and quality in the same place where traffic is controlled.
The Developer plan is free with a limited log allowance and short retention, while the paid plan starts at 49 dollars per month with a much larger log allowance and longer retention. Semantic caching can cut repeated token spend, and virtual keys with budgets help stop runaway usage. Teams that only need tracing may find Portkey broader than necessary, but teams running several models in production often value the control layer more than the dashboard alone.
What we like- One API for many providers
- Fallbacks, load balancing and caching
- Budgets, guardrails and prompt management
Watch out for- Paid value centres on logs, retention and governance
- Free log allowance is small for production use
Starting priceSee website
Free planYes
HoneyHive focuses on agents that run for many steps, hand off work and take time to finish. Its trajectory view follows a full agent run, including retries, loops and handoffs, rather than showing only isolated model calls. Live scoring, drift detection and alerts help teams catch a quality drop before users report it, and flagged sessions can be routed to human reviewers for judgement.
A free Developer plan gives a starting event allowance for a pilot, while larger use is priced through an enterprise arrangement. That lack of a published self-serve price is a drawback for teams that want to budget without a sales call. The product fits best when evaluation quality, human review and long agent runs are central, and when the buyer can accept a custom pricing discussion in return for that depth.
What we like- Long-running agent traces and trajectory views
- Live scoring and alerts
- Dataset creation from production failures
Watch out for- Paid pricing is not published as a self-serve list price
- Smaller teams may need a sales conversation to size cost
#6
Weights and Biases Weave
Best for: Teams already using Weights and Biases
Visit wandb.ai
Starting price$60 per month
Free planYes
Weave is the LLM tracing and evaluation layer inside Weights and Biases, so it fits teams that already track experiments, models and datasets there. Developers can trace LLM calls, agent steps and retrieval calls, score outputs and compare runs in the same workspace used for model work. That shared context helps when a team moves from fine-tuning and experimentation into production LLM features.
The free plan covers personal use with limited storage and Weave ingestion, while Pro starts at 60 dollars per month for a block of seats with larger storage and ingestion allowances. Teams should check eligibility rules and seat limits, since Pro is aimed at smaller teams and larger organisations move to custom enterprise pricing. For teams with no prior Weights and Biases use, the broader ML platform may feel like more than needed for tracing alone.
What we like- Weave tracing inside a familiar ML platform
- Experiment tracking alongside LLM evals
- Strong visualisation and reporting
Watch out for- Pro eligibility and seat limits need checking
- Ingestion overages can add cost at scale
Starting priceSee website
Free planYes
Traceloop takes a standards first path. Its OpenLLMetry SDKs capture LLM traces using OpenTelemetry conventions, and the data can flow to Traceloop or to other observability backends a team already runs. That choice matters for organisations with an existing investment in tools for logs, metrics and traces, because LLM data can join the same incident workflow instead of living in a separate silo.
A free tier gives a starting span allowance for a pilot, while larger hosted use requires a sales discussion, so buyers should request a written quote based on real span volume. The product also covers quality checks such as faithfulness and relevance, plus custom evaluators for domain specific rules. Teams that want a standalone dashboard with the least setup may prefer a hosted-first rival, but standards focused teams often prefer Traceloop for portability.
What we like- OpenTelemetry standards and OpenLLMetry SDKs
- Broad provider and framework coverage
- Export to many observability backends
Watch out for- Hosted paid pricing is not published as a simple list
- Teams need an existing observability stack to get full value
Starting price$19 per month
Free planYes
Opik, built by Comet, combines tracing, evaluation and production monitoring with an open source core that teams can run themselves. Its free cloud tier gives a useful span allowance, and Pro Cloud starts at 19 dollars per month with a larger span allowance. Test Suites let teams write plain-text assertions for how an agent should behave, which gives a clear pass or fail result without building a full metric set first.
Prompt management, experiments and a playground help teams compare versions and test changes before release. Production monitoring adds scoring, alerts and cost views, so a regression found in live traffic can become a test case for the next release. Self-hosting gives full data control, but it also means running and updating the backend. For teams that want open source freedom with a low cost hosted start, Opik is a strong shortlist pick.
What we like- Fully open source core with self-hosting
- Test suites with plain-text assertions
- Prompt optimisation and playground tools
Watch out for- Some advanced enterprise controls sit on custom plans
- Self-hosting needs operational effort
How we chose
We compared vendor pricing pages, documentation and public reviews on G2, Capterra and Reddit, and checked each homepage to confirm the product exists and matches its name. We favoured tools that cover tracing, evaluation and production monitoring together, rather than tools that only log API calls. We also weighed pricing clarity, free plan usefulness, self-hosting options, framework coverage and support for agent workloads, since those factors decide whether a tool stays useful after the pilot.
What to check before you buy
Start with a real workload, not a demo prompt. Run one production-like agent for a week, count traces, spans and tokens, and compare that volume with each plan allowance. A tool that looks cheap at demo volume can become costly once every tool call, retry and retrieval step is traced. Set sampling for high volume paths, and keep full traces for failures and high value sessions.
Next, decide how you will judge quality. A trace alone does not tell you if an answer was good. Pick a small set of scorers for your use case, such as faithfulness for retrieval answers, task completion for agents and toxicity for user facing chat. Run those scorers on live traffic, review low scoring sessions with a human, and turn repeated failures into dataset cases. That loop matters more than any single dashboard chart.
Data, retention and export
Check where data is stored, how long it is kept and how you can export it. Short retention can hide a slow drift problem, while long retention raises cost and privacy risk. If you handle personal or regulated data, confirm redaction, access controls and region options before you instrument production. Ask for the data export format in writing, so a future move does not strand your history.
Cost controls and alerts
Set budget alerts, per-key limits and usage dashboards from day one. Agent loops and retry storms are the usual cause of surprise bills, so add alerts for error rate, latency, token spend and failed tool calls. Review cost per conversation, not just total spend, because a rising cost per conversation often signals a prompt or retrieval problem rather than simple growth.
Frequently asked questions
What is LLM monitoring?
LLM monitoring is the practice of tracing model calls, measuring latency, cost and errors, and scoring output quality in production. It helps teams find failures, compare prompt and model changes, and stop small quality drops from reaching many users.
Do I need a separate tool if I already log API calls?
Basic API logs show status and timing, but they rarely show the full agent path, retrieval steps, prompt versions or quality scores. A dedicated tool links those pieces into one trace, which makes debugging and evaluation far faster.
How much should a small team budget for LLM monitoring?
A small team can often start free, then move to a paid plan in the 19 to 60 dollar per month range for modest volume. Agent workloads cost more than simple chat, so measure span and trace volume during a pilot before choosing a plan.
Can I self-host an LLM monitoring tool?
Yes, Langfuse, Opik and Traceloop offer open source options that can run on your own infrastructure. Self-hosting gives data control, but it adds work for updates, backups, scaling and security.
What is the difference between tracing and evaluation?
Tracing records what happened during a run, step by step. Evaluation judges whether the result was good, using rules, code scorers or LLM judges. You need both, since a complete trace of a poor answer still needs a score to flag it.
Sources we checked (8)
- https://www.langchain.com/pricing
- https://langfuse.com/pricing
- https://arize.com/pricing
- https://portkey.ai/docs/product/product-feature-comparison
- https://www.honeyhive.ai/pricing
- https://wandb.ai/pricing
- https://www.traceloop.com/pricing
- https://www.comet.com/site/pricing/
Is your product missing or out of date?
We review this list regularly. Vendors can request a listing, send updated pricing or ask about featured placement.
Shihab Mia · Founder, Rankite
Shihab has 5+ years in B2B and SaaS SEO. He writes Rankite's software buying guides from pricing pages, documentation and public user reviews. More about Shihab.