How we make this list: we compare pricing pages, product documentation and public user reviews. Ratings come from G2, Capterra or Trustpilot, with the date we checked them.
What is LLM evaluation software
LLM evaluation software tests whether a large language model application gives good answers. It runs prompts against a dataset, scores the outputs against expected answers or against a judge model, and shows where the application fails: which prompts hallucinate, which ones break the format, and which ones drift after a prompt change. The evaluation tool is where test datasets live, scores are computed, and regressions are caught before they reach users.
Direct answer: LangSmith is the top pick for most teams because it has a free Developer tier, a Plus plan at $39 per seat a month, and the deepest integration with LangChain. Langfuse is the best pick for a team that wants a free self-hosted option, with a Cloud Hobby tier free for 50,000 units and Core at $29 a month. Braintrust is the best pick for a team that wants evaluation and dataset management in one place, with Starter free and Pro at $249 a month. Arize Phoenix, Ragas, DeepEval and Promptfoo are the best picks for a team that wants free open source evaluation. Galileo is the best pick for an enterprise that already runs Splunk, from $100 a month. Start with LangSmith for a managed tool, or Langfuse for free self-hosting.
Key takeaways
- LangSmith Developer is free for 1 seat and 5,000 traces a month, and Plus costs $39 per seat a month for 10,000 traces.
- Langfuse Cloud Hobby is free for 50,000 units, Core costs $29 a month, Pro costs $199 a month, and Enterprise costs $2,499 a month.
- Braintrust Starter is free for 1 GB and 10,000 scores, and Pro costs $249 a month for 5 GB and 50,000 scores.
- Galileo is now Splunk Agent Observability after the Cisco acquisition in August 2026, from $100 a month per 1.2 million spans a year billed annually.
- Arize Phoenix, Ragas, DeepEval and Promptfoo are free and open source, with no licence cost.
How we picked
We compared eight tools on five points: how evaluations are defined and run, the scoring and judge models available, tracing and observability of live runs, the cost model, and how easy it is to leave. A tool scored well when a team could run an evaluation in minutes, see exactly which prompts failed, and compare scores across prompt versions. Prices come from vendor pricing pages and vendor supplied directory listings, checked 21 November 2026. Where a vendor was acquired or a product was renamed, that is flagged in the entry. Prices are search verified; vendor homepages were not individually opened in the browser for this guide. This guide covers tools that evaluate LLM outputs, distinct from the live best llm observability tools page, which covers tracing of live applications.
Comparison table
| Tool | Best for | Starting price | Free plan |
| LangSmith | Managed evaluation with LangChain | Developer free; Plus $39/seat/mo | Yes, Developer |
| Langfuse | Free self-hosting, open source | Hobby free; Core $29/mo | Yes, Hobby |
| Braintrust | Evaluation plus datasets | Starter free; Pro $249/mo | Yes, Starter |
| Galileo | Enterprise observability, now Splunk | From $100/mo | Free trial |
| Arize Phoenix | Open source tracing and eval | Free (open source) | Yes, self-hosted |
| Ragas | Open source RAG evaluation | Free (open source) | Yes, self-hosted |
| DeepEval | Open source unit testing for LLMs | Free (open source) | Yes, self-hosted |
| Promptfoo | Open source prompt testing | Free (open source) | Yes, self-hosted |
What a team of 5 pays to evaluate an LLM app
A team of 5 that evaluates an LLM application every week pays very different amounts across this list. On LangSmith Plus at $39 per seat a month, 5 seats cost $195 a month, with 10,000 traces per seat, plus $1.50 per LCU and $1.00 per LSU. On Langfuse Pro at $199 a month, the same team pays $199 a month flat, with overage at $8 per 100,000 units, and Core at $29 a month covers a smaller team. On Braintrust Pro at $249 a month, the team pays $249 a month flat for 5 GB and 50,000 scores, and Starter is free for 1 GB and 10,000 scores. Galileo, now Splunk Agent Observability, starts at $100 a month per 1.2 million spans a year billed annually. Arize Phoenix, Ragas, DeepEval and Promptfoo cost $0, and the team pays only for its own compute.
What is the difference between LLM evaluation and LLM observability
LLM evaluation and LLM observability are two parts of the same job, and most tools on this list do both. Evaluation tests the application before it ships: it runs prompts against a dataset, scores the outputs, and compares versions to catch regressions. Observability watches the application after it ships: it traces live requests, logs the prompts and outputs, and alerts the team when latency, cost or error rates rise. Evaluation answers whether the new prompt is better. Observability answers whether the live application is working right now. The tools on this list combine them: LangSmith, Langfuse, Braintrust and Galileo all trace live runs and run evaluations on the same data.
A small team should start with the tools that are free to try and cheap to keep. Langfuse is the best pick for a small team that wants a managed option, with a Hobby tier free for 50,000 units and Core at $29 a month. LangSmith is the best pick for a small team that builds with LangChain, with a Developer tier free for 1 seat and 5,000 traces a month, and Plus at $39 per seat a month. Braintrust is the best pick for a small team that wants datasets and evaluation in one place, with Starter free for 1 GB and 10,000 scores. Arize Phoenix, Ragas, DeepEval and Promptfoo are the best picks for free open source evaluation. Galileo is the best pick for a small team inside a Splunk shop, from $100 a month.
How to choose LLM evaluation software
Choose by the framework the team builds with, the budget, and whether the team wants a managed cloud or a self-hosted tool. If the team builds with LangChain, pick LangSmith, with Developer free and Plus at $39 per seat a month. If the team wants a free self-hosted tool with a cheap managed option, pick Langfuse, with self-hosting free under MIT and Core at $29 a month. If the team wants datasets, evaluation and prompt management in one place, pick Braintrust, with Starter free and Pro at $249 a month. If the team runs Splunk, pick Galileo, now Splunk Agent Observability, from $100 a month. For free open source evaluation, pick Arize Phoenix for tracing, Ragas for RAG, DeepEval for unit testing, or Promptfoo for prompt testing. Run an evaluation on a real dataset before committing: a tool that cannot show which prompts fail is not an evaluation tool.
Quick comparison
| Tool | Best for | Starting price | Free plan |
|---|
LangSmith | Managed evaluation with LangChain | Developer free; Plus $39/seat/mo | Yes, Developer tier free for 1 seat and 5,000 traces a month |
Langfuse | Free self-hosting, open source | Hobby free; Core $29/mo | Yes, Hobby tier free for 50,000 units; self-hosted free under MIT |
Braintrust | Evaluation plus datasets | Starter free; Pro $249/mo | Yes, Starter free for 1 GB and 10,000 scores |
Galileo | Enterprise observability, now Splunk | From $100/mo | Free trial available |
Arize Phoenix | Open source tracing and eval | Free (open source) | Yes, free and open source, self-hosted |
Ragas | Open source RAG evaluation | Free (open source) | Yes, free and open source |
DeepEval | Open source unit testing for LLMs | Free (open source) | Yes, free and open source |
Promptfoo | Open source prompt testing | Free (open source) | Yes, free and open source |
Is your product missing or out of date?
We review this list regularly. Vendors can request a listing, send updated pricing or ask about featured placement.
Request a listing
These eight tools cover the main ways teams evaluate LLM applications in 2026, from free open source libraries to enterprise observability platforms.
Starting priceDeveloper free; Plus $39/seat/mo
Free planYes, Developer tier free for 1 seat and 5,000 traces a month
LangSmith is the top pick for a team that builds with LangChain. The Developer tier is free for 1 seat and 5,000 traces a month, which is enough to test the tracing and evaluation on a real project. Plus costs $39 per seat a month for 10,000 traces per seat, with usage charges of $1.50 per LCU and $1.00 per LSU on top. Enterprise is custom, with single sign on, audit logs and a dedicated success manager. A team of 10 pays $390 a month before usage, so a large team should compare the flat priced options below.
What we like- Free Developer tier, deepest LangChain integration, strong tracing and dataset management, Plus at $39 per seat a month is clear per seat pricing
Watch out for- Plus is per seat so the bill grows with the team, usage charges of $1.50 per LCU and $1.00 per LSU add up, Enterprise is custom, best for LangChain users
Starting priceHobby free; Core $29/mo
Free planYes, Hobby tier free for 50,000 units; self-hosted free under MIT
Langfuse is the best pick for a team that wants a free self-hosted evaluation tool with a cheap managed option. The self-hosted edition is free under MIT, and the Cloud Hobby tier is free for 50,000 units. Core costs $29 a month, Pro $199 a month, and Enterprise $2,499 a month, with overage at $8 per 100,000 units. The platform covers tracing, evaluation, datasets and prompt management, and it is open source, so the team can inspect the code and self-host without a vendor.
What we like- Free self-hosted edition under MIT, Hobby tier free for 50,000 units, Core at $29 a month, strong tracing and evaluation in one product
Watch out for- Pro at $199 a month and Enterprise at $2,499 a month are big jumps, overage at $8 per 100,000 units adds up, ClickHouse acquired Langfuse in January 2026 so the roadmap may change
#3Braintrust
Best for: Evaluation plus datasets4.3/5 on
G2 (58 reviews, checked Oct 2026)
Visit braintrust.dev
Starting priceStarter free; Pro $249/mo
Free planYes, Starter free for 1 GB and 10,000 scores
Braintrust is the best pick for a team that wants datasets and evaluation in one place. Starter is free for 1 GB and 10,000 scores, which covers a small team testing one application. Pro costs $249 a month flat for 5 GB and 50,000 scores, which is simpler to budget than per seat pricing. Enterprise is custom, with single sign on, audit logs and private deployment. A team that outgrows Starter pays $249 a month, with no plan in between.
What we like- Free Starter tier, datasets and evaluation in one place, Pro at $249 a month is flat pricing, strong for teams that iterate on prompts
Watch out for- Pro at $249 a month is a jump from free, Enterprise is custom, the free tier caps at 1 GB and 10,000 scores, smaller community than LangSmith
Starting priceFrom $100/mo
Free planFree trial available
Galileo is the enterprise pick for LLM evaluation and observability, and it is now Splunk Agent Observability. Cisco acquired Galileo in August 2026, and the product was folded into Splunk, so the pricing page now redirects to Splunk and the brand name has changed. Pricing starts from $100 a month per 1.2 million spans a year billed annually. A team that does not run Splunk should weigh the change.
What we like- Strong evaluation and observability for agents, now part of Splunk with enterprise backing, from $100 a month per 1.2 million spans a year, good for teams that already run Splunk
Watch out for- Now Splunk Agent Observability after the Cisco acquisition in August 2026, pricing page redirects to Splunk, the brand change confuses older comparisons, enterprise focus
Starting priceFree (open source)
Free planYes, free and open source, self-hosted
Arize Phoenix is the open source pick for tracing and evaluation. It is free, and it traces LLM calls, tool calls and retrievals, then evaluates the outputs against the datasets the team defines. It works with any framework, and it is strong for RAG and agent debugging. The paid Arize platform adds managed features, governance and support for enterprises. For a team that wants free evaluation with no vendor, Phoenix is the fastest start.
What we like- Free open source tracing and evaluation, strong for RAG and agent debugging, works with any framework, no vendor lock in
Watch out for- Open source only, so no managed cloud, the team runs and maintains it, enterprise features depend on the paid Arize platform
Starting priceFree (open source)
Free planYes, free and open source
Ragas is the open source pick for RAG evaluation. It is free, and it scores retrieval augmented generation on the metrics that matter: faithfulness, answer relevancy, context precision and context recall. It works in Python, with no vendor, no account and no cost. Ragas evaluates RAG, not general prompt testing, so a team that needs broader evaluation should pair it with Promptfoo or DeepEval. For a RAG team, Ragas is the standard free tool.
What we like- Free open source evaluation built for RAG, metrics for faithfulness, answer relevancy and context precision, works in Python, no vendor lock in
Watch out for- RAG only, so no use for general prompt testing, the team runs it and there is no vendor support, the metrics need tuning for each dataset
Starting priceFree (open source)
Free planYes, free and open source
DeepEval is the open source pick for unit testing LLM applications. It is free, and it lets a team write tests for an LLM application like software tests: assert the answer is faithful, not toxic, and in the right format. The tests run in CI, so a prompt change that breaks an answer is caught before it ships. Confident AI sells the managed cloud, which is quoted. The metrics need calibration for each dataset, so validate them on your own data before trusting the scores.
What we like- Free open source unit testing for LLM applications, metrics for hallucination, toxicity and bias, works in Python with pytest style tests, Confident AI Cloud for teams that want managed
Watch out for- Open source only for the core, Confident AI Cloud is quoted, the team runs and maintains it, the metrics need tuning for each use case
Starting priceFree (open source)
Free planYes, free and open source
Promptfoo is the open source pick for prompt testing. It is free, and it runs a prompt against a dataset, compares the outputs side by side, and scores them against the assertions the team defines. Promptfoo tests prompts, it does not trace live runs, so a team that needs observability should pair it with Langfuse or Arize Phoenix. Enterprise is quoted for teams that want a managed version. For a team that wants free prompt testing, Promptfoo is the fastest start.
What we like- Free open source prompt testing, compares prompts side by side, works from the command line and in CI, no vendor lock in
Watch out for- Prompt testing only, so no tracing or observability, the team runs it and there is no vendor support, Enterprise is quoted
Frequently asked questions
What is the best LLM evaluation software
LangSmith is the best pick for most teams, with a free Developer tier and Plus at $39 per seat a month. Langfuse is the best pick for free self-hosting, with a Hobby tier free for 50,000 units and Core at $29 a month. Braintrust is the best pick for datasets and evaluation in one place, with Pro at $249 a month. Arize Phoenix, Ragas, DeepEval and Promptfoo are the free open source picks. Galileo is the best pick for a Splunk shop, from $100 a month.
How much does LLM evaluation software cost
LangSmith Plus costs $39 per seat a month, so a team of 5 pays $195 a month before usage. Langfuse Core costs $29 a month and Pro costs $199 a month, with overage at $8 per 100,000 units. Braintrust Pro costs $249 a month flat. Galileo starts from $100 a month per 1.2 million spans a year billed annually. Arize Phoenix, Ragas, DeepEval and Promptfoo are free, and the team pays only for compute.
What is the difference between LLM evaluation and LLM observability
Evaluation tests the application before it ships: it runs prompts against a dataset, scores the outputs, and compares versions to catch regressions. Observability watches the application after it ships: it traces live requests, logs the prompts and outputs, and alerts the team when latency, cost or error rates rise. Evaluation answers whether the new prompt is better. Observability answers whether the live application is working right now.
Do I need a paid plan or is the free tier enough
It depends on trace volume. LangSmith Developer is free for 1 seat and 5,000 traces a month, Langfuse Hobby is free for 50,000 units, and Braintrust Starter is free for 1 GB and 10,000 scores. A team that evaluates every week, runs many prompts, or has more than a few seats will outgrow the free tier and should budget $29 to $249 a month on the published prices. The open source tools are free at any volume.
Can I evaluate a RAG application with these tools
Yes. Ragas is built for RAG evaluation, with metrics for faithfulness, answer relevancy, context precision and context recall, and it is free and open source. Arize Phoenix traces retrievals and evaluates RAG outputs, and it is free and open source. LangSmith, Langfuse and Braintrust all support RAG evaluation with datasets and tracing, at their published prices. Pick Ragas or Phoenix for free RAG evaluation, or LangSmith, Langfuse or Braintrust for managed.
Sources we checked (8)
- https://www.langchain.com/langsmith
- https://langfuse.com
- https://www.braintrust.dev
- https://www.galileo.ai
- https://phoenix.arize.com
- https://docs.ragas.io
- https://deepeval.com
- https://promptfoo.dev
Is your product missing or out of date?
We review this list regularly. Vendors can request a listing, send updated pricing or ask about featured placement.
Request a listing
Shihab Mia · Founder, Rankite
Shihab has 5+ years in B2B and SaaS SEO. He writes Rankite's software buying guides from pricing pages, documentation and public user reviews. More about Shihab.