Open engineering utility · pricing snapshot
LLM-as-a-Judge
Cost Calculator
LLM-as-a-judge uses a language model to score an AI system's output against a rubric such as correctness, groundedness, relevance, safety, or task completion. This calculator estimates the judge-model inference bill and the separate observability-platform bill from one normalized monthly workload.
Default answer: 10,000 application traces, 100% ingestion, 100% evaluation, one judge call, and one month of LangSmith Developer retention produce a $95.00 monthly estimate: $45.00 for Claude Sonnet 5 judge tokens, $25.00 for base-trace overage, and $25.00 to upgrade all traces beyond the 14-day base window.
- Coverage
- 5 vendors · 12 public plans
- Default judge
- Claude Sonnet 5 · $2 input / $10 output per MTok
- Default workload
- 10K traces · 1,500 input + 150 output tokens
- Billing dimensions
- Events · data volume · retention time
- Result states
- Exact · lower bound · not computable
- Data policy
- USD list price before tax · source-dated · no tracking
The normalization work belongs to custom software and AI product engineering because telemetry shape, evaluation coverage, retention, and model routing are architecture decisions before they are invoice lines.
- Runs in the browser
- No workload data leaves the page
- Shareable URL state
- JSON and CSV export
Interactive cost model
Calculate monthly LLM evaluation cost
Enter application traffic first, then separate telemetry ingestion from judge coverage. Observations and scores affect platform meters. Judge calls and tokens affect model inference.
01 / Workload
Define the workload
02 / Estimate
Monthly result
$9.50 per 1K application traces
- Ingested traces
- 10,000
- Evaluated traces
- 10,000
- Judge calls
- 10,000
- Judge model
- $45.00
- Plan fee
- $0.00
- Usage overage
- $25.00
- Data overage
- $0.00
- Extended retention
- $25.00
- Platform subtotal
- $50.00
5,000 base traces/month included. One month requests the 400-day extended tier because base retention is 14 days.
- Monthly USD estimate before tax.
- One decimal GB equals 1,000,000,000 bytes.
- Evaluation rate applies after sampling.
- One sampled application trace maps to one LangSmith base trace.
Transparent calculation chain
LLM-as-a-judge cost formula
The model and platform branches must be calculated separately. A platform may bill every stored event while the judge runs on a sampled subset. Collapsing application traffic, telemetry, scores, and model calls into one trace count hides the main cost drivers.
ingested traces = application traces × sampling rateevaluated traces = ingested traces × evaluation ratejudge calls = evaluated traces × judge calls per evaluated tracemodel cost = judge calls × ((input tokens × input $/MTok) + (output tokens × output $/MTok)) ÷ 1,000,000processed GB = ingested traces × bytes per trace ÷ 1,000,000,000GB-months = processed GB × retention monthsmonthly total = model cost + plan fee + usage overage + data overage + retention cost
Default unit economics
- Input cost per call
- 1,500 × $2 ÷ 1,000,000 = $0.0030
- Output cost per call
- 150 × $10 ÷ 1,000,000 = $0.0015
- Total judge cost per call
- $0.0045
- Judge cost per 1,000 calls
- $4.50
- Judge cost per 100,000 calls
- $450.00
- Judge cost per 1,000,000 calls
- $4,500.00
| Input change | Model-cost change | Platform effect | Interpretation |
|---|---|---|---|
| +100 input tokens per judge call | +$20.00 | None directly | Longer rubrics, references, or retrieved context scale input spend linearly. |
| +10 output tokens per judge call | +$10.00 | None directly | Verbose rationales can outweigh their small token count because output is priced higher. |
| A second judge call per evaluated trace | +$450.00 | May add another stored score | Multi-rubric and ensemble evaluation multiply model calls even when trace count is unchanged. |
| +1 observation per ingested trace | $0.00 | +1 Langfuse unit and +1 Arize span per trace | Instrumentation depth can increase platform cost without changing judge inference. |
| +1 score per evaluated trace | $0.00 unless it needs another judge call | +1 Langfuse unit and +1 Braintrust score | A stored score and an LLM call are related but not identical billing events. |
| +1 month of retention | $0.00 | Depends on fixed window or variable GB-month rate | Retention is always a coverage constraint, but only some vendors publish a variable charge. |
Compare the meter, not the logo
Three observability billing families
Event count
LangSmith meters base traces. Langfuse counts traces, observations, and scores. Arize counts spans. Braintrust counts stored scores. Event shape changes the invoice even at the same application traffic.
Data volume
Braintrust bills processed GB and Arize AX Pro publishes additional-GB pricing. Bytes per trace depend on prompt length, response length, attributes, metadata, and the number of recorded steps.
Volume × time
Confident AI meters GB-months. Braintrust Pro prices data beyond 30 days. LangSmith uses base and extended trace tiers. Langfuse and Arize publish fixed access windows per plan.
Normalized plan evidence
LLM observability pricing comparison
The table compares public list-price mechanics, not feature quality or vendor suitability. The modeled boundary states exactly when the calculator withholds a total instead of extrapolating an unpublished price.
| Vendor / plan | Base fee | Meter and included allowance | Retention | Published overage used | Modeled boundary | First-party source |
|---|---|---|---|---|---|---|
| LangSmith Developer | $0 · 1 seat | 5K base traces/month | 14-day base; 400-day extended | $0.005/additional base trace; $0.0025/extended upgrade | More than 1 seat or more than 400 days | LangSmith pricing |
| LangSmith Plus | $39/seat/month | 10K base traces/month | 14-day base; 400-day extended | $0.005/additional base trace; $0.0025/extended upgrade | More than 400 days | LangSmith pricing |
| Langfuse Hobby | $0 | 50K units/month; unit = trace + observation + score | 30 days data access | No public Hobby overage | Units above 50K or retention above 30 days | Langfuse pricing; unit definition |
| Langfuse Core | $29/month | 100K units/month | 90 days data access | $8, $7, $6.50, then $6 per 100K graduated bands | Retention above 90 days | Langfuse pricing |
| Langfuse Pro | $199/month | 100K units/month | 3 years data access | $8, $7, $6.50, then $6 per 100K graduated bands | Retention above 3 years | Langfuse pricing |
| Braintrust Starter | $0 | 10K scores + 1 GB processed/month | 14 days | $2.50/1K scores; $4/GB | Retention above 14 days | Braintrust pricing |
| Braintrust Pro | $249/month | 50K scores + 5 GB processed/month | 30 days included; up to 180 days | $1.50/1K scores; $3/GB; $0.50/GB/month after 30 days | Retention above 180 days | Braintrust pricing |
| Arize AX Free | $0 | 25K spans + 1 GB/month | 15 days | No public Free overage | Either allowance exceeded or retention above 15 days | Arize pricing |
| Arize AX Pro | $50/month | 50K spans + 10 GB/month | 30 days | $0.0008/additional span; $3/additional GB | Retention above 30 days | Arize pricing |
| Confident AI Free | $0 | 1 GB-month | Variable within GB-month allowance | Excess trace spans are dropped | Returns a lower bound above 1 GB-month | Confident AI pricing |
| Confident AI Starter | $200/month | 5 GB-months | Variable | $1/additional GB-month | Numeric list-price path remains computable | Confident AI pricing |
| Confident AI Team | $2,000/month | 75 GB-months | Variable | $1/additional GB-month | Numeric list-price path remains computable | Confident AI pricing |
Normalization assumptions: one sampled application trace maps to one LangSmith base trace. One observation maps to one Arize span. Langfuse partial 100K bands are prorated. These are calculator conversions, not vendor-authored workload guarantees.
Reproducible examples
Worked LLM evaluation cost scenarios
Each row names the workload details that change the answer. Totals use the same deterministic functions as the interactive calculator and exclude tax, discounts, and hosted evaluator token fees.
| Scenario | Normalized workload | Judge model | Platform | Total | State and decision meaning |
|---|---|---|---|---|---|
| Default LangSmith Developer | 10K app / 10K ingested / 10K evaluated / 1 month | $45.00 | $50.00: $25 base overage + $25 extended upgrade | $95.00 | Exact. All selected conversions are public. |
| Production sampling | 100K app × 25% sampled × 20% evaluated = 5K judge calls; 14-day retention | $22.50 | LangSmith Developer $100.00 base overage | $122.50 | Exact. Sampling reduces both telemetry and downstream judge calls. |
| Langfuse unit expansion | 1M ingested; 100K evaluated; 2 observations; 1 score = 3.1M units; 90 days | $450.00 | Core $29 + $219 graduated usage = $248.00 | $698.00 | Exact under proration assumption. Observations dominate unit count. |
| Braintrust score overage | 100K ingested and evaluated; 100K scores; 1 GB; 30 days | $450.00 | Pro $249 + $75 score overage = $324.00 | $774.00 | Exact. Included model credits are not deducted. |
| Arize AX Pro span overage | 30K ingested × 2 observations = 60K spans; 0.3 GB; 30 days | $135.00 | Pro $50 + $8 span overage = $58.00 | $193.00 | Exact. The public additional-span conversion is applied. |
| Confident AI Free coverage gap | 200K traces × 10 KB × 1 month = 2 GB-months | $900.00 | $0.00; only 1 GB-month retained | $900.00 lower bound | Lower bound. Zero fee does not mean full telemetry coverage. |
Operational use
How to use the estimate in an evaluation budget
Measure a real trace sample
Export a bounded sample from production or staging. Count application traces, observations, stored scores, prompt tokens, completion tokens, and serialized bytes. A measured median plus a high-percentile scenario is more useful than a generic average trace.
Separate ingestion from evaluation
Set sampling to the telemetry coverage you intend to retain. Set evaluation rate to the subset that receives an LLM judge. A team can ingest 100% for debugging and judge 5% for quality monitoring.
Represent every rubric call
If correctness, groundedness, and safety use separate prompts, enter three judge calls. If they produce three stored scores from one combined call, enter one judge call and three scores. This distinction changes model and platform cost differently.
Stress the retention boundary
Run the intended compliance or debugging window, not the vendor default. A plan may be inexpensive at 14 days and unusable at 90 days even when monthly event volume is unchanged.
| Architecture choice | Input to change | Expected cost effect | Check before accepting |
|---|---|---|---|
| Online judge on a production sample | Evaluation rate | Judge tokens and stored scores fall; ingested telemetry may not | Confirm the sampled set still covers rare failures and high-risk segments. |
| Multiple rubric prompts | Judge calls per evaluated trace | Token cost scales with call count | Check whether a combined rubric preserves calibration and diagnostic value. |
| Verbose chain-of-thought-like rationale | Output tokens | High sensitivity at output-token rate | Request only the explanation needed for audit or debugging. |
| Deep agent tracing | Observations and bytes per trace | More Langfuse units, Arize spans, and data volume | Do not record redundant payloads or secrets solely for observability. |
| Long incident or compliance retention | Retention months | Upgrade, variable storage, plan change, or not-computable state | Validate legal retention and deletion requirements outside this cost tool. |
| Self-hosted observability | Not represented | Cloud plan fee may disappear; infrastructure and operations appear | Build a separate TCO model for compute, storage, backups, upgrades, and on-call work. |
Scope control
What the calculator includes, excludes, and refuses to guess
Included
- Direct judge-model input and output tokens
- Judge-call multiplicity
- Trace sampling and post-sampling evaluation rate
- Plan and seat fees for listed self-serve tiers
- Published event, score, span, GB, and GB-month overages
- Published fixed retention windows and variable retention charges
- Known lower-bound and not-computable states
Excluded
- Taxes, currency conversion, discounts, startup credits, and negotiated contracts
- Prompt caching, batch API discounts, data residency multipliers, and tool-specific model fees
- Platform-hosted online-evaluator token charges
- Human annotation, rubric design, calibration, and adjudication labor
- Application inference outside the judge calls
- Self-hosted compute, storage, networking, backups, security, and maintenance
- Business value, model quality, latency, reliability, and vendor feature fit
Failure modes that can invalidate a cost estimate
| Failure mode | Likely bias | Why it happens | Corrective check |
|---|---|---|---|
| Treating one trace as one billable unit everywhere | Understates platform cost | A trace can contain many observations, spans, and scores. | Inspect each vendor's meter and count the matching stored objects. |
| Applying evaluation rate before telemetry sampling | Misstates both branches | The two percentages describe different gates. | Calculate ingested traces first, then evaluated traces. |
| Equating a stored score with a judge call | Can overstate or understate model cost | One call can emit several scores; repeated calls can emit one final score. | Count actual model invocations and stored score objects separately. |
| Using 10 KB as an invariant | Either direction | Trace payload varies with prompts, outputs, metadata, and instrumentation depth. | Measure serialized bytes from a representative workload sample. |
| Ignoring retention | Understates charge or overstates coverage | Some plans have short fixed windows or paid extended storage. | Run the required retention window and inspect the result state. |
| Subtracting platform credits from direct model cost | Understates model spend | Credits may cover only vendor-hosted features or built-in models. | Keep provider-billed tokens separate unless a contract proves the offset. |
| Reading free as unlimited | Overstates coverage | Free tiers can stop ingestion, drop excess spans, or omit overage prices. | Check both allowance and behavior after the allowance. |
| Treating a dated snapshot as a quote | Unknown | Vendors can change prices and plan terms after verification. | Reopen the linked first-party page before a purchase or renewal. |
Entity definitions
LLM evaluation and observability glossary
- Application trace
- One end-to-end application operation before the calculator applies telemetry sampling. Depending on the system, it may represent a chat turn, agent run, retrieval request, or workflow execution.
- Ingested trace
- An application trace sent to the observability platform after sampling. This calculator assumes one sampled application trace becomes one vendor trace where a trace-based meter is used.
- Evaluated trace
- An ingested trace selected for scoring after the evaluation rate is applied. It may receive zero, one, or several judge calls and stored scores.
- Observation
- A recorded step inside a trace, such as an LLM generation, tool call, retrieval, event, or span. Langfuse counts observations as units; the Arize conversion treats observations as spans.
- Score
- A stored quality result such as correctness, relevance, groundedness, safety, or task completion. A score may come from an LLM judge, code evaluator, human annotator, or user feedback.
- Judge call
- One invocation of the judge model. Multiple rubric prompts, retries, debate, or ensemble voting can create multiple calls for one evaluated trace.
- LLM-as-a-judge
- An evaluation method in which a language model applies explicit criteria to another model's output and returns a label, score, comparison, rationale, or structured verdict.
- MTok
- One million tokens, the unit used for model input and output pricing. Provider tokenizers can produce different counts for the same text.
- Processed GB
- Decimal gigabytes derived here from ingested traces multiplied by bytes per trace, divided by one billion. Vendor definitions can include or exclude different payload components.
- GB-month
- One gigabyte retained for one month. It combines data volume and time, so either larger traces or longer retention raises the measured quantity.
- Exact result
- Every selected component has a public numeric conversion, the workload stays within supported windows, and calculator assumptions are stated.
- Lower-bound result
- The known cost is calculable but full requested coverage is not delivered. The displayed amount is a floor, not a complete equivalent service.
- Not-computable result
- A necessary price is unpublished or a fixed retention boundary is exceeded. Known subtotals remain visible, while total and cost per 1,000 traces are withheld.
Focused answers
LLM-as-a-judge cost FAQ
How much does one LLM-as-a-judge evaluation cost?
With the default 1,500 input tokens, 150 output tokens, and Claude Sonnet 5 rates of $2 and $10 per million tokens, one judge call costs $0.0045 before platform charges. Change model rates and token shape for another model or rubric.
Are trace sampling and evaluation rate the same?
No. Sampling controls which application traces are ingested. Evaluation rate is applied after sampling and controls which ingested traces receive judge calls. Ingesting 100% and evaluating 5% preserves broad debugging coverage while limiting judge inference.
How should multiple judges or rubric dimensions be entered?
Count model invocations in the judge calls per evaluated trace field and persisted outputs in scores per evaluated trace. Three separate rubric prompts usually mean three calls and three scores. One combined prompt returning three dimensions means one call and three scores.
Why can a result be not computable?
The state appears when a required public conversion is absent or the requested retention exceeds a published fixed window. Known components remain visible, but the calculator does not invent the missing charge.
Does the calculator include hosted evaluator token fees?
No. The judge-model estimate assumes direct provider billing. Platform-hosted evaluator charges, plan credits, caching, batch discounts, taxes, and negotiated discounts are excluded. Enter a custom effective token rate only when the billing contract supports it.
Can this compare a self-hosted observability stack?
Not directly. Self-hosting replaces some SaaS fees with compute, object storage, databases, backups, upgrades, security controls, and engineering operations. Those costs require a separate total-cost-of-ownership model.
Ownership and evidence
Sources, authorship, and correction policy
Maintainer and publisher
Maintained by Dmytro Nasyrov, Founder & CTO of Pharos Production. Published by Pharos Production as an open engineering utility.
- Relevant credential
- PhD in Artificial Intelligence
- Version
- 1.1.0
- Published
- Pricing semantically verified
- Independent review
- Independent external review is not claimed.
AI-assisted disclosure
This page and its implementation were AI-assisted. Numeric pricing fields were checked against linked first-party pages, normalized into a public dataset, and exercised by deterministic tests. AI assistance is not a substitute for vendor confirmation or independent review.
Verification boundary
Source-level semantic verification and URL monitoring are different controls. The scheduled reachability check detects unreachable source URLs but does not verify pricing semantics. A person must review plan meaning before changing the registry date or numeric fields.
Correction and update policy
Report a price, formula, source, or accessibility defect through GitHub Issues. A correction must identify the affected claim, first-party source, effective date when known, code path, test impact, and changelog entry. Historical releases remain in Git.