LLM Judge Cost Methodology

Scope and methodological principles

The calculator answers one bounded question: what monthly list-price cost follows from a stated LLM judge workload and a selected public observability plan? It does not rank vendors, predict model quality, estimate implementation labor, or replace a vendor quote.

1. Normalize workload before price. Application traffic, ingested telemetry, evaluated traces, judge calls, stored scores, observations, bytes, and retention remain separate quantities.
2. Price model inference and platform usage separately. A provider token bill and an observability invoice can respond to different workload gates.
3. Use first-party sources for price facts. Vendor pricing pages and official product documentation outrank summaries, search snippets, and third-party comparisons.
4. Do not interpolate an unpublished conversion. If a necessary overage or fixed-retention path has no public price, the total becomes not computable.
5. Expose assumptions as assumptions. Cross-vendor entity mappings and partial-band proration are calculator rules, not quoted vendor facts.
6. Keep arithmetic deterministic. The same inputs produce the same result in the UI, share URL, JSON export, CSV export, and automated test suite.
7. Date semantic verification. A URL returning HTTP 200 does not prove that its price, allowance, or meter definition is unchanged.

Two-branch calculation model

The workload splits after telemetry ingestion. One branch calculates judge-model tokens. The other converts stored telemetry into the selected vendor's billing meter. They rejoin only at the monthly total.

F01 · Ingested traces

I = A × (S ÷ 100)

F02 · Evaluated traces

E = I × (Q ÷ 100)

F03 · Judge calls

J = E × C

F04 · Judge-model cost

M = J × ((Tin × Pin) + (Tout × Pout)) ÷ 1,000,000

F05 · Processed decimal GB

G = I × B ÷ 1,000,000,000

F06 · Stored scores

R = E × K

F07 · Langfuse units

U = I + (I × O) + R

F08 · Arize spans

P = I × O

F09 · GB-months

H = G × L

F10 · Platform subtotal

V = plan fee + seat fee + event overage + data overage + retention cost

F11 · Monthly total

Total = M + V, only when result state permits a defensible total.

All monetary components retain six-decimal precision in the pure calculation function. The interface formats dollars to cents. Tier boundaries and quantities are calculated before display rounding.

Variable dictionary and entity boundaries

Vendor meter normalization rules

Vendor terms are preserved where possible. Cross-platform comparison still requires explicit conversion from the normalized workload to each billing entity. The Status column distinguishes first-party fact from calculator assumption.

Langfuse graduated-band calculation

Paid plans include 100,000 units. Additional units enter four progressive bands. A partial 100,000-unit block is prorated rather than rounded up. For 3.1 million units:

1. First 100,000 units: included.
2. Next 900,000 units at $8 per 100,000: 9 × $8 = $72.
3. Remaining 2.1 million units at $7 per 100,000: 21 × $7 = $147.
4. Total usage overage: $72 + $147 = $219.

The longer Pharos Production engineering analysis of LLM observability cost provides further architectural context. It is related editorial analysis, not independent evidence for the vendor prices recorded here.

Retention is both a cost variable and a coverage constraint

Retention changes the numeric charge only where a public variable-retention rate exists. It still constrains every platform. If requested retention exceeds a fixed published window, the estimate becomes not computable instead of remaining exact.

Exact, lower-bound, and not-computable decision tree

Exact

Every selected numeric component has a public conversion, the requested workload stays inside fixed plan constraints, and any cross-vendor assumption is declared. Exact describes arithmetic under the model, not a guaranteed invoice.

Lower bound

The known monetary charge is calculable, but the selected plan does not retain the full requested workload. Confident AI Free above 1 GB-month is the current lower-bound path.

Not computable

A necessary overage conversion is unpublished, a seat constraint is violated, or requested retention exceeds the selected plan's published window. Known subtotals remain visible. Total and cost per 1,000 application traces are withheld.

1. Validate all numeric inputs, percentages, seat count, and plan ID.
2. Derive ingested traces, evaluated traces, judge calls, scores, spans, processed GB, and GB-months.
3. Calculate judge-model cost independently from platform rules.
4. Apply plan fee, included allowances, published event/data overages, and retention logic.
5. If a hard constraint or missing conversion exists, set not_computable.
6. Otherwise, if the plan drops requested coverage, set lower_bound.
7. Otherwise set exact and expose total plus cost per 1,000 application traces.

Reproducible default worked case

The source HTML includes this default result before JavaScript runs. The application recalculates the same values on page load.

Application and ingested traces

10,000 × 100% = 10,000

Evaluated traces

10,000 × 100% = 10,000

Judge calls

10,000 × 1 = 10,000

Judge input cost

10,000 × 1,500 × $2 ÷ 1,000,000 = $30.00

Judge output cost

10,000 × 150 × $10 ÷ 1,000,000 = $15.00

Judge-model subtotal

$30.00 + $15.00 = $45.00

LangSmith Developer base overage

(10,000 - 5,000 included) × $0.005 = $25.00

Extended retention upgrade

One requested month exceeds 14-day base retention: 10,000 × $0.0025 = $25.00

Platform subtotal

$0 plan + $25.00 usage + $25.00 retention = $50.00

Monthly total

$45.00 + $50.00 = $95.00

Cost per 1,000 application traces

$95.00 ÷ 10 = $9.50

Change retention to 14 days, represented as 14 ÷ 30 = 0.4667 months, and the extended upgrade disappears. The same workload then totals $70.00.

Claim-to-source evidence ledger

The ledger enumerates the claims that carry the pricing model plus material derived rules. It does not copy long vendor passages. Each source-backed claim points to the first-party page whose current content was semantically checked on August 13, 2026.

The machine-readable counterpart is claims.json. Source metadata lives in sources.json. Pricing values live in models.json and vendors.json. Publishable copies are tested for full equality with repository-root data.

Reproducibility, URL state, and exports

The calculator is a static browser application. No server receives workload inputs. The pure calculation module accepts a single object, validates every numeric field and plan ID, and returns a structured result.

Deterministic checks

• Reference token cases at 10K, 100K, and 1M judge calls.
• Sampling before evaluation.
• Judge-call multiplicity and export round trips.
• Every modeled plan's allowance, overage, retention, and uncertainty state.
• Input validation at the pure boundary.
• URL serialization, JSON export, and CSV export.
• Root and published pricing-data mirror equality.
• Native semantic HTML, structured-data graph, contextual Pharos links, visible disclosures, robots, and sitemap contracts.

Run the project with Node 20 or newer: node --test. The separate source-reachability task checks a bounded official URL list. It does not modify pricing data or claim semantic freshness.

Editorial governance and update policy

Authorship

Maintained by Dmytro Nasyrov, Founder & CTO of Pharos Production. The repository history records each accepted change.

AI-assisted disclosure

Research, drafting, code generation, and quality checks were AI-assisted. Numeric fields were semantically checked against first-party sources and covered by deterministic tests. Independent external review is not claimed.

Correction and update policy

A correction identifies the claim ID, current first-party source, effective date if known, affected normalized fields, calculation behavior, test delta, and changelog entry. No verification date is advanced by a reachability check alone.

Source conflict policy

Prefer current first-party page content over cached snippets. If two live first-party sections disagree materially, mark the field unresolved and withhold the affected total until the conflict is resolved.

Pricing change policy

Update the root data, publishable mirror, calculation constants, tests, visible table, claim ledger, schema metadata, version, and changelog in one reviewed change.

Licensing

Source code is MIT licensed. Normalized data and editorial methodology are available under CC BY 4.0, subject to the original vendors' rights in their pages and brands.

Endorsement boundary

No listed vendor sponsors or endorses this calculator. A source link proves where a public fact was read, not product suitability.

Version history