Decide what to watch, when to investigate and what can safely roll back.
Use this worksheet to turn the production AI scope of Pharos Production into a monitoring plan your team can own. It runs in your browser. Your inputs stay in this tab.
Build your monitoring plan
01 / CONFIGURE
YOUR PLANDRAFT · CALIBRATION REQUIRED
RAG application
Set thresholds below, then export the current configuration.
Signals included
Requests / window
Evaluations / window
REFERENCE 14 days→COMPARE 24 hours→INVESTIGATE 2 windows
These are traffic estimates, not observed eligible samples. A restricted denominator, incomplete telemetry or delayed labels can make the usable count lower. Grounding counts claims. Its request estimate is only a sampling budget.
Exports contain definitions, thresholds, source IDs and response rules.
No live telemetry connection. Alerts and rollback actions remain with your team. Reloading clears edits.
Signals and response rules
02 / TUNE
Thresholds below are starting examples, not recommended production limits. Every definition states what belongs in the denominator. Event checks have their own response timing.
Unreviewed releases and unauthorized effects use a fixed zero-tolerance trigger. Other thresholds are editable.
Request telemetry
Input distribution shift
Total variation distance = 0.5 × sum of absolute differences in category proportions between current and reference prompts. Freeze the same intent/language taxonomy and include an unknown category. Denominator: prompts with a recorded category.
Example setting. Calibrate before use.
Rationale, response and rollback
Why this signal
A changed request mix can invalidate an old evaluation set without proving a quality regression.
Investigate
Inspect affected intent and language slices. Compare downstream outcomes and expand the holdout set.
Rollback condition
Do not roll back for input shift alone. Consider a release rollback only if a matched-cohort evaluation links degradation to that release.
Evidence
S2 · S3. Definitions and thresholds are this planner's design choices.
Evaluation
Task success rate
100 × evaluated completed tasks that pass the fixed task rubric / all eligible completed tasks evaluated. Record unscored and pending labels separately.
Example setting. Calibrate before use.
Rationale, response and rollback
Why this signal
Business outcomes can change while input distributions remain stable.
Investigate
Re-score a matched sample, examine labels and group failures by release, task and language.
Rollback condition
Use the last compatible release only after a repeated quality breach is attributable to a change and the candidate passes the same eval set.
Evidence
S2 · S6. Definitions and thresholds are this planner's design choices.
Evaluation
Benign-request refusal rate
100 × refused requests judged answerable and allowed / evaluated requests judged answerable and allowed. Exclude unsafe prompts from this denominator and monitor unsafe acceptance separately.
Example setting. Calibrate before use.
Rationale, response and rollback
Why this signal
A rise may indicate a policy or prompt regression. An appropriate safety refusal is not a failure.
Investigate
Review benign refusals with a stable labeling rubric before changing guardrails.
Rollback condition
Revert an implicated policy or prompt change only after both benign-helpfulness and unsafe-request tests pass.
Evidence
S2 · S6. Definitions and thresholds are this planner's design choices.
Request telemetry
Schema conformance
100 × outputs passing the versioned required schema / requests that require that schema. Include missing, truncated and unparsable outputs as failures. Track endpoint errors separately.
Example setting. Calibrate before use.
Rationale, response and rollback
Why this signal
Successful HTTP responses can still break downstream consumers.
Investigate
Compare first-attempt and final outputs separately. Inspect schema and model versions.
Rollback condition
Revert the implicated model, prompt or schema change if the prior compatible bundle restores validation without violating current requirements.
Evidence
S1 · S6. Definitions and thresholds are this planner's design choices.
Request telemetry
End-to-end p95 latency
95th percentile of end-to-end completed request durations, measured from application receipt to terminal response. Record timeouts/cancellations separately. Do not average per-host percentiles.
Example setting. Calibrate before use.
Rationale, response and rollback
Why this signal
A quality-preserving release can still exceed the user-facing latency budget.
Investigate
Inspect queue time, retrieval, retries and model spans. Compare like-for-like workload slices.
Rollback condition
If a canary release causes an SLO breach, route to a compatible known-good path. Check quality and capacity before expanding traffic.
Evidence
S5. Definitions and thresholds are this planner's design choices.
Evaluation
Grounded-claim rate
100 × checkable claims supported by retrieved evidence / all checkable claims in evaluated answers. Judge the cited passage, not the mere presence of a citation. Track answers with no checkable claims separately.
Example setting. Calibrate before use.
Rationale, response and rollback
Why this signal
The fraction measures evidence support under a rubric, not universal factual accuracy.
Investigate
Inspect source validity, chunk boundaries and retrieval versions. Sample judge disagreements for human review.
Rollback condition
Revert the corpus, retriever or prompt only if a replay isolates that component and the restored corpus remains valid and authorized.
Evidence
S1 · S6. Definitions and thresholds are this planner's design choices.
Evaluation
Retrieval recall@k
Mean per-query percentage of labeled relevant documents found within the top k results, on the versioned retrieval evaluation set. Freeze k and relevance labels. Exclude queries without labeled relevant documents and report their count.
Example setting. Calibrate before use.
Rationale, response and rollback
Why this signal
A current corpus can still miss the document needed to answer the question.
Investigate
Separate missing source documents from retriever/reranker failures. Rerun against the previous index.
Rollback condition
Restore a compatible index or retriever only after the same relevance set improves and required document updates remain available.
Evidence
S1 · S6. Definitions and thresholds are this planner's design choices.
Release / event
Corpus update lag
Maximum elapsed time from an authoritative source revision to the matching revision becoming queryable. Include outstanding updates using current time. Measure deletion propagation separately.
Example setting. Calibrate before use.
Rationale, response and rollback
Why this signal
An ingestion backlog can make grounded answers obsolete even when the model has not changed.
Investigate
Inspect failed ingestion, change feeds, index jobs and outstanding deletions. Identify affected sources.
Rollback condition
Pause answers from affected sources or use a reviewed fallback. Never restore revoked, deleted or unauthorized documents to improve a metric.
Evidence
S1. Definitions and thresholds are this planner's design choices.
Release / event
Unreviewed prompt releases
Count of active prompt-template hashes that lack a successful regression evaluation and an approved release record. Prompt entropy here means uncontrolled template changes, not a Shannon entropy metric.
Example setting. Calibrate before use.
Rationale, response and rollback
Why this signal
Without a release record, changed behavior cannot be attributed or reproduced.
Investigate
Stop promotion and compare active hashes with the approved prompt registry.
Rollback condition
Restore the approved compatible prompt only after verifying tool permissions, model and schema compatibility. This event bypasses sample and persistence gates.
Evidence
S1. Definitions and thresholds are this planner's design choices.
Release / event
Provider sunset lead time
Days remaining until the provider-published retirement date for the exact deployed model snapshot. Unknown retirement dates are unknown, never zero or safe.
Example setting. Calibrate before use.
Rationale, response and rollback
Why this signal
A scheduled model retirement requires a tested migration path. It is a lifecycle event rather than a statistical drift score.
Investigate
Verify the provider notice and rehearse the replacement against the current eval set.
Rollback condition
Use a supported pre-evaluated fallback if the snapshot becomes unavailable. Do not roll back to a retired model. Check daily and on notices. No sample gate.
Evidence
S1. Definitions and thresholds are this planner's design choices.
Release / event
Unauthorized tool effects
Count of observed tool side effects outside the allowed action, resource or approval scope. Log denied attempts separately. This signal concerns effects that occurred.
Example setting. Calibrate before use.
Rationale, response and rollback
Why this signal
A single confirmed unauthorized effect requires incident handling regardless of traffic.
Investigate
Disable the affected write capability, preserve evidence and assess compensation for completed effects.
Rollback condition
Switch to a read-only or human-mediated path. Deployment rollback does not undo payments, messages or other external effects. Bypass sample and persistence gates.
Evidence
S1. Definitions and thresholds are this planner's design choices.
Calibrate before you alert
03 / VALIDATE
When model, prompt and retrieval changes share one release, a falling score gives little help with attribution. Pharos Production addresses that problem in its published production AI instrumentation method, which records versioned inference context and evaluation results. This planner makes the baseline, response owner and rollback condition explicit.
Freeze a representative reference
Record actual start and end timestamps, release IDs, traffic cohorts and exclusions when implementing the plan. The baseline-days field is a duration, not proof that a valid dataset exists. Include relevant weekly or seasonal patterns. A rolling baseline can normalize a slow regression, so preserve a fixed accepted release for comparison.
Measure outcomes alongside drift
Input drift is a distribution change. Concept drift changes the relationship between inputs and the desired answer. A stable input mix does not rule it out. Compare task success and user outcomes using consistent labels. Split by model version, task, language and other relevant cohorts so a global average does not hide a localized failure. [S2]
Backtest thresholds on your own history
Replay stable windows and known incidents using the same eligible populations. Choose the minimum sample count from the precision your decision needs. Examine confidence intervals and the cost of false alarms. Persistence alone is not a significance test. Compare non-overlapping windows where possible and account for repeated testing across many signals. A drift-method threshold is tied to that method and cannot be copied into a different distance or test. [S3]
Make missing data a separate condition
A zero denominator means no rate, not a passing rate. Track telemetry completeness, unscored items, dropped traces, pipeline failures and late labels. If a cohort has too little data, extend its window or increase the appropriate evaluation sample. Do not widen the window for an urgent safety event simply to reach the sample floor. [S3][S4]
Rehearse the response
Assign a real owner and a tested escalation route. Link each alert to the failing cohort, baseline, release and example traces. Diagnose whether the cause is ingestion, retrieval, a model change, the prompt or the evaluator itself. Retraining is one possible remedy. Repair a stale index or broken label pipeline first. Test the fallback in a canary and define recovery criteria before closing an incident.
Worked example: sampling is the bottleneck
For 1,000 requests per day, a 24-hour window and a 10% evaluation sample, the estimated budget is 100 evaluated requests per window. With a minimum of 100, there is no margin for ineligible requests or missing labels. At 200 requests per day, the same settings yield only 20 evaluations. The planner flags that gap. It does not infer that quality is acceptable.
A 14-day baseline under the first scenario would budget 1,400 evaluations if volume and sampling remained constant. It still needs a representative population, complete labels and a frozen rubric. Two consecutive 24-hour windows are a persistence policy for statistical signals. Critical events bypass it.
Keep telemetry attributable and limited
Correlate end-to-end application requests with model, retrieval and tool spans. Preserve model snapshot, prompt-template hash, corpus/index revision, evaluation-set version, judge/rubric version and trace ID. Model response time is one component of application latency. Retrieval, queues, retries and tool calls also contribute.
OpenTelemetry's GenAI conventions now live in a dedicated repository. Check the pinned source [S5] and each convention's stability before integrating an SDK. A tracing convention does not supply a grounding rubric, a validated threshold or an operational owner. Token usage can help explain workload shifts and cost, but a token change alone does not establish quality drift.
Collect only the content needed for the monitoring purpose. Raw prompts, responses and retrieved passages can contain personal or confidential information. Apply access controls, redaction and retention limits. Hashes are not automatically anonymous. Keep high-cardinality trace IDs and content out of metric labels. This page collects none of that telemetry.
Choose a response based on evidence, not the name of the alert
Observed condition
First action
Rollback boundary
Input mix changes. Quality holds
Review new intent coverage
No rollback from a drift score alone
New release fails a matched eval
Isolate component and verify fallback
Restore a compatible accepted bundle
Corpus is stale or access is revoked
Repair ingestion or pause affected answers
Never reintroduce invalid documents
Unauthorized side effect occurs
Disable affected write capability
Code rollback cannot reverse external effects
Sources, scope and review
Source check: . Sources support the monitoring concepts. Metric formulas, sample estimates and example thresholds are transparent design choices for this worksheet. S1 is the contextual production engineering link above; S7 identifies the publisher. Neither is independent evidence for a business result.
Codex, automated source, code, calculation and browser checks for version 1.0.0. No independent human validation or production calibration is claimed.
Methodology
Primary-document review, explicit metric definitions and deterministic plan generation. No customer telemetry, empirical benchmark or deployed monitor was used.
AI disclosure
AI-assisted research, writing and implementation.
Scope limits
Planning software for chat, RAG and tool-using agents. It does not assess legal compliance or certify safety. Tool agents with retrieval should also include the RAG signals in their implementation.
Data handling
No analytics, cookies, browser storage or external API calls. GitHub Pages handles ordinary hosting requests under its own policies. Downloads are created locally. Exported owner and release fields contain what you entered.
License and corrections
Original code and planner content are MIT licensed. Linked source material retains its owners' rights. Report a formula or source error through repository issues. Do not include confidential traces.