Open method · Version 2026-08-18.1
Methodology and Source Ledger
This page makes the decision procedure inspectable. It defines what each answer means, which rule wins, why a result can be blocked, how advisories are attached, and where every material claim comes from.
Purpose and scope
Classify the next experiment, not the permanent architecture
In scope
The method applies to language-model workloads where a team is deciding whether the current gap is external knowledge access, specialized model behavior, both, or neither. It covers retrieval data, labeled examples, evaluation, provenance, lexical and semantic retrieval, reranking, fine-tuning, PEFT/LoRA, privacy topology, runtime measurement, and operations.
Out of scope
The result does not select a vendor, model, vector store, embedding model, reranker, hosting region, security control set, legal basis, budget, or acceptance threshold. It does not certify production readiness, factual accuracy, privacy, compliance, latency, cost, or business value.
Decision procedure
Validation, precedence, then advisory tests
The pure engine reads only versioned registries and the twelve answers. It contains no hidden weights or provider defaults.
Validate the record
Each registered criterion must have one answer: Yes, No, or Unknown. Unknown IDs, missing criteria, invalid values, malformed rules, and duplicate priorities fail with stable error codes.
Apply decision rules
Decision rules are evaluated in ascending, unique priority. The first matching rule selects one outcome. Hard dependencies and missing evidence therefore run before candidate rules.
Attach advisories
Advisory rules can add lexical retrieval, citation-faithfulness, runtime, privacy, or hybrid-lifecycle tests. They never change the winning outcome.
Export provenance
The record retains answers, outcome, winning and advisory rule IDs, blockers, rejected options, unresolved inputs, next steps, source IDs, claim IDs, and registry versions.
Input dictionary
What the twelve questions mean
A Yes answer should point to an artifact or operating capability. Unknown is not a soft No; it marks evidence that can change the result.
| Criterion ID | Evidence question | A defensible Yes requires | Why it changes the decision |
|---|---|---|---|
fresh_facts | Must facts change independently of a model release? | Named sources, update events, freshness owner, and tolerated staleness. | Changing knowledge favors an external update path rather than facts stored only in parameters. |
source_citations | Must users inspect passages behind material claims? | Claim-level evidence requirement, preserved source IDs, and a faithfulness review method. | Retrieval can carry provenance; tuning alone does not reconstruct it. |
behavior_change | Is the primary gap a stable task behavior? | Observable failure class and desired behavior for format, style, classification, extraction, or instruction following. | A trained adaptation needs a behavior target distinct from missing knowledge. |
representative_examples | Are permitted examples available? | Realistic inputs, desired outputs, failure cases, data rights, coverage, and a held-out split. | Fine-tuning without representative examples is not recommended. |
evaluation_set | Can alternatives be compared? | Versioned tasks, failure classes, baseline outputs, acceptance criteria, and review policy. | No candidate can win without a shared evaluation contract. |
authoritative_corpus | Does a retrieval source of truth exist? | Content owners, permissions, stable IDs, metadata, updates, effective dates, and deletion semantics. | RAG or hybrid is blocked when current or cited facts lack a governed corpus. |
exact_term_retrieval | Do exact identifiers or phrases matter? | Representative exact-term queries and known semantic-retrieval misses. | Adds a lexical/BM25, fusion, and reranking advisory test. |
strict_runtime_budget | Is there a measured latency or request-cost ceiling? | Defined traffic shape, quality target, percentile or unit, and test environment. | Adds an end-to-end benchmark; it does not bias toward either label. |
privacy_topology | Are data paths mapped? | Storage, ingestion, retrieval, training, inference, logging, retention, access, region, and deletion paths for chosen services. | Architecture labels alone do not establish privacy or retention behavior. |
retrieval_operations | Can the retrieval lifecycle be operated? | Owners and runbooks for ingestion, indexing, evaluation, ACL changes, stale-state detection, rollback, and deletion. | A knowledge requirement is blocked when no retrieval operating path exists. |
training_operations | Can the model lifecycle be operated? | Owners for datasets, provider/model availability, training, evaluation, releases, regressions, rollback, and deprecation. | A tuning or hybrid candidate needs an actual release capability. |
stable_narrow_task | Is the trained behavior maintainable? | A bounded task definition and evidence that desired behavior is not changing faster than the model release process. | Broad or moving targets are better tested with prompts and workflows first. |
Rule precedence
Why one result wins
Lower priority numbers run first. The order protects hard dependencies and evidence gates from being overridden by an attractive candidate pattern.
| Priority | Rule ID | When it matches | Outcome | Decision consequence |
|---|---|---|---|---|
| 10 | RULE-BLOCKED-NO-CORPUS | Fresh facts or citations are required; authoritative corpus is No. | Blocked | Reject RAG, hybrid, and tuning-as-knowledge-store. Establish the governed source first. |
| 20 | RULE-INSUFFICIENT-UNKNOWN-CORE | Any core requirement, data asset, eval, or operating answer is Unknown. | Insufficient evidence | Assign an evidence owner and resolve the unknown before choosing. |
| 30 | RULE-INSUFFICIENT-NO-EVAL | Evaluation set is No. | Insufficient evidence | Create representative tasks and acceptance criteria. |
| 40 | RULE-BASELINE-NO-EXAMPLES | Behavior change is Yes; representative examples are No. | Baseline first | Prompt and collect permitted success/failure examples before training. |
| 50 | RULE-BLOCKED-NO-RETRIEVAL-OPS | Knowledge or citations are required; corpus exists; retrieval operations are No. | Blocked | Assign the ingestion, evaluation, ACL, freshness, and rollback lifecycle. |
| 60 | RULE-BLOCKED-NO-TRAINING-OPS | Behavior and examples are Yes; training operations are No. | Blocked | Assign model-data, evaluation, provider, release, and rollback ownership. |
| 70 | RULE-HYBRID-CANDIDATE | Knowledge need and behavior need both exist; both assets and operating paths exist. | Hybrid candidate | Test retrieval and behavior adaptation separately before combining. |
| 80 | RULE-RAG-CANDIDATE | Knowledge or citations are required; corpus and retrieval operations exist. | RAG candidate | Build and evaluate a retrieval baseline with provenance and stale-state tests. |
| 90 | RULE-FINE-TUNING-CANDIDATE | Stable behavior, examples, and training operations exist; changing facts and citations are not required. | Fine-tuning candidate | Compare prompt-only and adapted models on task quality and regressions. |
| 100 | RULE-BASELINE-NO-PRIMARY-GAP | Changing facts, citations, and behavior change are all No. | Baseline first | Document a concrete failure before adding architecture. |
| 110 | RULE-INSUFFICIENT-FALLBACK | Validated answers do not match another candidate. | Insufficient evidence | Record the requirement or capability that prevents a match. |
Advisory rules
Exact-term retrieval
ADVISORY-LEXICAL-RETRIEVAL adds a BM25 or lexical path, signal fusion, and reranking comparison when codes, names, or exact phrases matter.
Claim-level citations
ADVISORY-CITATION-FAITHFULNESS requires each material generated claim to be checked against the cited passage.
Runtime budget
ADVISORY-RUNTIME-BENCHMARK requires end-to-end latency and cost measurement for the actual provider, corpus, traffic shape, and quality target.
Mapped privacy
ADVISORY-PRIVACY-TOPOLOGY preserves the verified data-flow map and requires provider controls to be rechecked before release.
Unresolved privacy
ADVISORY-PRIVACY-UNRESOLVED adds storage, retrieval, training, logging, retention, access, and deletion mapping.
Dual lifecycle
ADVISORY-HYBRID-OPERATIONS binds corpus, index, dataset, base model, adapter or tuned model, and evaluation suite as separate release inputs.
Claim ledger
Material statements and their boundaries
`VERIFIED` means the statement follows directly from the named source within the stated scope. `BOUNDED` means the rule is a conservative synthesis whose limits are explicit.
| Claim ID | Status | Bounded statement | Source IDs |
|---|---|---|---|
CLAIM-RAG-RUNTIME-RETRIEVAL | Verified | RAG supplies retrieved external information at inference time rather than encoding every changing fact in model parameters. | RAG paper; Microsoft |
CLAIM-RAG-FRESHNESS-FIT | Bounded | Retrieval is a suitable candidate when required knowledge changes independently and the corpus can be maintained. | Microsoft; practitioner guide |
CLAIM-RAG-PROVENANCE | Bounded | Retrieved passages can carry provenance; claim-to-passage support still needs testing. | RAG paper; Anthropic |
CLAIM-FINE-TUNING-BEHAVIOR | Verified | Fine-tuning is a path for task behavior, format, or specialization, not continuous fact synchronization. | Microsoft; OpenAI |
CLAIM-EXAMPLES-REQUIRED | Verified | A tuning decision needs representative examples and an evaluation set for the intended task. | OpenAI; Microsoft |
CLAIM-EVAL-FIRST | Bounded | Prompting, retrieval, and training changes should be compared against a defined baseline. | OpenAI; practitioner guide |
CLAIM-LEXICAL-SEMANTIC | Verified | Lexical and embedding retrieval recover different signals and can be combined when both matter. | Anthropic |
CLAIM-RERANKING | Verified | A reranker can reorder an initial candidate set before context reaches the model. | Anthropic |
CLAIM-CORPUS-QUALITY | Bounded | A RAG recommendation presumes an authoritative, permissioned corpus and an update path. | Microsoft; practitioner guide |
CLAIM-LORA-METHOD | Verified | LoRA freezes pretrained weights and learns low-rank update matrices. | LoRA paper |
CLAIM-HYBRID-PATTERN | Bounded | Retrieval and adaptation can be combined when evidence and behavior are independent needs. | RAFT; Microsoft |
CLAIM-RAFT-SCOPE | Verified | RAFT is one retrieval-aware tuning method using relevant and distractor documents in domain-specific RAG settings. | RAFT paper |
CLAIM-DUAL-OPERATIONS | Bounded | A hybrid creates retrieval-index and model-version lifecycle work. | RAFT; practitioner guide |
CLAIM-PRIVACY-TOPOLOGY | Bounded | Privacy and retention depend on provider, endpoint, storage, deployment, and configuration. | OpenAI data controls; Microsoft |
CLAIM-RUNTIME-MEASUREMENT | Bounded | Latency, quality, and cost tradeoffs must be measured on the target workload. | Microsoft; OpenAI |
Primary-source ledger
What each source is allowed to support
All eight URLs were checked on 18 August 2026. Research results and vendor statements remain scoped to their documented settings; they are not generalized into universal performance guarantees.
- SOURCE-PHAROS-GUIDE: practitioner framing only. Quantitative statements were not reused without primary support. The canonical URL is recorded in the source registry.
- SOURCE-RAG-PAPER: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Used for the RAG architecture definition and provenance-oriented framing, not universal production outcomes.
- SOURCE-MICROSOFT-RAG-VS-FT: Augment language models with RAG or fine-tuning. Used for qualitative selection factors; examples are not treated as thresholds.
- SOURCE-ANTHROPIC-CONTEXTUAL-RETRIEVAL: Introducing Contextual Retrieval. Used for lexical/semantic retrieval and reranking patterns; experimental outcomes stay provider-specific.
- SOURCE-OPENAI-MODEL-OPTIMIZATION: Model optimization. Used for an eval-first optimization loop and representative-example requirements; product availability can change.
- SOURCE-LORA-PAPER: LoRA: Low-Rank Adaptation of Large Language Models. Used for the method definition; benchmarks are not generalized.
- SOURCE-RAFT-PAPER: RAFT: Adapting Language Model to Domain Specific RAG. Used for one retrieval-aware tuning pattern, not a claim that hybrid always wins.
- SOURCE-OPENAI-DATA-CONTROLS: Data controls in the OpenAI platform. Used only to show that service and endpoint configuration determine treatment; it is not a universal privacy statement.
Repository data files preserve titles, publishers, type, publication or update date when available, verification status, checked date, URL, and use boundary. The reachability checker is non-semantic: an HTTP response does not prove a claim.
Limitations
Known boundaries and change risks
No universal numeric thresholds
The method contains no generic dataset-size, document-count, accuracy, latency, token, cost, compute-saving, or hybrid-uplift threshold. Teams must set workload-specific acceptance criteria before seeing candidate results.
No automatic provenance guarantee
RAG can preserve document and chunk IDs, but citation generation and claim support are separate system behaviors. A citation-present rate is not a faithfulness measure.
No architecture-level privacy guarantee
Corpus storage, embedding services, retrieval, prompts, inference, fine-tuning uploads, logs, evaluation systems, retention, regions, access, and deletion all affect the boundary.
Provider and model drift
Fine-tuning availability, supported models, endpoint policies, pricing, context limits, retrieval services, and data controls can change. Recheck the live provider contract for every release.
Qualitative rules
Yes, No, and Unknown expose evidence gaps but compress detail. Use the exported record as an architecture-conversation input, then attach the underlying artifacts and measurements.
Review status
Research synthesis and implementation were AI-assisted and deterministically checked against the registries. Independent human technical review is not claimed. The publisher remains responsible for corrections and future versions.
Version and change policy
Update history
| Date | Version | Change | Evidence action |
|---|---|---|---|
| 2026-08-18.1 / app 1.0.0 | Initial release: criteria, ordered rules, advisories, six outcomes, examples, exports, semantic article, open data, and original diagrams. | All source URLs checked; unsupported universal thresholds excluded; claim and rule references validated. |
Correction request
Email info@pharosproduction.com with the page URL, claim or rule ID, a primary contrary source, and the proposed correction. A material accepted change updates the registry version, visible page, changelog, tests, and source-check date.
Reproducibility
The dependency-free source, registries, tests, and workflow are available in the public GitHub repository. Canonical data and public data copies are checked byte for byte before deployment.