Same question.
Same decision.
A record you can audit.
Many teams use RAG — retrieve company documents, then ask an AI to answer. That is fine for reading help. It is the wrong tool when the outcome must be stable, explainable, and free of agent over-action bias. The Decision Flywheel is built for that job.
Flip rate, intervention bias, and latency from the EAV-DT study arXiv:2606.29280 M · drift adaptation from the Living Decision Flywheel evidence M
What RAG is — and where it falls short for decisions.
- Your question is matched to snippets from company documents.
- Those snippets are pasted into a prompt for a language model.
- The model writes an answer in natural language.
Useful when people need help reading a knowledge base. Fragile when the output is a business decision that must be the same tomorrow as today.
The answer changes every time
Ask the same internal AI the same question twice and you often get a different answer. That is how probability models work — they sample words. For policy, credit, compliance, or student support, that is not a process you can run a business on.
No clear audit trail
A RAG system may have looked at a document, but the answer rarely records which policy, clause, or record drove the outcome. When someone asks “why did we do that?”, you are left reconstructing a chat — not reading a decision record.
Agent bias — measured in study
LLM advisory agents tend to over-act: they recommend intervention when the correct institutional choice is often to do nothing. That bias wastes staff time, erodes trust, and standard “AI judge” scores miss it because fluent wrong advice still looks good on paper.
AI agents over-intervene — RAG does not fix it.
On a public education dataset, when the correct institutional policy said do nothing for about seven in ten students, a zero-shot GPT-4o agent still recommended action for about 73% — a 43 percentage-point false-positive over-intervention rate. Commercial RAG and SQL-augmented retrieval showed comparable miscalibration. M
Unnecessary contacts per cycle
At a 10,000-student scale, that over-intervention pattern implies thousands of avoidable advisor contacts every cycle — alert fatigue and wasted professional time.
Decision flip rate (DT)
The Decision Transformer path produces the same action for the same state — an architectural guarantee, not a temperature tweak. Measured at 0% flip rate.
Decision on CPU
In-process decision latency under 5 ms on a commodity CPU — about 455× faster than the zero-shot LLM arm and about 2,500× faster than the SQL-RAG pipeline in the study.
The paper’s strongest claim is the intervention-bias contrast — supervised Decision Transformer and snapshot policies eliminate that over-prescription; zero-shot LLM and RAG-style arms do not. Absolute accuracies are task-specific; we lead with the bias finding. Read the paper
Fairness that you can check
On the same study setting (OULAD), the Decision Transformer policy showed no significant disparity by disability or socioeconomic deprivation — auditable through lineage. Zero-shot LLM advisors on the same task showed severe bias that standard safety tooling missed. M
Policies that keep learning — without a quarterly re-prompt.
RAG systems go stale until someone re-indexes and re-prompts. The Decision Flywheel continually retrains the policy on fresh operational outcomes. On a real 251,000-case procurement log (BPI-2019), the continually retrained flywheel beat a frozen policy by +0.06 AUC (signed-rank p≈0.009, positive across every seed). A label-permutation placebo collapsed the gain about 389× — the value is freshness, not volume. M
Business rules as living data
Goals, thresholds, and policies sit in a governed store with the operational state. Raise a risk threshold and downstream decisions update with a full lineage — measured hundreds of times cheaper than rebuilding everything from scratch.
The decision is not a chat
A language model may help explain the outcome in plain English. It does not own the decision. Same complete state → same action — every time — with a record of what drove it.
Reproduction harness: github.com/Verificate-Dev · flywheel run all-local
From “retrieve and hope” to a Decision Flywheel.
| Typical RAG + LLM | Decision Flywheel | |
|---|---|---|
| What happens to a question | Find document chunks → ask the model to write | Project full business state → deterministic decision |
| Same input tomorrow | Often a different answer | Same action — 0% flip rate |
| Audit trail | Chat log; weak link to policy | Per-decision lineage — what attributes drove it |
| Agent bias | Over-intervention measured in study | Supervised policy calibrated to institutional rules |
| Who decides | The language model | The Decision Transformer — LLM may narrate only |
| Where it runs | Often a public API | Your environment — sovereign by design |
HELIX can still serve chat and document Q&A with a confidence score. When the outcome is an operational decision, the Decision Flywheel owns it — so business schema and policy stay separate from a probability model.
Replace RAG where decisions matter.
Book a call for academic or commercial access — or read the architecture and every measured result in the technical deep dive.