Governed Decision Learning.
Lessons from building a self-improving AI decision engine: an architecture that turns operational experience into certified, explicit and revocable knowledge, the evaluation discipline we held it to, and what more than twenty pre-registered studies showed, including the results that went against us.
Agentic systems make thousands of small decisions: which tool to bind to a step, which model to try, what context to assemble, when to escalate. Each decision leaves a trace and, eventually, an outcome. The promise of "AI that learns from experience" is that those traces become better decisions. The risk is that they become unexamined behaviour: patterns nobody stated, validated on the same cases that suggested them, impossible to inspect and impossible to retire.
This paper reports our experience building a governed decision-learning engine that attaches to any agentic system and treats learned knowledge as explicit, scoped claims. Candidates are discovered, frozen, certified only on independent later evidence, offered at the decision point with their evidence level, monitored, and withdrawn or revised under human authority. We describe the eight design principles that resulted, the evaluation discipline we applied (hash-frozen pre-registration, single scoring, independent readers, sealed seeds, exact tests, simplest-baseline comparisons), and the results of seven study families on simulated worlds, software change history and real agent-platform traces.
The central finding is a split. Governed discovery produced exact, auditable and retirable knowledge with zero false discoveries where an ungoverned miner produced hundreds; an expressive parametric learner still recovered more decision reward. Real-world histories supported only coarse knowledge, unconstrained LLM hypothesis generation was clearly worse than statistics, and several evaluations ended, by design, as insufficient evidence. We report these results as they were registered and draw twelve lessons for teams building or buying AI that claims to improve itself.
AT A GLANCE
Six numbers from the programme
precision, recall and scope match on planted decision structure, in 30 of 30 runs
governed runs, 100 of them on worlds with nothing to find. The same miner without the gate: 58 to 204 per world
of decisions still taking a retired action after a reversal: governed engine against an expressive parametric learner
agent-platform decisions reconstructed, with two independently written readers agreeing on every one
of candidates from unconstrained LLM hypothesis generation were hallucinated or ungrounded
comparable cases a four-day usage window produced, against the 30 registered: reported as insufficient evidence
FIGURE 1
The governed knowledge lifecycle
Every piece of learned knowledge moves through the same seven stages. The stages that matter most are the ones most systems skip: freezing a candidate before its evidence exists, and certifying it only on cases it did not come from.
- 01 Observe
Decisions, alternatives and outcomes are recorded as dated facts. What a record does not say is stored as unknown, never guessed.
- 02 Propose
Miners and other proposers turn experience into candidate claims. A candidate is a hypothesis with a scope, not knowledge.
- 03 Freeze
A candidate is fixed before its evidence is gathered. Everything it was discovered from is excluded from its evidence.
- 04 Certify
Only later, independent cases can certify a claim, against floors that are named and versioned in advance.
- 05 Advise
Certified knowledge is offered at the decision point with its evidence level. Without settling support, the engine abstains and says what is missing.
- 06 Monitor
Independent and operational monitors watch every claim in use. An alarm opens a review; it never silently changes knowledge.
- 07 Withdraw or revise
People withdraw, keep or narrow a claim through a governed revocation. The record keeps who decided, when, and why.
Introduction & motivation
Every enterprise running agents in production eventually asks the same question: the system has made tens of thousands of decisions, and we know how many of them went well, so why is it not getting better? The usual answers are fine-tuning, a contextual bandit, or a retrieval layer over past runs. Each can move a metric. None, on its own, answers what a regulator, an auditor or a responsible engineering lead will ask next: what exactly has the system learned, on what evidence, and how do we make it stop?
We set out to build an engine that could answer those questions by construction. It attaches to an agentic system as a separate component, observes decisions and their outcomes, and is meant to learn two things: which decisions produce better outcomes, and which organisation of capabilities (agents, tools, models, enterprise systems and human decision points) should produce them. Our own agent platform is one host among several, not the container. Knowledge the engine learns is a first-class object with a scope, a standing and a provenance chain, rather than a shift in weights.
The harder part turned out to be honesty about results. Learning systems are easy to make look good: tune on the test set, compare against a weak baseline, count correlated runs as independent, read outcomes that were visible when the decision was made. We therefore treated the engine's development as a research programme first, with every evaluation pre-registered and frozen before a single outcome was read, and only then as a product.
What this paper is, and what it is not
It is an experience report. It describes the architecture we arrived at, the discipline we used, the results we registered and the lessons we draw from them. It is not a product announcement and it does not claim that the engine improves decisions in production; no study we report establishes that, and we say so where it matters. The engine's internal name is not used here: we describe it by what it does. Host identifiers, request texts and personal data from the studied systems do not appear here; every figure is a count or an aggregate.
Eight design principles
Each principle is tagged with its basis: demonstrated where a registered study showed it working, engineering where it is a design rule enforced in code and tests. We keep the two apart; an engineering rule that passes its tests demonstrates that the code meets its specification, not that the specification creates value.
Discovery is separated from certification
The component that finds a pattern never decides whether it is true. A candidate is frozen first, then certified only on cases it was not discovered from. In our planted-structure study this separation, not the miner, is what produced exact claims and zero false discoveries: remove the gate and the same miner buries the true pattern in hundreds of false ones.
Basis · Demonstrated on simulated worldsKnowledge is an explicit, scoped claim
Every unit of learned knowledge is a statement with a scope, an assertion type, a standing and a provenance chain. It can be read, queried, compared, superseded and withdrawn. A parametric learner holds the same structure as coefficients that nobody can inspect or retire.
Basis · Demonstrated on simulated worldsDescribing is not prescribing
A protocol certifies only what its evidence establishes. Observational and historical evidence can certify a descriptive claim ("in this scope, X tends to go with Y"), never an action claim ("do X"). The candidate ledger, the binding step and admission each enforce this separately.
Basis · EngineeringFour kinds of evidence, never merged
Discovery evidence; independent certification on cases the claim was not supplied to; post-application observation, tallied by exposure; and controlled intervention. A claim that was shown at a decision is never validated by what followed it: showing knowledge is an exposure, and exposed cases are kept out of that claim's evidence.
Basis · EngineeringTime is read as of the decision
Every fact carries the time it became known. A decision sees only what existed at that moment: what was known before the work, observed during it, inferred after the result, and what happened later are four separate classes. Nothing arrives in the past, and an update that did not finish is refused on open.
Basis · EngineeringThe core knows no domain
The learning, evidence and governance core contains no domain vocabulary. Domains bind through ports and adapters, and a deliberately unrelated reference domain runs as a conformance check. Architecture tests fail the build if domain terms leak into the core.
Basis · EngineeringA minimal immutable record, an erasable content store
The append-only log holds only the minimal authoritative material: identifiers, times, standings, counts. Names, free text and sensitive payloads live in a separate content store and are referenced by handle, so a person's data can be removed from managed storage without breaking the integrity of the record.
Basis · EngineeringPeople stay authoritative, numbers stay honest
The engine observes and advises; nothing it returns can authorise, verify or complete an action. Human decisions on knowledge go through the service and are recorded; a recovery never replays or invents them, it returns them to people to decide again. An observed proportion is reported as two integers, never as a probability or a confidence word that no calibration supports.
Basis · EngineeringWhat the engine is made of
The core is a deterministic, model-free library: an admission service that decides what enters the knowledge log; truth maintenance that marks claims stale when the facts they were declared against change; a candidate ledger with explicit standings (insufficient evidence, contradictory, descriptive, prescriptive, suspended, withdrawn and others); an evidence plane with its own statistics; a discovery plane that may never import the evidence plane; and a decision runtime behind a policy-engine port, with an open policy language as the default. Retrieval, embeddings and language models are optional plugins behind ports, never dependencies of the core. More than 700 automated tests cover it, including architecture tests for the package boundaries and for domain independence.
Outputs carry an explicit evidence level on every item, chosen from a fixed vocabulary: verified fact, declared rule, historical observation, learned hypothesis, certified knowledge, or unknown. When the evidence does not settle a choice, the engine abstains and names what is missing. That single rule removes most of the ways an advisory system can quietly overstate itself.
How we evaluated: ten disciplines
The methods below are standard in clinical and experimental science[1][2] and rare in applied machine learning, where analysis choices made after seeing data and underpowered comparisons are common[3]. We applied them to every registered study. More than twenty evaluations were frozen this way; each has a manifest of hashes and a verifier that must report the freeze intact before and after the run.
| Discipline | What it means in practice | Prevents |
|---|---|---|
| Pre-registration and hash freeze | Hypotheses, arms, metrics, sample sizes, analysis and stop rules are written before any outcome is seen; the plan, code and data manifests are pinned by hash and verified before and after every run. | Moving the goalposts after seeing results |
| Run once, score once | The registered analysis runs one time. Anything computed afterwards is labelled post-hoc and never presented as registered. | Forking paths and silent re-runs |
| Independent readers | Outcomes and decisions are read by two separately written programs that must agree before any analysis starts. | One parser's bug becoming a finding |
| Sealed confirmatory seeds | Evaluated instances derive from a preimage committed by hash and revealed only at evaluation, so no arm or tuning step can see them. | Tuning on the test set |
| The simplest baseline first | Every mechanism faces the simplest method that could do the same job: a plain counter, a flat registry with an exact scope filter, a refitting learner. A gain is credited only above it. | Crediting complexity with what simplicity achieves |
| Exact tests and multiplicity control | Exact permutation and sign-flip tests where sizes are small; Holm correction across the registered family of tests; bootstrap intervals beside every effect. | Overstated significance |
| Power before the window | How many comparable cases a test needs is computed before data collection, and a minimum is registered. Below it, no outcome is read. | Reading a null as "no effect" |
| Model-free dry runs first | Every pipeline runs end to end with no model before any paid model call, so data and plumbing limits surface cheaply. | Paying to discover a data problem |
| Provenance and dependence | Each run is classified by how it was started, and correlated runs are counted as one dependence group. | Counting one test script as many users |
| Amend in the open | A defect found after a freeze becomes a new, pinned amendment record that states what changed and which decision it affected. | Quiet fixes that rewrite history |
Fairness between arms
In comparative studies every arm receives the same task stream and the same proposal stream, so the governed arms never benefit from a better proposer. Mechanisms are kept separable along an ablation ladder, from no knowledge at all, through a flat registry, a flat registry with exact scope filtering, certified knowledge, and the full relational structure. A gain is attributed to a mechanism only above the rung below it. When a simpler arm matched a more complex one, we recorded it and did not tune the experiment until the complex one won.
Language that matches the evidence
A guardrail that does not fire means "no significant harm detected", never "no worse". A confirmed threshold is not a point estimate. A count of runs is not a count of independent observations. "Insufficient evidence" and "evidence of no effect" never share a name. These rules sound pedantic until a slide deck turns one into the other.
What the studies showed
Table 2 summarises seven study families. Each verdict is the registered one; post-hoc readings are labelled as such in the subsections that follow.
| Study | Question | Data | Registered verdict |
|---|---|---|---|
| A | Can a governed engine find, certify, use and withdraw decision knowledge nobody gave it? | Planted-structure simulator; 1,060 runs | Knowledge: decisive. Reward: no advantage |
| B | Can governed structural discovery beat strong refitting learners? | Two generated domains; 1,020 sealed runs | Met in one domain, falsified in the other |
| C | Does a strong LLM, generating hypotheses over code structure, find better knowledge than statistics? | Two open-source repositories; six-month window | Stop (registered) |
| D | What can be certified from software change history alone? | 13 public repositories, two large open-source projects | Coarse knowledge only |
| E | Can the engine read real agent decisions faithfully, and what do they hold? | 633 runs, 2,649 decisions; 4,194 plan steps | Reconstruction faithful; nothing certified |
| F | Does finer context add value above a tool's own track record on new usage? | Four-day frozen usage window | Insufficient evidence |
| G | Can an LLM judge stand in for expert grading of answer quality? | Pilot against expert grades | Parked |
4.1 Study A: knowledge and reward are different things
We built a simulator with planted decision structure: an exclusive-or pattern confined to two cells covering 12% of traffic. In the shifting worlds the pattern was later reversed or narrowed. Two further worlds contained nothing to find. Six arms ran over 1,060 pre-registered runs with no errors: a parametric learner that cannot represent the structure, an expressive parametric learner (ridge regression over interaction features), the governed engine, the same miner without the certification gate, an oracle given the true claims, and a random proposer under the gate.
On knowledge, the governed engine was decisive. It discovered the planted pattern in 30 of 30 and 29 of 30 runs and stated it exactly: scope match, precision and recall all 1.000. Across 180 runs, 100 of them on worlds with nothing to find, it made zero false discoveries and promoted none of the planted decoys. The same miner without the gate found the cells but described them wrongly (precision 0.445) and produced between 58 and 204 false claims per world. The random proposer under the gate promoted nothing in 180 runs, which is the point: the gate, not the miner, is what makes the knowledge correct.
On reward, it was not. The parametric learner recovered 0.84 of the achievable gain against the governed engine's 0.28, a significant difference (Holm-corrected p ≤ 0.001 in both worlds), and our registered no-go condition on decision performance was triggered. Differences in global success were within ±0.003 in every direction, because the planted effect was small against the cost of exploration and certification.
The reversal told the rest of the story. After the planted action was reversed, the parametric learner was still taking it in 58% of affected decisions at the end of the run. The governed engine had withdrawn the claim in 50 of 50 reversal runs, always inside three times its own certification cost, and took the retired action in 0.01% of decisions. After the reversal it also beat the arm that was handed the true claims (p = 0.0155), because that arm could withdraw knowledge but had no way to discover the revision. Revision was the weakest capability: 13 of 28 applicable runs reached the surviving sub-scope.
4.2 Study B: structural discovery, met once and falsified once
The second study asked whether the engine could grow its own relational structure (compositions, conditions, sequences, groups, exceptions) as an append-only log of typed, reversible operations, each with its own evidence contract, and beat strong learners that refit from scratch every round: a same-vocabulary refit, a temporal-logic rule learner[14], and a sparse regression[15]. Two generated domains with different mechanisms were used, a relational world and a process world, with planted compositions, exceptions, change points and new entities.
The design was the most demanding of the programme. Confirmatory instances (30 per domain, ten seeds in each of three families) were derived from a preimage committed by hash and revealed only at evaluation. Tuning ran on a separate family, and the selection was locked before any arm touched a sealed instance. The primary matrix (780 runs) and reuse runs (240) completed without errors. Five registered differences per domain were tested with an exact two-sided sign-flip test over all 230 flips, Holm-corrected across the five, with stratified bootstrap intervals.
In the process world, the governed structure beat the best refitting learner by +0.162 F1 (Holm p = 9.3 × 10−9), positive in all three families and above the registered +0.10 margin: met. In the relational world the gain was +0.040 F1, significant (p = 3.2 × 10−3) but below the margin: not met, and with a second criterion also failing there, the framework's hypothesis was falsified on that world as registered. Reusing learned structure on a new instance of the same family gave a relative gain of 0.594 [0.551, 0.637] and 0.895 [0.852, 0.941]: met in both domains. The stability criterion, which bounds how much adopted structure churns once the world is stable, was met in neither.
While reading the outputs we found a defect in our own analysis code: it compared the sum of three family means with a threshold registered per instance, three times the intended scale. P-values were unaffected. A second, pinned amendment re-read the frozen outputs on the registered scale; one verdict changed from met to not met, and the stricter reading is the one reported above. Generated worlds are not real-world evidence, and we do not present them as such.
4.3 Study C: language models as hypothesis generators
A natural idea is to let a strong language model read the structure of a system and propose the knowledge directly. We screened it on two contrasting open-source repositories, with a six-month evaluation window after a year of history. Three arms ran through identical, unchanged validation: statistics only; coarse structure units with the same statistics; and the same structure with hypothesis generation by a strong open-weight model pinned to an exact version and provider, with no fallback.
The registered decision was stop. Net later independent support for frozen candidates was 0.83 for statistics only, 0.46 for structure plus statistics and 0.11 for the language-model arm. Guardrails fired for both structure arms; in the language-model arm 44% of candidates were hallucinated or ungrounded[12]. Durable specific knowledge was zero in every arm. Under our pre-registered rule, a failed upper-bound screen means no smaller model and no scaling.
We read the result narrowly. It falsified unconstrained hypothesis generation over coarse structure with no statistical pre-filter and no evidence feedback. It did not test a model used as a constrained interpreter over statistics, structure and code semantics, nor cumulative refinement. "Semantic discovery failed" would be an overstatement, and we do not make it.
4.4 Study D: what change history can and cannot teach
Software change history is the largest body of decision-plus-outcome data most organisations own. We ran model-free learning over 13 public repositories from a live software-engineering benchmark, then over two large, active open-source projects. Only coarse knowledge certified reliably. Area-level descriptions (which directories a kind of task tends to involve) reached 96% of tasks; file-level knowledge reached 15%, and most of those lessons were inconclusive or later withdrawn. Adding the text of pull requests and issues, read as of when it was written, raised request-time knowledge only at the source-directory level (15% to 26% of tasks); file- and test-level request knowledge stayed between 0% and 0.7%.
The lesson is not that history is useless: coarse priors are cheap, safe and real. It is that specific knowledge needs experience recorded at the moment of the request, with an objective outcome, which history rarely contains.
4.5 Study E: real agent-platform decisions
A historical diagnostic over 4,194 usable tool-step episodes from stored plans of our agent platform found a failure rate of 3.2%, nearly all in one early cluster. Fourteen candidates were discovered; none was confirmed and none certified. With so little variation in outcomes and a deliberately limited representation, the result describes the limits of that combination, not whether planning decisions hold learnable structure.
The second evaluation read 633 runs and 2,649 decisions (721 tool bindings, 1,667 model attempts, 261 tool calls) from an on-premises deployment. The first attempt stopped at its own gate: seven of eight reconstruction requirements held, with the adapter and an independently written checker in complete agreement on every decision, alternative count and outcome, but for 88 decisions the outcome carried the same second as the decision and was visible at decision time. The registered result was "not evaluated: adapter limitation", and no outcome was tabulated. We corrected the temporal reading at source precision and ran a new, separately frozen evaluation.
The second run passed all eight requirements. References were retrievable for 687 of 707 evaluation decisions. For the tool bound to a step, a tool's own track record was nominally informative: a usable result in 96 of 100 cases where the record said the tool did better there, against 3 of 10 where it said worse. No association passed the unchanged discovery floors (0 of 529). For the same context, the engine's reference layer returns exactly the integers a plain counter returns, and we report that as equivalence. The data came from one deployment, mostly one caller and two-thirds from one week, and 206 of the 633 runs had been started by a program; they were counted in 431 dependence groups, never as independent observations.
4.6 Study F: a frozen usage window that came up short
To test whether finer context adds value above a tool's own track record on new usage, we froze a four-day prospective window, with dates, procedure, outcome rule and a single primary test fixed in advance. Power was computed beforehand: 24 comparable cases with varying outcomes give 0.60 power to detect a 45-point gap, and 48 give 0.91, so a credible test needed roughly 480 runs under the historical mix. The window produced 49 runs, 31 dependence groups, 20 eligible decisions and 9 comparable cases against the 30 registered. Under the frozen rule, no outcome of the window was read. The study is closed as insufficient evidence, which says nothing about the effect in either direction.
4.7 Study G: an LLM judge as the outcome
Many learning loops need an answer-quality signal, and an LLM judge is the cheapest one available[13]. We tested a fixed, low-cost judge at temperature zero, blinded to the producing model and judged twice for self-consistency, against expert grading on a three-item rubric. Its best weighted agreement was a kappa of about 0.47–0.48, not reliable enough to serve as an outcome a learner would optimise. We parked automated quality scoring and kept outcomes to what records establish.
Twelve lessons for teams building AI that learns
None of these is new to experimental science. All of them were expensive to learn in practice, and most are absent from how self-improving AI is usually built and sold.
Separate finding a pattern from believing it
The gate made the difference between exact claims and hundreds of false ones. Any system that learns from experience needs a step where candidates wait for evidence they did not come from.
Measure knowledge and reward separately
They disagree. A parametric learner recovered more of the available gain; the governed engine produced the only correct, auditable and retirable statements. One score would have hidden both facts.
Explicit knowledge can be retired; weights cannot
After a reversal, the parametric learner kept taking the retired action in 58% of affected decisions; the governed engine, 0.01%. In regulated settings the ability to stop is worth more than a few points of average reward.
The simple baseline is the real bar
For the same context, our reference layer returns the same integers as a plain counter, and we report that as equivalence, not as a win. Value has to show up above the simplest method or it is not value.
Constrain language models when they propose
Left to generate hypotheses freely, a strong model produced 44% hallucinated or ungrounded candidates and the lowest net support of any arm. The open question is a model as a constrained interpreter of evidence, which that study did not test.
History gives coarse knowledge, cheaply
Change history reliably supports area-level descriptions and almost nothing finer at the moment of a request. Specific knowledge needs live, outcome-grounded experience, which is scarcer than most roadmaps assume.
Time leaks are the default, not the exception
Our own reader gate stopped an evaluation because 88 outcomes carried the same second as their decisions and were visible at decision time. We reported it as an adapter limitation, fixed it under a new identity, and ran again.
Know where your traffic came from
Of 633 real runs, 206 had been started by a program. Automated test runs are useful for describing behaviour and worthless as independent observations. Counting them as users manufactures confidence.
Insufficient evidence is not evidence of no effect
A frozen four-day window produced 9 comparable cases against the 30 registered. We read no outcome and published the coverage. Extending the window or lowering the gate after the fact would have been easy and wrong.
Automated judges are proxies until proven
An LLM judge agreed with expert grading at a weighted kappa near 0.47. We parked automated quality scoring rather than let an unvalidated proxy become the outcome a learner optimises.
When you find your own bug, amend in the open
While reading one evaluation we found our analysis code compared a sum of three family means with a per-instance threshold, three times the registered scale. A pinned amendment re-read the outputs; one verdict changed from met to not met, and we report the stricter one.
Privacy is an architecture decision
Keeping text out of the immutable record and behind handles is what makes erasure possible without corrupting evidence. It is much harder to retrofit than to design in.
Nine questions to ask any vendor of self-improving AI
Frameworks such as the NIST AI Risk Management Framework[16], ISO/IEC 42001[17] and the EU AI Act[18] ask organisations to understand, monitor and control the behaviour of AI systems over their lifetime. A system that changes its own behaviour from experience makes that harder unless its learning is governed. These are the questions we would want answered before trusting one, including our own.
| Ask | Why it matters | A good answer looks like |
|---|---|---|
| What exactly is an outcome, and when is it known relative to the decision? | If outcomes are visible at decision time, the learner is reading the answer key. | A written outcome rule, timestamps at source precision, and an as-of read for every decision. |
| Which simpler method did the learned behaviour beat, and by how much? | Most "learning" gains disappear against a per-option counter or a static rule. | A named baseline, an effect size with an interval, and data the system never saw. |
| Was the evaluation plan fixed before the results were seen? | Analyses chosen after seeing data find something almost every time. | A dated, versioned plan, ideally hash-pinned, and a statement of what was post-hoc. |
| Can I inspect a learned rule, see its scope and withdraw it? | Behaviour you cannot see or stop is a liability under any AI governance framework. | Explicit claims with scope, evidence and standing, and a governed withdrawal path with an audit trail. |
| Can anything uncertified reach the decision path? | Candidate patterns are hypotheses; acting on them is an experiment on your operations. | A certification gate that is mandatory, with references shown separately from advice. |
| How do you stop the system from validating its own advice? | Cases where advice was shown are not independent evidence for that advice. | Exposure tracking, and exposed cases excluded from the evidence of the claims shown. |
| How much of the evaluation traffic came from scripts or test agents? | Automated runs inflate counts and are correlated by design. | Provenance per run and dependence groups counted once. |
| What happens to personal data in the learning record on an erasure request? | Learning stores quietly accumulate text, names and prompts. | Text and names kept out of the immutable record, referenced by handle, removable from managed storage. |
| What did you test that did not work? | A team with only wins has either not tested much or not reported it. | Specific negative results, and what they do and do not rule out. |
Limitations & what comes next
What these results do not establish
Studies A and B used generated worlds; their results describe the mechanisms on those worlds, not real-world learning. Studies D and E used public repositories and our own platform's deployments; they are single-organisation, largely single-caller data. No study in this paper shows that the engine improves decision outcomes in production, and we do not claim it. The certification floors are deliberately conservative, and some classes of knowledge cannot be certified within the evaluation horizon by construction: for example, a success rate of 0.6 against 0.5 within 150 observations. We kept the floors unchanged rather than lower them to produce findings; matching evidence requirements to claim types is future work.
Where the engine is going
The engine has moved from research into product engineering. It now runs as an advisory component behind a versioned, machine-readable contract, being integrated into a first host at observe-and-advise levels only: it can rank and explain options, flag declared rules and their staleness, and abstain, but never authorise, verify or complete an action. Raw retrieval of past experience stays a first-class baseline, because results like Study E's say it must be beaten, not assumed beaten. The next evaluations are on real tasks against simple retrieval and dependency baselines, under the same discipline described here.
Conclusion
"Self-improving AI" usually means a system whose behaviour drifts in a direction someone hopes is better. We set out to build the opposite: an engine whose every learned belief is explicit, scoped, certified on evidence it did not come from, offered with its evidence level, and withdrawable by a person. The architecture works as designed where we could test it; it produced knowledge an ungoverned miner and a parametric learner could not, and it let go of knowledge when the world changed.
The research also told us, plainly and more than once, where the value was not: in reward against a strong learner, in unconstrained model-generated hypotheses, in fine-grained knowledge from history, in small windows of usage. We think those results are as useful to an enterprise AI programme as the positive ones, and that the discipline that produced them (fix the plan, score once, beat the simplest baseline, report what failed) is the part most worth copying.
References
- [1] Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences 115(11), 2600–2606.
- [2] Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine 2(8), e124.
- [3] Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14, 365–376.
- [4] Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6(2), 65–70.
- [5] Good, P. I. (2005). Permutation, Parametric, and Bootstrap Tests of Hypotheses (3rd ed.). Springer.
- [6] Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in data mining: formulation, detection, and avoidance. ACM Transactions on Knowledge Discovery from Data 6(4).
- [7] Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. NeurIPS.
- [8] Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys 46(4).
- [9] Doyle, J. (1979). A truth maintenance system. Artificial Intelligence 12(3), 231–272.
- [10] Agrawal, R., Imieliński, T., & Swami, A. (1993). Mining association rules between sets of items in large databases. ACM SIGMOD.
- [11] Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence 1, 206–215.
- [12] Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys 55(12).
- [13] Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks.
- [14] Liu, Y., Ma, Y., Hildebrandt, M., Joblin, M., & Tresp, V. (2022). TLogic: Temporal logical rules for explainable link forecasting on temporal knowledge graphs. AAAI.
- [15] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B 58(1), 267–288.
- [16] National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1.
- [17] ISO/IEC 42001:2023. Information technology — Artificial intelligence — Management system.
- [18] Regulation (EU) 2024/1689 of the European Parliament and of the Council (Artificial Intelligence Act). Official Journal of the European Union, 12 July 2024.
- [19] Regulation (EU) 2016/679 (General Data Protection Regulation), Article 17: right to erasure.
Download the full white paper
Enter your work email and name and we'll send a download link for the print-optimised PDF edition, for offline reading, internal review and citation.