An AI proof of concept (POC) is a short, timeboxed test of whether an AI approach can do one specific task well enough, safely enough and cheaply enough to justify a pilot. This template gives you the POC charter, success criteria with numeric thresholds, an evaluation plan with a sized test set, a go/no-go rubric and a filled example, so the POC ends in a decision rather than a demo.
Key takeaways
- 01 Fix the success thresholds before anyone builds. A POC without numbers ends in an argument about impressions.
- 02 Test on representative, messy data, including the documents and requests that break things, not on a hand-picked clean sample.
- 03 Measure cost per task and latency next to quality and safety. A prototype can pass on accuracy and still fail on cost.
- 04 Size the test set on purpose: about 140 cases estimate a 90% success rate to within five points at 95% confidence.
- 05 Agree the exit options at the start: go to pilot, iterate once, change approach, or stop.
What an AI proof of concept is and who owns it
An AI proof of concept answers a narrow question: can this approach do this task, to this standard, on our data, at a cost and risk we would accept? It is not a demo, which shows that something can work once, and it is not a pilot, which tests whether real users benefit. A POC runs against a prepared test set, is graded by domain experts, and finishes on a fixed date with a decision.
| Role | Responsibility |
|---|---|
| Decision owner | A named executive who takes the go/no-go decision and holds the budget for the next stage |
| POC lead | Runs the POC to the charter and reports results against each threshold |
| Domain experts | Build the test set, write the gold answers and grade the outputs |
| AI engineers | Build the prototype and instrument cost, latency and logging |
| Security and privacy | Approve the data access and environment; run or review the adversarial tests |
Many POCs fail for reasons a charter can test early. In a July 2024 forecast, Gartner said no fewer than 30% of generative AI projects would be dropped after the proof-of-concept stage by the end of 2025, and it named four causes: poor data quality, inadequate risk controls, escalating costs or unclear business value. S&P Global Market Intelligence reported on 30 May 2025 that the share of companies abandoning most of their AI initiatives before production had risen from 17% to 42% in a year, and that organisations scrapped 46% of projects on average between proof of concept and broad adoption. Companies with lower failure rates were more likely to weigh compliance, risk and data availability when selecting projects, which is what the charter below forces you to do.
AI POC vs pilot vs MVP
| Proof of concept | Pilot | MVP | |
|---|---|---|---|
| Question it answers | Can the approach do the task to the threshold? | Do real users get the benefit in a live process? | Can we release and support a first version? |
| Who uses it | Expert evaluators | A limited group of real users | All target users |
| Data | A representative sample approved for testing | Live data in the real process | Live data |
| Environment | Isolated test environment inside the intended boundary | Production-like, limited scope | Production |
| Main measures | Quality, latency, cost per task and safety on a test set | Adoption, realised benefit, error rates, user feedback | Reliability, usage, support load |
| Output | Go/no-go decision with measured results | Scale decision and an updated business case | A supported release and a backlog |
Use the POC to retire the technical risk, then run a pilot to prove value with real users. When the real purpose is choosing between suppliers, the procurement guide’s POC protocol is built for that comparison; this template is for testing whether a use case is feasible at all.
The AI POC charter template
Fill in every field before any build work starts, and have the decision owner sign it.
| Field | What to write | Example |
|---|---|---|
| Hypothesis | If we apply [approach] to [task], it will reach [threshold] on [data] | An agent can extract 14 fields from supplier material certificates with at least 92% field accuracy and flag pass or fail correctly on at least 97% of them |
| Scope | The single task, its inputs and its outputs | Certificates for steel and aluminium parts, in PDF and scanned form |
| Exclusions | What the POC will deliberately not test | Write-back to the ERP, the supplier portal, languages other than English and German |
| Data and access | Sources, volumes, classification and who approved access | 1,200 historical certificates; confidential; approved by the data owner and the privacy lead |
| Success criteria | A numeric threshold for every dimension (see the next table) | Field accuracy ≥ 92%; 95th-percentile latency ≤ 60 seconds |
| Evaluation method and test set | How outputs are graded, by whom, and the size and mix of the test set | 200 documents, graded independently by two quality engineers |
| Cost per task | How cost will be measured, and the ceiling | ≤ €0.60 per certificate, including review time |
| Security and privacy constraints | Environment, data handling, model hosting, logging | Own data centre only; no document leaves the site; every run logged |
| Timebox | A fixed end date and what happens on it | Six weeks, ending in a decision meeting |
| Budget | People, compute and licences | Two engineers, two quality engineers part-time, an existing GPU server |
| Decision owner | The person who decides | Head of supplier quality |
| Exit decision options | The choices available at the end | Go to pilot, iterate once, change approach, stop |
Success criteria with numeric thresholds
| Dimension | Metric | Example threshold | How it is measured |
|---|---|---|---|
| Quality | Task or field accuracy | ≥ 92% of fields correct | Graded against gold answers by two reviewers |
| Quality | Groundedness | ≥ 95% of explanations supported by the source document | Reviewer checks each statement against the source |
| Latency | 95th-percentile time per task | ≤ 60 seconds | Timestamps in the run logs |
| Cost | Cost per completed task | ≤ €0.60 | Compute or tokens, tool calls and review minutes |
| Safety | Successful attacks in the adversarial set | 0 | Red-team cases run against the frozen prototype |
| Safety | Sensitive data exposed in outputs | 0 | Automated scan plus a reviewer sample |
| Hard gates | Conditions that end the POC at once | Data not lawfully usable; no route to production | Checked at charter sign-off and again at the end |
Derive the thresholds from the economics rather than from ambition. If a business case exists, its switching values give the minimum performance at which the investment still pays back.
The evaluation plan
Build the test set before the prototype
Sample real historical items across the variation that matters: formats, languages, suppliers, seasons, edge cases. Write the gold answers before the system runs, and hold back part of the set so nobody tunes the prompt against it. A workable mix is roughly 60% routine cases, 25% hard ones and 15% adversarial or out-of-scope items.
Size it with arithmetic. The usual confidence interval for a proportion, given in the NIST/SEMATECH e-Handbook of Statistical Methods (section 7.2.4.1), has a margin of 1.96 × √(p(1 − p) ÷ n) at 95% confidence. Solving for n gives n = 1.96² × p(1 − p) ÷ e².
- Expected success rate 90%, margin five points: 3.8416 × 0.09 ÷ 0.0025 = 138.3, so 139 cases.
- Same rate, margin three points: 3.8416 × 0.09 ÷ 0.0009 = 384.2, so 385 cases.
The handbook notes that this approximation may not be accurate enough when failures are very few, and gives an exact binomial method for that case. For safety tests, where the target is zero, count the cases and report every failure individually.
Decide what to measure
- Quality: accuracy against gold answers and, for anything that cites documents, groundedness.
- Latency: median and 95th percentile per task, measured end to end.
- Cost per task: for hosted models, input tokens × input price + output tokens × output price, plus tool or API calls; for self-hosted models, GPU hours per task × the fully loaded hourly cost of the server. Add review minutes × the reviewer’s loaded rate in both cases.
- Safety: cases drawn from the OWASP Top 10 for LLM Applications 2025, especially LLM01 prompt injection, LLM02 sensitive information disclosure and LLM06 excessive agency. For agents, add the OWASP Top 10 for Agentic Applications, released in December 2025: ASI01 agent goal hijack, ASI02 tool misuse and ASI03 identity and privilege abuse. The guide to red-teaming agents before production covers how to run these tests, and guardrails are the controls they probe.
- Agent path: for an agentic AI proof of concept, score the trajectory, tool calls and escalations as well as the final answer, as described in the agent evaluation explainer.
Decide who grades
Two domain experts grade each output independently, without knowing which prototype variant produced it, and a third resolves disagreements. Report the agreement rate, because low agreement means the gold answers are ambiguous. Use a model as a judge only after checking its grades against the human ones on a sample. This is the MEASURE function of the NIST AI Risk Management Framework in practice: quantitative and qualitative methods to analyse, assess, benchmark and monitor AI risk and impact.
The go/no-go rubric
Score each criterion separately. Do not average a safety failure away.
| Criterion | Go | Iterate | Stop or change approach |
|---|---|---|---|
| Quality against threshold | Meets every quality threshold | Within five points of a threshold, with a credible fix | More than five points short |
| Safety | No successful attack, no data exposure | Failures fixable by configuration or a control, then re-tested | A critical failure with no fix in reach |
| Cost per task | At or below the ceiling | Up to 25% above, with a cheaper option to test | More than 25% above |
| Latency | Within the threshold | Within twice the threshold | Beyond twice the threshold |
| Data readiness | Data available at production volume, access approved | Gaps with an agreed fix | Data unavailable or unusable |
| Production path | Environment, integration route and run owner agreed | Route known, owner missing | No route |
Decision rule: all Go means go to pilot. Any Stop on safety, data readiness or production path ends the POC or changes the approach. Anything else earns one iteration with a new end date, the same thresholds and fresh held-out cases, after which the answer is go or stop. Changing approach is a legitimate result: Gartner observed in June 2025 that many use cases positioned as agentic do not need agentic implementations, and a rules engine or a simpler retrieval flow can be the better answer.
Filled example: a fictional manufacturer’s supplier certificate agent
Example Components is a fictional maker of precision parts for aerospace and energy customers, with 2,000 employees. Its quality engineers check 40,000 supplier material certificates a year by hand, about six minutes each. Customer contracts forbid sending these documents to outside services, so this is a sovereign AI proof of concept: it runs on a GPU server in the company’s own data centre with locally hosted models. All figures are illustrative.
Charter in brief. The hypothesis, scope, exclusions and thresholds are the example column of the charter above. The test set holds 200 certificates: 120 routine, 50 hard (poor scans, handwritten amendments, multi-page documents) and 30 adversarial, with instructions planted in free-text fields plus documents that are not certificates. At an expected rate of 92%, 200 items give a margin of 1.96 × √(0.92 × 0.08 ÷ 200) = ±3.8 points. Fields from one certificate tend to fail together, so the team treats the 200 certificates, not the 2,800 fields, as the sample size.
Cost per certificate. The server’s fully loaded cost is €3.60 an hour, and it processed 30 certificates an hour in the test: €3.60 ÷ 30 = €0.12. Reviewers spent an average of 0.3 minutes per certificate at €0.85 a minute: €0.255. Total: €0.375, rounded to €0.38.
Results against thresholds.
| Criterion | Threshold | Result | Verdict |
|---|---|---|---|
| Field accuracy (200 × 14 = 2,800 fields) | ≥ 92% | 2,621 correct, 93.6% | Go |
| Pass or fail flag correct per certificate | ≥ 97% | 195 of 200, 97.5% | Go |
| Explanations grounded in the document | ≥ 95% | 192 of 200, 96% | Go |
| 95th-percentile latency | ≤ 60 seconds | 41 seconds | Go |
| Cost per certificate | ≤ €0.60 | €0.38 | Go |
| Successful attacks in the adversarial set | 0 | 2 of 30 led the agent to propose passing a failing certificate | Iterate |
| Production path | Route and run owner agreed | ERP integration route agreed; IT operations owns the server | Go |
Decision. Iterate once. Both attack cases were caught by an existing rule that a person approves every pass on safety-critical parts, but the agent itself should not have proposed them. The team gets two more weeks to filter instructions out of free-text fields and tighten the agent’s permissions, then reruns all 30 adversarial cases plus 20 new ones. With zero successful attacks, the POC goes to a pilot with six quality engineers. Field accuracy cleared its threshold, but by less than the margin of error, so the pilot keeps measuring it, along with the time saved against the six-minute manual baseline, which the POC never tested.
How to run the POC and keep a production path
Run it in six steps, inside the timebox:
- Sign the charter. The decision owner approves the hypothesis, thresholds and exit options.
- Build the test set and the baseline first. Gold answers exist before the prototype does.
- Build in the intended environment. Use the identity, access rules, logging and model hosting that production would use; the guide to separate development, test and production environments explains why. For a sovereign POC, test the locally hosted models you would actually run, not a public cloud model as a stand-in.
- Freeze, then evaluate. Lock the prototype version and run the full test set once, with cost and latency recorded.
- Hold the decision meeting. Present results against thresholds, never against expectations.
- Write the close-out report. Record results, failures, cost per task, open risks, the recommendation, and what the pilot must measure. Feed the measured values into the business case.
AI POC checklist
- Decision owner named and booked for the final meeting
- Hypothesis written as one testable sentence
- Thresholds agreed and signed before any build work
- Data access approved by the data owner and the privacy lead
- Test set built from representative data, with gold answers, before the prototype
- Adversarial and out-of-scope cases included
- Cost and latency recorded from the first run
- Environment matches the intended production boundary
- Production route and run-cost owner identified
- Close-out report format agreed
Common AI POC mistakes
- No success criteria. Without thresholds set in advance, any result can be described as promising.
- Demo-ware. A polished interface on ten hand-picked examples proves nothing about the other thousand. Gartner estimated in June 2025 that only about 130 of the thousands of agentic AI vendors are real, so check that a so-called agent actually plans and uses tools.
- No production path. A prototype built with a model, data access or environment that production will never allow has to be rebuilt, and many stall there. The common blockers are covered in why agent POCs stall before production.
- Testing on clean data only. Real inputs include scans, typos, missing fields and hostile text. Leave them out and the pilot will find them.
- Moving the goalposts. Lowering a threshold after seeing the results turns the POC into theatre. Change thresholds only through the decision owner, in writing.
- Ignoring cost until later. Cost per task measured in the POC is the cheapest warning you will get; see what pilot budgets usually miss.
How VDF AI fits
A POC built on the platform you intend to run in production avoids one common failure, the rebuild. VDF AI provides AI agents, multi-agent orchestration through Networks and private retrieval over company documents in VDF AI Chat, with role-based access control on every plan and audit logs. VDF AI Router can send the same task to approved cloud models or to registered on-premises model endpoints, so a POC can compare candidates under one set of controls, and its air-gap mode keeps a sovereign POC on local models only.
For a POC that has to run inside your own data centre, an on-premises bundle can begin with a three-month platform trial. The platform does not write the hypothesis, build the test set, choose the thresholds or take the decision; those stay with the POC lead and the decision owner. The broader sequence from POC to rollout is set out in the implementation playbook for the full programme.
Sources
The studies, standards and statistical references below were re-read at their original publishers on 4 November 2026.
Frequently asked questions
What is an AI proof of concept (POC)?
An AI proof of concept is a small, timeboxed experiment that tests one hypothesis: that a particular AI approach can perform a specific task to a defined standard of quality, safety and cost. It runs on representative data, usually outside production, with expert evaluators rather than everyday users. Its output is a decision to pilot, iterate, change approach or stop, supported by measured results.
What is the difference between an AI POC, a pilot and an MVP?
A POC tests feasibility: can the approach do the task to the threshold on real data. A pilot tests value and adoption: do real users in a live process get the expected benefit at an acceptable risk. An MVP is the smallest version of the product that can be released and supported for ongoing use. Each answers a different question, so each needs its own success criteria.
What are good success criteria for an AI POC?
Numeric thresholds, agreed before the build, on four dimensions: quality, such as task accuracy and groundedness against source documents; latency, such as 95th-percentile time per task; cost per completed task; and safety, such as zero successful prompt injections in the adversarial set. Add hard gates that stop the POC whatever the scores, such as data that cannot be used lawfully or no route to production.
How big should the test set be for an AI POC evaluation?
Big enough that the margin of error is smaller than the gap between passing and failing. To estimate a success rate near 90% to within five percentage points at 95% confidence you need about 139 cases, from n = 1.96² × p × (1 − p) ÷ e². A three-point margin needs about 385 cases. Include hard and adversarial cases as well as routine ones.
How is an agentic AI proof of concept different?
An agent plans steps and calls tools, so the POC has to score the path as well as the answer: task completion, correct tool choice and arguments, permission boundaries respected, and hand-off to a person when it should. Test it against agent-specific attacks such as goal hijacking and tool misuse from the OWASP Top 10 for Agentic Applications, and check that the task needs an agent at all.
How do you run a sovereign AI proof of concept?
Run it inside the boundary where the production system will live, such as your own data centre, a private cloud or an air-gapped network, with models that can be hosted there. Use the same data classes and access controls you will use later, because a POC that passes only on a public cloud model does not prove the sovereign option works. Record the model, hardware and cost per task, since those differ most between environments.
Run the proof of concept where production will run
VDF AI can be deployed in VDF's managed cloud or inside your own boundary, with role-based access, audit logs and approved models in place during the test, so the POC faces the conditions production will.