AI proof of concept: how to run one that proves something

An AI proof of concept is a short, fixed test of one question: can this approach do this task, on your real data, to a standard you set in advance? Run it on past cases with known right answers, agree the pass mark and the stop rule before anyone builds, and expect a decision at the end, not software.

What is an AI proof of concept, and how is it different from a pilot?

The words get used interchangeably, and that is how small businesses end up paying for a "pilot" that was really a demo. Each stage answers a different question.

Stage	The question it answers	What it runs on	Who uses it	What you keep
Proof of concept	Can this approach do the task on our data, well enough?	Past records with known right answers	Nobody outside the project	The test set, the scores, a decision
Pilot	Does it save time in our real process?	Live work, alongside the old way	A few staff, checked by a person	A measured before-and-after
Production build	Can we rely on it every day?	All the live work	The whole team	A supported, monitored system

A proof of concept never touches a customer and never replaces anyone's work. Its only output is evidence. The pilot comes next, and our 90-day plan for implementing AI in a small business covers how to run that stage in parallel with the existing process.

The UK government's service manual describes the same idea for digital projects in general: in the alpha phase you test your riskiest assumptions with code "just complex enough to let you test different ideas, not production quality code", and you should expect to throw much of it away (How the alpha phase works, GOV.UK Service Manual, last updated May 2019). An AI proof of concept is an alpha with one assumption in it.

When don't you need a proof of concept?

When you are buying a mainstream tool for a mainstream task. If you want an assistant to draft replies, summarise meetings or tidy documents, and the product is already in the software your team uses, the vendor has done the feasibility work. Skip straight to a pilot and measure whether it saves time.

A proof of concept earns its cost when something is genuinely uncertain:

• You would be building, not buying. A bespoke extraction, classification or question-answering system on your own documents. Our guide to build vs buy for SMEs covers that choice.
• A vendor's claim depends on your documents. "Reads any invoice" is a claim about their test set, not yours.
• The task needs judgement. Triage, risk scoring and anything where two experienced staff sometimes disagree.
• Being wrong is expensive. Money moving, advice given or decisions made about people.

If your processes are not written down yet, a proof of concept is premature. Our AI readiness assessment is a quicker way to find that out.

How do you write the question a proof of concept must answer?

In one sentence, with a number in it, before anyone writes code. If you cannot fit the question into a sentence, the scope is too wide.

Weak: "See whether AI can help with supplier invoices."

Strong: "On 150 past supplier invoices, can the system extract supplier, date, net, VAT and total with at least 95% of fields correct, flagging the rest for a person, by 6 November?"

Anthropic's guidance on evaluating language-model applications makes the same point about success criteria: they should be specific, measurable, achievable and relevant, and its worked example replaces "the model should classify sentiments well" with a numeric target on a held-out test set (Define success criteria and build evaluations, Claude Docs, read October 2026).

Write two more lines next to the question:

1. The pass mark. The number above which you move to a pilot. Set it from what the task needs, not from what you hope the model will manage.
2. The stop rule. The result below which you stop, and the date you stop on even if the work is "nearly there". In our experience the stop rule is the line most often left out, and it is the one that keeps a two-week test from becoming a three-month one.

How do you build a test set from your own records?

From real past work, with the right answers agreed before testing starts. This is the part that makes a proof of concept prove something, and it is mostly a job for the person who does the task today, not the developer.

Pick cases that look like the real workload. Anthropic's guidance is to design tests that "mirror your real-world task distribution", including edge cases. For invoices that means the handwritten one, the credit note, the supplier who puts VAT in a strange place. A test set of tidy examples measures nothing.

Size it so one odd result cannot swing the verdict. We usually aim for 50 to 200 cases. That is our working range, not a statistical rule; a narrow, repetitive task needs fewer than a varied one.

Write the right answer down first. If the answer key is written after the system has run, people unconsciously mark it generously.

Hold some back. Keep a slice of cases the builder never sees until the final run. Tuning a prompt against the same examples you score on flatters the result.

Score more than one thing. The government's AI Playbook separates model metrics, which "measure how well your technology is performing", from service metrics, which show whether the business goal is met, and notes a model can score well while the service still disappoints (AI Playbook for the UK Government, GOV.UK, February 2025). Alongside accuracy, record how long a person takes to check each output and what a wrong answer would have cost.

Tooling exists for this. Microsoft's Foundry platform, for example, describes a pre-production stage that "validates performance through evaluation datasets" and lets you bring your own data (Observability in generative AI, Microsoft Learn, updated August 2026). For an SME proof of concept, a spreadsheet with one row per case and a column per field is usually enough.

What about personal data in a proof of concept?

Treat it as real processing, because it is. The ICO lists "the use of innovative technologies, or the novel application of existing technologies (including AI)" among the factors that can make processing high risk, and a data protection impact assessment has to be completed before the processing starts (When do we need to do a DPIA?, ICO, guidance marked as under review in October 2026).

In practice, three habits keep a proof of concept on the right side of this:

• Use anonymised or redacted records wherever the task allows it.
• Check the AI provider's terms on whether your inputs are retained or used for training, and use business or API terms rather than a consumer account.
• If the test needs real customer, staff or applicant data, decide on the DPIA in week one, not after the results are in.

Our guide to GDPR and AI for UK SMEs goes through the checks in more detail. It is not legal advice, and neither is this.

How long should it take, and what should you get at the end?

Two to four weeks of elapsed time for one narrow question is our planning band. Our breakdown of what AI consulting costs UK SMEs puts the effort for a narrow proof of concept at 5 to 15 days and shows the price at current day rates.

At the end you should hold four things:

1. A score sheet. Every test case, the system's answer, the right answer, pass or fail.
2. A failure log. Not just how often it was wrong but how. Models tend to fail in patterns, and a pattern can often be fixed with better inputs.
3. A running-cost estimate. What the approach would cost per month at your real volumes, with the assumptions written down.
4. A recommendation. Pilot, fix and retest once, or stop.

Stopping is a legitimate result. A proof of concept that fails cheaply in three weeks has done its job better than a project that limps on for six months.

What are the warning signs of a proof of concept designed to pass?

Most bad proofs of concept are not dishonest. They are set up so that nothing could make them fail. Watch for:

• The demo runs on the supplier's sample data, not yours.
• The examples were chosen after seeing which ones worked.
• The pass mark appears after the results.
• The scope grows mid-test. "While we're at it, could it also…" resets the clock and the evidence.
• The person who built it is the only one who marks it.
• There is no stop rule, so every result becomes a reason for another phase.

If a proposal for AI work has no measurable question at the top, it is worth asking for one before you sign. That is where our AI strategy work starts, and when the answer is a build, our AI development team takes it from proof of concept to production. The AI practice overview covers how the two fit together.

Frequently asked questions

How long should an AI proof of concept take?

For one narrow question, two to four weeks of elapsed time is our planning band, and our AI consulting cost guide puts the effort at 5 to 15 days. If it needs months, the question is too broad or the data is not ready, and that finding is itself a useful answer.

What is the difference between an AI proof of concept and a pilot?

A proof of concept tests whether the approach can do the task on your real data, using past examples with known right answers. A pilot puts a working tool into the live process, alongside the old way, to see whether it saves time in practice. Proof of concept first, pilot second.

How many examples do we need to test an AI proof of concept?

Enough that one odd result cannot swing the verdict. We usually aim for 50 to 200 real past cases, including the awkward ones, with the right answer agreed in advance by the person who does the job today. That is our working range, not a statistical rule.

Do we need a DPIA for an AI proof of concept?

Possibly. The ICO lists innovative technology, including AI, as a factor that can make processing high risk, and a DPIA has to be done before the processing starts. If the test can run on anonymised records, do that; if it needs real personal data, settle the DPIA question first.

Should we keep the code from a proof of concept?

Usually not. Proof-of-concept code is built to answer a question quickly, not to run every day. Keep the test set, the scores and the log of failures, because those carry straight into the build. Rebuild the code properly once the decision is made.

---

Easy Insight is a UK consultancy for AI, web, apps and data — senior specialists only, no juniors.