AI Center of Excellence: How to Set Up an AI CoE, Choose Its Structure and Run the First 90 Days

What an AI center of excellence is for

Most large organisations now have more AI activity than anyone can see. Copilots arrive with office suites, vendors switch on assistants inside existing products, and business teams build agents in whatever builder they were shown last. Without a coordinating team, the same problems get solved several times over, and nobody can tell the board how many AI systems the company runs.

An AI CoE exists to fix that. Its job has three parts:

• Direction: decide which problems are worth solving with AI and in what order, based on value, risk and effort.
• Guardrails: set the standards, risk tiers and approval steps that every project follows, and keep the inventory current.
• Leverage: provide a shared platform, reusable components and expertise so that each new use case starts from a working base rather than a blank page.

What it is not: a delivery bottleneck that builds everything, an innovation lab that produces demos nobody adopts, or a compliance office that only says no. The CoEs that last tend to measure themselves on what other teams ship through them.

Three operating models

The choice of AI center of excellence structure mostly comes down to where delivery happens and where control sits. The three common patterns are below.

A sensible path for a mid-sized organisation is to launch centralised, publish standards and stand up the platform within the first two quarters, then add spokes in the two or three functions with the strongest demand. Each spoke usually pairs a business product owner with one or two builders who use the hub’s platform. Go federated only when the platform enforces policy by itself; if a division can bypass the inventory or the gateway, the council cannot see what it answers for.

AI center of excellence structure: roles

Titles vary, but the work does not. A first CoE can cover these roles with a handful of people, several of them part-time, and grow from there.

Two design points matter more than the org chart. First, keep the risk function independent: the liaison works with the CoE, but the governance committee that approves high-risk uses should not report to the team that wants adoption numbers to rise. Second, give every AI system in the inventory a named business owner outside the CoE. When the CoE owns everything, it becomes the place blame lands, and business teams stop treating the systems as theirs.

The AI CoE charter

The charter is a short document, three to five pages, that the sponsor signs. It makes the CoE’s authority explicit so that every later argument has a reference point. A workable charter covers:

1. Mandate: the problem the CoE exists to solve and the scope (all AI, generative AI only, or AI above a risk threshold).
2. Decision rights: what the CoE approves, what it advises on, and what it can stop. Spell out whether a business unit can buy an AI product without CoE review.
3. Operating model: centralised, hub-and-spoke or federated, and how that will change.
4. Membership and roles: named people, time allocations and the escalation route.
5. Interfaces: how the CoE works with procurement, security, privacy, legal, internal audit and the architecture board.
6. Standards it maintains: risk tiers, evaluation thresholds, logging, data classification for AI and the approved model list.
7. Funding: whether the CoE is a cost centre, charges back platform usage, or co-funds pilots.
8. Metrics and reporting: what goes to the sponsor and the board, and how often.
9. Review date: charters written in year one are usually wrong by year two.

The governance policy itself, with its risk tiers and committee terms of reference, is a separate document. Our AI governance policy template includes a committee charter you can adapt, and the AI governance framework crosswalk shows how one control set can serve several references at once.

Intake and prioritisation

Intake is where a CoE earns or loses the trust of the business. A long form and a slow answer push teams around it; a quick, useful answer means the CoE hears about ideas while they are still cheap to change.

A workable intake flow has four stages:

• Submit: a short form that captures the problem, the current process and its cost, the data involved, who would act on the output and how much autonomy the AI would have. The AI use case intake template gives a ready-made form, a one-page canvas and a scoring sheet.
• Triage within a week: the CoE screens for duplicates, obvious blockers and the risk tier. Some ideas are better served by an existing tool, some by a non-AI fix.
• Score: value, risk and effort, scored by the same people each cycle so scores are comparable. The method in our guide to ranking AI use cases by value, risk and effort is a reasonable default.
• Decide and publish: the portfolio decision, made monthly or quarterly, is shared with everyone who submitted. Rejected ideas get a reason.

The intake record becomes the first entry in the AI inventory. That link matters: an inventory built separately from intake is always out of date. The Academy lesson on building an AI system register covers the fields worth keeping.

How the CoE connects to governance obligations

Regulation does not require a centre of excellence, but it does create recurring work that someone has to own. These are the references a CoE in a regulated organisation most often maps its processes to. Each was checked on the publisher’s site in October 2026.

EU AI Act. According to the European Commission’s timeline page, the Act entered into force on 1 August 2024 and became applicable on 2 August 2026, with exceptions. Prohibited practices and the AI literacy obligation applied from 2 February 2025. Following the AI Omnibus, high-risk obligations for Annex III use cases apply from 2 December 2027, and rules for AI embedded in regulated products under Annex I from 2 August 2028 (verified October 2026). For a CoE, three parts matter most:

• AI literacy (Article 4). Providers and deployers must take measures so that their staff have sufficient AI literacy. The Commission’s Q&A says no certificate is needed, that keeping an internal record of training is optional rather than mandated, and that national market surveillance authorities supervise the duty from 2 August 2026. The CoE’s enablement programme is the natural home for this.
• Deployer duties for high-risk systems (Article 26). Deployers must use such systems according to the provider’s instructions, assign human oversight to people with the necessary competence, training and authority, and keep automatically generated logs for at least six months. The CoE’s approval gates and platform logging are where these become routine.
• Inventory and classification. Knowing which systems might be high-risk, or which fall under the transparency rules, starts with the intake record. The EU AI Act compliance checker helps with a first pass.

ISO/IEC 42001:2023. Published on 18 December 2023, the standard sets requirements for establishing, implementing, maintaining and continually improving an AI management system, and applies to any organisation that provides or uses AI-based products or services. Because it is a management-system standard, it can be audited and certified. A CoE that already runs intake, risk tiers, an inventory and periodic review has much of the operational material an auditor will ask for. Our ISO 42001 guide covers certification in detail.

NIST AI Risk Management Framework. Released on 26 January 2023, the AI RMF is intended for voluntary use and is organised into four functions: Govern, Map, Measure and Manage. NIST added a Generative AI Profile (NIST AI 600-1) on 26 July 2024, and its site states that AI RMF 1.0 is being revised under the White House AI Action Plan. Many CoEs use its vocabulary to structure their risk register even outside the US.

A public-sector model. The US Office of Management and Budget memo M-25-21, dated 3 April 2025, shows a government version of this structure. It required each agency to retain or designate a Chief AI Officer within 60 days, and each CFO Act agency to convene an AI Governance Board within 90 days, chaired at Deputy Secretary level with the Chief AI Officer as vice-chair and the agency’s AI use case inventory in the officer’s care. Private firms are not bound by it, but the division of labour is a useful reference for a charter.

The shared platform the CoE should own

The fastest way for a CoE to lose control is to let every project pick its own model provider, vector store and agent tool. The fastest way to lose the business is to block projects until a perfect platform exists. The middle path is a small set of shared capabilities, available early, that make the governed route the easiest route.

Deployment location is a policy decision as much as a technical one. Public or low-sensitivity work can run on cloud models with suitable contracts, while customer records, employee files, trade secrets and health data are often better kept on infrastructure the organisation controls. The result is usually a mixed estate in which a gateway decides, per request, which model a given data class may reach. The AI gateway page explains that routing pattern, and the AI readiness assessment helps a new CoE see which capabilities it already has.

Evaluation deserves early attention because it is the capability most often skipped. A CoE that sets a quality bar for each risk tier, and asks every project to show results against it, avoids a long list of arguments later. For retrieval-based assistants, our guide to RAG evaluation sets out the metrics and tools.

Metrics the CoE should report

Report a few numbers consistently rather than many numbers once. Each metric below can be produced from the intake record, the inventory and platform logs, without a survey.

Measure benefits against the baseline written down at intake, not estimates made after the fact. A rising inventory count is not success on its own either; it often means the register is finally catching shadow AI that already existed.

The first 90 days

Days 1 to 30: mandate and baseline.

• Agree the charter with the sponsor, including decision rights and the review date.
• Name the core team and the risk, security and privacy liaisons.
• Run a discovery exercise: interview function heads, collect the AI tools already in use, and start the inventory from what you find.
• Publish a one-page interim rule set: approved tools, banned data uses and how to ask for help.

Days 31 to 60: process and platform.

• Launch the intake form and the first triage cycle.
• Agree risk tiers and the approval gates for each, using a template rather than a blank page.
• Stand up the minimum shared platform: gateway with logging, one retrieval source and identity integration.
• Start the AI literacy programme with role-based modules for executives, builders and everyday users. The AI literacy training use case shows one way to deliver it.

Days 61 to 90: first results.

• Take two or three use cases from intake to production on the shared platform, with business owners named and baselines recorded.
• Run the first portfolio review and publish decisions to everyone who submitted.
• Report the first metrics to the sponsor, including what the inventory found.
• Decide where the first spokes will sit and what support they need.

The first use cases should be visible, reviewable and low-risk: drafting, summarising and answering questions from approved documents, with a person checking the output. A CoE that ships two modest systems in its first quarter has more authority than one that publishes a perfect framework and nothing else.

Common failure modes

• The lab trap. A CoE that measures itself on prototypes produces prototypes. Tie its goals to systems in production and benefits realised.
• Policy without plumbing. A rule that cloud models may not see customer data means little if nothing in the network or the gateway enforces it.
• Orphaned systems. Every live system needs a business owner, a review date and a retirement plan, or the inventory fills with tools nobody maintains.

How VDF AI fits

VDF AI gives an AI CoE a shared platform it can run on its own infrastructure: on-premises, in a private cloud it controls, air-gapped, or through VDF AI Cloud for teams that prefer a managed service. The parts map directly to the capabilities above.

The AI Router acts as the model gateway. It routes requests across local open-weight models and approved cloud models, applies policies and budgets, and records an audit entry for every call, so the CoE can decide which data classes may reach which models and prove it later. Private RAG connects knowledge sources such as SharePoint and Microsoft 365, Confluence, Jira, Google Drive, GitHub and Notion, and retrieval respects the permissions of each user. Agent Hub is a no-code builder with an MCP tool registry and per-role tool grants, which lets spokes build their own agents while the hub decides which tools each role may use. Human approval steps, role-based access control on every plan, and audit logs support the oversight and logging duties described earlier.

On-premises deployments sign in natively through Microsoft Entra ID; other identity providers connect through an SSO-aware reverse proxy. VDF AI holds no certifications of its own, so the CoE’s management system and controls carry the compliance weight, as they would with any platform.

To see where your organisation stands before chartering a CoE, take the AI readiness assessment, review the AI agent platform for the builder and registry, or compare deployment options on the pricing page.