An AI operating model is the set of people, rules, records and backup arrangements that keeps an AI tool useful after the pilot team leaves.

In everyday terms, it answers four questions: who checks the work, who can stop it, what happens when a case does not fit, and how the organisation continues when the AI is unavailable.

The technology matters. This interview concentrates on the part people meet at work: decisions, handovers, unusual cases, supplier changes and the pressure to keep moving when the system is uncertain.

1. Why do AI projects look successful during pilots but struggle after launch?

Pilots take place under kinder conditions than ordinary work. They usually have selected data and interested staff. Experts are nearby. Live work brings busy periods and incomplete information. It also involves people who were not part of the experiment.

A support team may test an AI tool that summarises customer calls. During the pilot, experienced agents notice weak summaries and correct them. After launch, casual staff trust the summary, the quality team checks only completed cases, and nobody knows whether a supplier update changed the result.

The pilot still did its job. It showed that the idea could help. What it did not show was whether the organisation could run it on a difficult day without the original team watching. That requires a named owner, a way to handle cases that do not fit, and a backup route when the tool fails.

I call the final check a bad Tuesday test. Before launch, try one busy period, one absent expert, one strange input and one supplier outage. Watch who notices the problem. Ask who may decide what happens next and how ordinary work continues.

Source note: S1, S2.

2. What changes when AI moves from an innovation team into a live business workflow?

The consequences move to the people doing the ordinary work. During a pilot, the project team can catch mistakes. After launch, an ordinary employee may be the first person to discover that the AI is wrong.

Take an AI assistant that helps code supplier invoices. A new invoice format arrives on a Friday afternoon. The assistant puts it in the wrong tax category. Accounts payable now needs to know whether to hold payment, who can decide, and what evidence should be kept.

Three things must leave the innovation team with the technology:

• the know-how to recognise common failures;
• permission for named people to pause or change the process;
• a record of the versions, decisions and incidents that explain what happened.

This is the point where a tool becomes an operating responsibility. The business owner carries the result. Technology keeps the connection running. Specialists help with data handling and privacy. Security staff may be involved too. Somebody still needs the final authority to act.

Practical test: Ask an ordinary employee to explain what they would do if the AI produced a doubtful answer and the project team could not be reached. Their answer will show whether the handover is real.

Source note: S6.

3. How can a CIO determine whether a process is ready for AI?

Begin with the job, not the software. A process is ready only when people understand what a good result looks like and what must happen when the normal rules do not apply.

Five plain questions reveal most readiness problems:

1. What result are we trying to improve?
2. Is the information lawful to use and good enough for this purpose?
3. Can we recognise a doubtful or unusual case before somebody is harmed?
4. Who can stop, challenge or correct the result?
5. How will we compare the new process with the way the work performs today?

That final comparison is the baseline. It means knowing the current time and quality before AI is introduced. It should also show how much effort the work requires.

An internal meeting-note assistant may need a light check: consent, safe handling of sensitive material and occasional sampling. A tool that influences customer hardship decisions needs much more because one unusual error could deny help to a vulnerable person.

Try this tomorrow: Take one proposed use and answer the five questions on a single page. Mark every uncertain answer as a condition that must be closed before launch. If nobody can explain how work continues without the AI, the process is not ready.

Source note: S1, S2.

4. Who should own an AI capability after implementation?

One person should own the business result and have enough authority to change the AI-supported process. That person must also be able to pause or retire it. Specialists can help, but a group cannot replace a clear decision-maker.

Suppose a recruitment screening assistant behaves differently after a supplier update. Human resources owns the effect on applicants and can suspend its use. Technology can switch off the connection or restore an earlier version. Privacy can examine how personal information was handled. Procurement can use the contract to question the supplier.

Each role matters, but the business owner carries the final outcome. Otherwise, every specialist may recognise part of the problem while each assumes someone else can decide.

A simple authority map is more useful than a complicated organisation chart. Write down who may:

• approve or deny a material change;
• stop automated action;
• accept the risk that remains;
• restore the service after a failure.

“Risk that remains” is the plain meaning of residual risk. No safeguard removes every possibility of error.

Do this next: Put one person and one deputy beside each decision. If two people believe the other has the power to stop the system, nobody truly owns it.

Source note: S1.

5. What is the difference between deploying an AI tool and operating an AI capability?

Deployment means the software is available. Operation means the organisation can use it safely on an ordinary day and recover when something goes wrong.

The difference is similar to buying a vehicle and running a delivery service. The vehicle matters, but the service also needs a driver and routes. Somebody must maintain it. Staff need to know what to do with a damaged parcel or a breakdown.

An AI contract assistant might be installed for 800 employees. Licence use rises and the launch appears successful. Then legal reviewers receive more weak drafts, confidential clauses enter public prompts, and procurement cannot tell which supplier version produced a recommendation.

In that situation, the software works while the service around it remains incomplete.

For each live AI use, keep a short service record:

• what it is meant to help with;
• who owns the result;
• what information it may use;
• what it must never decide;
• how doubtful cases are handled;
• how people work when it is unavailable.

Small-organisation version: Keep this on one protected page and review it after a major supplier or workflow change. Review it when policy changes too. The record matters more than the system used to store it.

Source note: S3.

6. Which post-launch problems are usually underestimated?

The quiet problems matter most because they can continue for months without looking like a failure. Inputs change. Instructions become stale. Staff spend longer checking the output. A workaround becomes the real process while the official procedure remains untouched.

Consider a tool that ranks sales leads. It begins favouring prospects with complete customer records. Staff notice that filling certain fields raises a lead’s position, so they change how they enter information. The dashboard remains healthy, but the tool is gradually measuring staff record-keeping rather than customer interest.

That is a feedback loop: people change their behaviour because of the AI, and that new behaviour changes what the AI sees.

Post-launch review should look for four kinds of movement:

• the information entering the system has changed;
• unusual cases or human corrections are increasing;
• checking work has moved to another team;
• the same incident or complaint keeps returning.

Zero corrections can be a warning. It may mean the AI is excellent, or it may mean staff have stopped questioning it.

Monthly check: Review a small sample of normal cases and difficult cases. Ask what changed. Find out who absorbed extra work. Check whether an earlier concern came back.

Source note: S4.

7. How should organisations handle incomplete, contradictory or unusual cases?

The AI must be allowed to say, “I do not have enough information.” A doubtful case should stop before the output quietly becomes a decision.

An insurance claim might contain two addresses, a recent surname change and a document that conflicts with the online form. Forcing that case through the normal queue produces a confident-looking answer from uncertain material. A safer system marks the conflict and sends the case to a named person.

This route is the exception path. An exception is simply a case that does not fit the normal rules.

Human checking takes time, so sending every case to a person defeats much of the value. Use a simple split:

• routine and reversible cases may continue within tested limits;
• doubtful or high-consequence cases stop for review;
• prohibited cases never proceed automatically.

The reviewer needs the original information and enough time. They also need real authority. A person who can see the problem but cannot change the decision is only providing the appearance of oversight.

Build one exception card: Write down the warning sign and the immediate safe action. Name the person who decides. Set the maximum wait and state what record must be kept. Test the card with three cases the team has not rehearsed.

Source note: S6.

8. What authority must employees have when they disagree with AI output?

Employees need a safe way to pause the action and reach someone who can decide. A comment box or complaint lodged after the event is too late when the AI is about to affect a payment or customer. The same is true for a decision about an employee.

A fraud system may block an urgent payment to a hospital supplier. The finance officer should be able to hold the transaction and contact a duty manager. The officer should not be able to release it alone. The duty manager can approve a temporary alternative after verifying the supplier through a known channel.

Authority should match the consequence:

• Green: reject a writing suggestion and continue.
• Amber: pause a decision and ask for review.
• Red: stop the action and use the emergency route.

Some boundaries must remain strict. An employee should not bypass privileged access or safety controls merely because the AI causes delay. The organisation must separate a legitimate challenge from an unsafe shortcut.

Give staff a one-page decision card: State what they may stop or change. Mark anything they must never bypass. Add the on-call contact and the record staff must keep. Test whether the named person can actually be reached during a busy period.

Source note: S1, S6.

9. How should staff be trained to challenge, override and escalate AI decisions?

Let people practise the difficult moment. A product demonstration teaches buttons. Rehearsal teaches judgement.

Use a real situation: an AI-generated supplier bank-change recommendation arrives late in the day. One employee notices that the email address looks unusual. A senior manager wants the payment released. The usual subject expert is away.

Now observe what happens. Does somebody pause the payment? Can staff find a known supplier contact? Who makes the final decision? What record is left for the next shift?

This is what an override means: a person with authority stops or changes the action suggested by the AI. Escalation means moving the unresolved decision to somebody with greater authority or knowledge.

A small organisation does not need an elaborate exercise. Twenty minutes once a month can be enough if the team uses the real workflow. The session is worthwhile only when it changes an instruction or permission. It might instead update a contact list or system setting.

After each rehearsal, ask four things: What did we notice? Who decided? Where did the process stall? What will we change before the next exercise?

Source note: S2, S7.

10. What should organisations measure beyond licences, users and prompt volume?

Measure whether the work improved, whether effort moved elsewhere, and whether the organisation can recover from a mistake. Usage tells you that people opened the tool. It does not tell you whether the result helped.

A service desk assistant may reduce drafting time by three minutes. Agents then spend four additional minutes checking technical details, while senior staff handle more escalations. Adoption is high, yet the service now takes longer.

Use three groups of measures:

1. Did the outcome improve? Look at usable results, customer experience and total time.
2. Did work move? Count checking, correction, waiting and extra effort in other teams.
3. Can we recover? Track doubtful cases, complaints, repeated failures and time to restore normal work.

The baseline is the same set of measures taken before launch. Without it, a faster AI response can be mistaken for a faster service.

Keep the scorecard small: Give each measure an owner and a source. Set the point at which somebody must act. Remove any measure that cannot change a decision. A chart that nobody uses is paperwork, not evidence.

Source note: S1, S3.

11. How should organisations account for security, privacy, data quality and vendor dependency?

Ask what information the AI sees, who can access it, how wrong information is corrected, and what happens if the supplier changes the service. These questions belong together because a weakness in one area can damage the others.

A transcription supplier may change its retention terms and its model in the same month. The customer then discovers that it cannot identify which recordings were used or reproduce an earlier result. Its specialised vocabulary cannot be moved easily either.

In plain language:

• Data quality: Is the information accurate enough for this job?
• Privacy: Are we allowed to use it, and have people been treated fairly?
• Security: Who can see or change the information and the system?
• Vendor dependency: What can no longer be done if the supplier changes or disappears?

Moving suppliers is rarely simple. Prompts may be portable while the history and special settings remain behind. Safety checks may not behave the same way with a replacement.

Before renewal, run a supplier-loss exercise: What can we take with us? Which work can continue manually? How long can we manage? Record where the data sits and who has access. Add the supplier’s notice duties and export method. State the tested fallback separately.

Technology teams may keep additional detail about model versions and connected systems. The business reader still needs the plain answer to those three continuity questions.

Source note: S4, S5, S8, S9.

12. How can leaders tell whether productivity improved rather than merely shifting work to employees?

Follow one piece of work from beginning to end and count everybody’s effort. A faster step is not a productivity gain when another person or team inherits the checking. Sometimes the customer inherits the correction.

A reporting assistant cuts an analyst’s drafting time by 30 minutes. A senior manager then spends 20 additional minutes checking the report. A data steward spends another 15 minutes resolving inconsistent definitions. The analyst is faster; the organisation is five minutes slower.

Hidden work often appears outside the project measures:

• employees maintain private prompt notes;
• managers settle doubtful cases after hours;
• customers correct automated messages;
• one experienced specialist becomes the permanent safety net.

The question is not whether AI saved time at one stage. Ask whether the finished service required less total effort without creating unacceptable risk.

Draw the work before and after AI: Record the preparation and waiting. Add the time spent checking and correcting. Include recovery time as well. Do this for an ordinary case and a difficult one. Then ask the people downstream what changed for them.

Publish the net result. Local speed can be useful, but it should not be presented as whole-of-service productivity.

Source note: S1.

13. When should an organisation retrain, redesign, pause or retire an AI capability?

Choose the response that matches the problem. Retraining cannot repair a broken workflow, and a policy change cannot correct a model that no longer recognises current examples.

Use four plain tests:

• Retrain when the job is still right but the examples the AI learned from are out of date.
• Redesign when the handover, screen, rule or authority is causing the failure.
• Pause when continued use may cause harm or nobody can show that it is safe.
• Retire when the purpose has disappeared, the cost exceeds the value or a simpler method works better.

A clinical scheduling assistant begins omitting one category after an upstream coding change. Retraining would be the wrong first move if the real defect is the connection between systems. Pause the affected output and restore the manual queue. Then find the broken link.

During an incident, speed matters. A named person may need permission to stop automated action before a committee meets. Agree on the authority and backup process in advance. Decide what evidence is required before restarting.

Write the stop conditions now: Use four headings: performance change, failed safeguard, possible harm and supplier loss. Beside each one, name the immediate action. Also name the person who decides when service can return.

Source note: S6, S7.

14. What is the one question a CIO should ask before declaring an AI project successful?

Ask: Can the organisation still produce a safe, useful and understandable result when the AI is wrong, unavailable or no longer suitable?

That question tests the whole operating model. Can people recognise a doubtful result? May they stop the action? Is there another way to complete the work? Can the organisation later explain what happened and learn from it?

The answer should match the risk. A writing assistant may need a simple fallback: stop using it and return to the normal editor. A system influencing claims, safety, employment or access needs a tested route for interruption, competent review and safe restoration.

A CIO reviewing an AI claims assistant could ask the team to demonstrate three moments: a failed input, a challenged recommendation and a supplier outage. The team should be able to show the safe queue and the person who decides. It should also show the record that remains and the test used before restarting.

Do not close the project on a usage chart: Ask the team to demonstrate one failure and recovery in the real workflow. A fallback that exists only in a policy has not yet been proved.

Source note: S1, S7.

Source notes

S1. NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0), 26 January 2023: https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-ai-rmf-10

S2. NIST, AI RMF Playbook, 30 March 2023: https://airc.nist.gov/airmf-resources/playbook/

S3. NIST, Generative Artificial Intelligence Profile, NIST AI 600-1, 26 July 2024: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

S4. ASD’s ACSC, UK NCSC, CISA and partners, Guidelines for Secure AI System Development, 26 November 2023: https://www.cyber.gov.au/business-government/secure-design/artificial-intelligence/guidelines-for-secure-ai-system-development

S5. OAIC, Guidance on privacy and the use of commercially available AI products, 21 October 2024, updated 17 January 2025: https://www.oaic.gov.au/privacy/privacy-guidance-for-organisations-and-government-agencies/guidance-on-privacy-and-the-use-of-commercially-available-ai-products

S6. OECD, AI Principles overview, updated May 2024: https://oecd.ai/en/principles

S7. NIST, Cybersecurity Framework 2.0, 26 February 2024: https://www.nist.gov/publications/nist-cybersecurity-framework-csf-20

S8. NIST, SP 800-218A: Secure Software Development Practices for Generative AI and Dual-Use Foundation Models, 26 July 2024, updated 25 June 2025: https://www.nist.gov/news-events/news/2024/07/secure-software-development-practices-generative-ai-and-dual-use-foundation

S9. CISA, 2023–2024 Roadmap for Artificial Intelligence, November 2023: https://www.cisa.gov/sites/default/files/2023-11/2023-2024_CISA-Roadmap-for-AI_508c.pdf

Explore more articles by Mani Padisetti 

Why Clean CRM Data is the Key to Unlocking AI’s Potential

Mani Padisetti: DNA Data Storage – Nature’s Solution to Our Digital Storage Crisis

Blockchain and AI: A Partnership to Mitigate Risk and Build Trust

AI as a Supercharged Assistant: Unlocking the Power of Technology with a Simple Mental Model

Eric Hill, Editor-in-Chief, CIO TechWorld

As a seasoned professional with over a decade of experience in technology and business magazine publication, I’ve had a front-row seat to witness the astounding pace at which the industry has evolved, often outstripping even Moore’s Law. My passion lies in crafting engaging technology articles that invite readers to immerse themselves in the dynamic and ever-changing world of technology. With each piece, I strive to create a window into this world, offering a glimpse of the amazing events and breakthroughs that continue to shape our future.