Worked analysis · hypothetical

Licensing an imported AI triage system

A health ministry in a middle-income country must decide whether to license an AI triage system that was trained and evaluated elsewhere.

Setting. A ministry with limited technical staff, a vendor headquartered abroad, patient data governed by national law, and clinics under pressure to expand capacity.

National

Procurement
Deployment
Monitoring

01
What outcome is being shaped?
The outcome is a pattern of triage decisions across the whole system: which patients are prioritized, whose conditions are systematically under-recognized, and whether the ministry becomes dependent on a vendor it cannot audit. The benefit sought is real: more patients seen with scarce clinical capacity.
Broadly shared benefit Affected people Deploying organization
02
What incentives currently produce or reinforce that outcome?
The ministry is rewarded for expanding coverage quickly and penalized for visible delay. The vendor is rewarded for closing the contract and has little incentive to fund local evaluation. Nothing in the current environment rewards demanding evidence on the local population.
Procurement leverage Deployment pressure AI developer or provider
03
Which institution has the authority to act?
The ministry, as licensing authority and deployer, holds the formal power. A medicines or devices regulator may also have jurisdiction. Whether either has the authority to require re-evaluation, audit access, or exit terms depends on the contract and the law.
Regulator or public authority Deploying organization Supervision and regulatory review
04
Does that institution have the capacity, independence, expertise, legitimacy, and accountability to act?
Ask whether the ministry has staff who can read an evaluation, access to the system to test it, independence from the vendor's technical narrative, and a way for patients to hold it to account. Formal authority without evaluation capacity is authority the vendor effectively exercises.
Evaluation capacity Regulator or public authority Accountability
05
What infrastructure does the mechanism depend on or reshape, and who controls it?
The system depends on the vendor's cloud, on patient data flowing to it, and on models trained on other populations. Who controls that data afterwards, whether the local population and languages are represented, and whether the country can switch vendors are infrastructure questions that determine the ministry's bargaining power.
Data, language resources, and data labor Compute Evaluation capacity
06
What safeguards translate the objective into practice?
Concrete safeguards: local validation on the served population before licensing, documented clinical oversight with the authority to override, logging that the ministry rather than the vendor controls, a complaint route for patients, incident reporting to the regulator, and contractual exit and audit rights.
Pre-deployment evaluation and red teaming Meaningful human oversight Traceability Complaint and appeal routes Incident reporting and post-deployment monitoring
07
Where in the AI lifecycle does the intervention operate?
The decisive moment is procurement, because leverage disappears once the system is embedded. Governance then continues through deployment and monitoring, where the pattern of triage outcomes actually appears.
Procurement leverage Incident reporting and post-deployment monitoring
08
Who decided, on what authority, and who was not represented?
The ministry decides, on a public mandate, but the evaluation population was chosen elsewhere, the terms were drafted by the vendor, and the patients and clinicians who will live with the system were not consulted. The international standards the vendor cites were written by bodies the country may not be party to.
International institution Civil society and independent researchers Affected people
09
Who bears the costs and risks, and who captures the benefits?
Patients bear the risk of systematic error, particularly groups under-represented in the training data. The vendor captures the revenue and the data. The ministry bears compliance and dependence costs. Clinicians bear the burden of overriding a system they were told performs well.
Affected people Broadly shared benefit Meaningful human oversight
10
What happens when the mechanism or technology scales across organizations, sectors, or borders?
If the vendor licenses the same system across many countries, a single flaw becomes a multinational pattern, and each ministry negotiates alone against a supplier with far more information. Pooled evaluation capacity across countries changes that asymmetry.
Concentration of power International institution Evaluation capacity
11
What becomes difficult to reverse if governance arrives too late, and how will the governance mechanism itself be revised if it turns out to be wrong?
Once clinics reorganize around the tool and staff skills atrophy, withdrawal becomes a service collapse rather than a decision. A licence with a review date, retained data rights, and a tested exit path keeps the choice a choice; a rigid ban forecloses a real benefit. Either way, the mechanism must say how it will be revised.
Lock-in Review and sunset provisions Reversibility window

What the analysis reveals

Performance is necessary and nowhere near sufficient. The ministry's real question is whether it has the capacity, leverage, and infrastructure to contest the vendor's conditions, and whether it keeps that ability after the system is embedded.

Sources drawn on

Governing AI for Humanity: Final Report
United Nations High-level Advisory Body on Artificial Intelligence · report · publication date not recorded · Final report, September 2024 (eISBN 9789211067873)
An advisory body report representing a majority consensus of its members in their personal capacity. It is not a UN member-state decision. Page locators refer to the PDF's printed page numbers.
un-hlab-governing-ai · Last checked 2026-09-06 · claims and passages
Recommendation on the Ethics of Artificial Intelligence
UNESCO · recommendation · published 2021-11-23 · Adopted text, 23 November 2021
unesco-ai-ethics · Last checked 2026-09-04 · claims and passages
Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law (CETS No. 225)
Council of Europe · treaty · published 2024-09-05 · Opened for signature 5 September 2024
The Council of Europe overview page was reviewed, not the treaty text itself. The Convention binds only parties for which it has entered into force, and is largely framed as obligations on states to adopt measures. Signature and ratification status changes; verify before relying on it. The treaty text itself (CETS 225) was opened on 19 October 2026 for the definitions comparison.
coe-framework-convention · Last checked 2026-09-06 · claims and passages
Artificial Intelligence Risk Management Framework (AI RMF 1.0)
National Institute of Standards and Technology · framework · published 2023-01-26 · 1.0
nist-ai-rmf-1 · Last checked 2026-09-04 · claims and passages

What outcome is being shaped?

What incentives currently produce or reinforce that outcome?

Which institution has the authority to act?

Does that institution have the capacity, independence, expertise, legitimacy, and accountability to act?

What infrastructure does the mechanism depend on or reshape, and who controls it?

What safeguards translate the objective into practice?

Where in the AI lifecycle does the intervention operate?

Who decided, on what authority, and who was not represented?

Who bears the costs and risks, and who captures the benefits?

What happens when the mechanism or technology scales across organizations, sectors, or borders?

What becomes difficult to reverse if governance arrives too late, and how will the governance mechanism itself be revised if it turns out to be wrong?