Trustworthy AI in Health: Governing a Double-Edged Sword

Artificial intelligence (AI) is increasingly embedded in healthcare and public health, extending from diagnostic support and clinical decision-making to surveillance, drug discovery, and patient-facing engagement tools. This diffusion occurs alongside a growing recognition that AI systems, whatever their demonstrated benefits, introduce challenges, such as algorithmic bias, opaque decision-making, and over automation, and where AI’s rapid adoption expands risk at a speed unseen in earlier digital health technologies. In high-stakes clinical and public health contexts, these risks are not theoretical; they carry direct implications for patient safety, equitable access to care, and public trust in health systems.

AI in healthcare therefore increasingly presents a double-edged sword: a technology capable of both advancing and undermining health outcomes, depending on how it is governed, implemented, and monitored.

This tension now reaches a critical juncture as the EU AI Act, the world's first comprehensive horizontal legal framework for AI, enters into force. It is against this backdrop that the Global Health Hub Germany’s 2025/2026 Working Group on Artificial Intelligence in Health produced the policy brief Trustworthy AI in Health: Governing a Double-Edged Sword.

For the health sector, the Act sets important new requirements. Yet the Working Group's analysis finds that meeting them does not automatically ensure AI systems perform equitably, remain transparent, or keep patients informed when AI is shaping their care. 

The brief's central analytical contribution is to probe the relationship between compliance under the EU AI Act and the less legally codified concept of trustworthiness. It finds, through empirical case analysis, that the two are not necessarily coextensive and that trustworthy AI governance requires additional regulatory action to complement the Act’s legal compliance.  

The Working Group examines two AI tools already in active use to demonstrate this: an AI-based "second reader" deployed within Germany's national mammography screening program, and an AI-supported chatbot used to triage referrals within the United Kingdom’s NHS. Both systems have delivered real, measurable benefits — faster reads, higher detection rates, improved referral completion. Both also show clear governance gaps: limited independent evidence, minimal disclosure to patients that AI is involved in their care, and little public infrastructure tracking how these systems perform once deployed.

As a core part of its analysis, the brief uses the FUTURE-AI framework — an internationally developed, expert-consensus guideline — to assess each case study across six principles: fairness, universality, traceability, usability, robustness, and explainability. It then pinpoints where existing implementation, and legal obligations fall short and offers targeted actions on what can be done.

Recommendations center on three priorities: keeping patients and clinicians at the heart of AI-supported care, strengthening meaningful human oversight and public accountability, and establishing governance structures flexible enough to monitor AI performance long after a system first reaches the market.

The care and investment placed on domestic governance mechanisms should inform credible participation in shaping international norms. Recommendations therefore focus on the strategic near-term to support decision-making by Germany's Federal Ministry of Health (BMG) and other relevant federal bodies as Germany operationalizes the EU AI Act for its own health sector and in its international health engagements and involvement in global health governance.

This policy paper is intended to inform as the country moves from legal transposition of the EU AI Act toward sector-specific implementation in health.

Trustworthy AI in Health: Governing a Double-Edged Sword is published by Global Health Hub Germany, hosted by the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH.

The article and policy brief are products of the Hub Working Group on AI in Global Health, which served as the Global Health Hub Germany’s cross-community annual theme for 2025/2026. The Hub Community on Global Women's Health has also covered the topic in a paper on “Feminist Perspectives on AI Deployment in Women's Health".

 For the upcoming annual theme, Hub communities have selected Universal Health Coverage (UHC). Members interested in contributing to the next phase of cross-community collaboration are warmly invited to reach out to Katrin Würfel (katrin.wuerfel(at)globalhealthhub.de).

For more information about this brief or the GHHG communities that produced it, please reach out to info(at)globalhealthhub.de.