Building Trust in AI-Enabled Healthcare Software: Lessons from MedicentreV3 HMIS
There is a version of AI in healthcare that looks impressive in a boardroom and fails quietly in a ward.
It is the version where engineers optimize for accuracy metrics, procurement teams tick the innovation box, and then the system gets deployed - only for nurses to work around it, doctors to ignore its suggestions, and the whole investment to sit technically functional but practically useless.
This failure mode is more common than the industry admits. And it doesn't happen because the AI was wrong. It happens because the people it was built for never trusted it.
At Hanmak Technologies, we have spent more than 17 years building MedicentreV3, a hospital management system now deployed across more than 250 health facilities in East and West Africa. We know what it takes to get a system adopted in a busy Kenyan district hospital, a faith-based clinic in Uganda, or a private facility in Ghana. We know how doctors and clinicians work, what they tolerate, and what they reject.
Now we are building AI into MedicentreV3 - a triage assistant, a clinical diagnosis support tool, and a prescription safety checker. And the hardest part isn't the AI. The hardest part is trust.
This is what we are learning.
Why Trust Is the Real Problem in Healthcare AI
Let's start with the uncomfortable reality.
Healthcare AI has produced some genuinely remarkable results in controlled research settings. Algorithms have matched, and in specific tasks, exceeded, specialist-level accuracy in reading radiology images, detecting diabetic retinopathy, flagging sepsis risk, and identifying certain cancers. The science is real.
But the gap between research performance and clinical adoption is vast, and it is not primarily a technical gap.
A 2023 survey of over 1,400 clinicians across multiple countries found that fewer than one in three trusted AI diagnostic tools enough to act on their recommendations without independent verification. That's not a problem of accuracy. That's a problem of confidence, transparency, and relationship.
When a senior consultant makes a clinical recommendation, a junior doctor knows how to calibrate their trust. They know the consultant's background, their specialty, their track record. They can ask why. They can push back. They can see the reasoning.
When an AI system surfaces a recommendation, none of that scaffolding exists by default. The system says sepsis risk: high or consider amoxicillin and provides no visible reasoning, no uncertainty range, no acknowledgment of what it might have missed. It expects to be trusted without earning it.
That's not how trust works. Not between people, and not between people and machines.
The Stakes Are Higher in Africa
Clinical AI is being developed primarily by teams in North America and Europe, trained predominantly on datasets from those populations, and then marketed globally, including into African healthcare markets that have fundamentally different patient demographics, disease profiles, and clinical contexts.
The implications matter.
A prescription safety algorithm trained mostly on Western drug formularies may not flag the same interaction risks relevant in a Kenyan formulary. A sepsis prediction model trained on hospital data from a country with high rates of viral illness will behave differently in a context where malaria and typhoid are common confounders. A triage tool optimized for emergency departments with full diagnostic capability will give different outputs than one designed for facilities where an X-ray machine may be the primary imaging resource.
This is not a hypothetical concern. It is a well-documented limitation of AI systems deployed outside their training context, and it is why health tech companies building for the African market have a distinct responsibility to validate their AI against local data, local workflows, and local disease contexts.
For us at Hanmak, this shapes every decision we make as we add AI to MedicentreV3. We are not importing an algorithm. We are building features for clinicians who work in specific environments, with specific constraints, treating specific patient populations. The AI has to be right for them, not right in aggregate across a global dataset.
Six Principles We Are Building Around
After extensive engagement with clinicians across our deployed facilities - nurses, clinical officers, general practitioners, pharmacists, hospital administrators - six principles have emerged as non-negotiable if AI features are going to be trusted and actually used.
1. Design for the User, Not the Algorithm
The most technically sophisticated AI feature is worthless if it adds friction to a workflow that is already under pressure.
Clinicians in African hospitals are frequently managing more patients per session than their counterparts in high-income settings. They are working with constrained time, limited diagnostic equipment, and support staff stretched thin. Any AI tool that adds steps, creates uncertainty, or demands attention in a way that disrupts flow will be ignored — quickly and permanently.
We design AI features around the workflow first. Where does this information need to appear? How many clicks does it require? What happens when the clinician agrees with the suggestion, and what happens when they don't? The algorithm is built to fit the workflow, not the other way around.
Role-based interface design matters here. A nurse using the triage assistant has different needs, different vocabulary, and a different decision-making process than the prescribing clinician who receives the patient afterwards. The AI surface they interact with must reflect that. One interface for all users is a shortcut that produces a poor experience for everyone.
2. Explain, Don't Just Suggest
"Our AI recommends X" is not a useful clinical communication.
"Based on the patient's temperature, heart rate, and presenting symptoms, there is an elevated likelihood of bacterial infection - consider a broad-spectrum antibiotic pending culture results" is a useful clinical communication.
The difference is reasoning. When a clinician can see why an AI is making a recommendation, they can do something that the AI cannot: apply judgment. They can ask themselves whether the reasoning fits this specific patient, whether there are factors the system hasn't accounted for, whether the suggestion aligns with what they are observing at the bedside.
Explainability is not a feature for regulators or ethics boards, though it matters for those purposes too. It is a feature for the clinician standing in front of a patient who needs to make a decision they can stand behind. Every AI recommendation in MedicentreV3 will show its reasoning, not as a technical dump, but as a clear, readable justification that a busy clinician can process in seconds.
3. Keep Humans in the Loop - Always
This principle sounds obvious. In practice, it shapes every design decision about how AI outputs are presented. An AI suggestion that requires one-click acceptance to execute without review is a system designed to replace clinical judgment.
An AI suggestion that surfaces as an input to a decision the clinician makes - and that requires active confirmation before any action is taken - is a system designed to support it.
The distinction matters enormously, both for safety and for trust. Clinicians are more likely to engage with, evaluate, and ultimately trust an AI system when they feel in control of what it produces. When they feel the system is trying to automate their role rather than assist it, they disengage.
Our design rule: every AI recommendation is a prompt, not an action. The clinician remains the author of every clinical decision. The AI is the well-read colleague offering a second opinion, never the one writing the prescription.
4. Be Honest About What the AI Doesn't Know
No AI system is equally confident across all inputs. An algorithm that has seen thousands of cases presenting with a classic pattern of symptoms will be more reliable than one facing an atypical presentation, an unusual comorbidity, or a data input that is incomplete or inconsistent.
The problem is that many AI systems don't communicate this variation in confidence. They surface a recommendation with the same visual weight whether they are 94% confident or 61% confident. Clinicians, not knowing the difference, either over-trust or, once burned by a bad recommendation, stop trusting at all.
We display uncertainty explicitly. When the AI's confidence is low, it says so - and it tells the clinician why. Recommendation confidence is limited due to incomplete vital signs or this presentation does not closely match training data - verify manually are messages that look like failure. They're actually trust-building. A system that knows what it doesn't know is a system worth trusting.
5. Train for the Tool, Not Just the Technology
Rolling out a new AI feature without structured clinical training is one of the fastest ways to kill adoption.
The challenge isn't teaching clinicians how to click buttons. It's helping them develop an appropriate mental model for what the AI is doing, what its limitations are, and how to use it as a tool rather than an authority.
Clinicians who understand why an AI might be wrong in certain cases are better equipped to catch those cases. Clinicians who understand the training data basis of a recommendation are better equipped to apply it judiciously. This level of AI literacy doesn't happen through a one-page manual at system launch.
We are delivering role-based training through our Learning Management System - purpose-built for how clinical staff learn, not how software engineers think. And training is not a one-time event. As AI features evolve, as clinicians gain experience with them and surface new questions, training must evolve too.
6. Measure What Matters - Then Improve
Acceptance rates and override rates for AI recommendations are not vanity metrics. They are a diagnostic tool.
If a triage AI is suggesting interventions that clinicians consistently override, there are two possible interpretations: the AI is wrong, or the clinicians don't trust it. Distinguishing between those two requires qualitative data - asking clinicians why they overrode the recommendation, not just measuring that they did.
This feedback loop is how AI gets better in deployment. The algorithm trained on historical data is a starting point. What it learns from real-world clinical use - which suggestions land, which get overridden and why, which contexts produce confident versus uncertain outputs - is how it becomes genuinely useful over time.
We are building override reason tracking and usability scoring into every AI feature from the start, not as an afterthought. The learning doesn't stop at launch.
What This Means for Healthcare Leaders Evaluating AI Tools
If you are a hospital administrator, clinical director, or health system decision-maker considering AI-enabled software, the questions below are the ones that matter.
On transparency: Can the system explain its recommendations in clinical language, or does it only output a result? If a clinician asks why, does the system have an answer?
On uncertainty: Does the system communicate confidence levels, and does it flag when a recommendation is based on incomplete or ambiguous data?
On workflow: Was the interface designed with input from clinicians who work in environments similar to yours? Or was it designed by engineers and tested in a very different context?
On local validation: Has the AI been tested on patient data and disease profiles relevant to your population? A global accuracy figure is not the same as accuracy in your ward.
On accountability: When the AI is wrong — and at some point, it will be — who is responsible? How does the system support the clinician in making defensible decisions rather than blindly following algorithmic outputs?
On training: What is the plan for building AI literacy among your clinical staff? Not just how to use the feature, but how to use it well — including knowing when not to use it?
The answers to these questions are more revealing than any demo.
The Longer Arc
We are still early in this journey at Hanmak. The AI features we are building into MedicentreV3 will be better in 18 months than they are today, and better again in three years. That's the nature of machine learning systems - they improve with data, with feedback, and with the honest acknowledgment of where they are falling short.
What we are not willing to compromise on is the human relationship at the centre of all of it. Healthcare AI that earns clinician trust, by being transparent, honest, explainable, and designed around real workflows, will change outcomes in meaningful ways. Healthcare AI that doesn't earn that trust will be routed around, no matter how technically accurate it is.
The algorithm is the easy part. The trust is the work.
