AI should remove repetitive work, not meaningful work. That line is easy to say and hard to hold when a demo is impressive.

Where it actually helped in HealthFirstAI: turning a doctor’s spoken rounds into structured fields that populate the chart, the medication order, and the discharge summary with the same information. That’s repetitive work — the same clinical fact, retyped into five formats. Removing it is the whole point of the product.

Where it looked good in a demo and needed a harder second look: anything that touches a clinical decision itself, not just the documentation of one. A model can draft a summary. It should never quietly draft a diagnosis and let a tired clinician click accept.

So we decided that when something looks wrong or unusual, the system should flag it, not fix it.

If someone enters a temperature of 197, the system should ask, “Are you sure?” It shouldn’t silently change it.

If a diagnosis was entered earlier and a later entry doesn’t match it, the system should flag the mismatch. It shouldn’t decide which one is correct.

The source of truth is always the doctor or the user.

That’s the line I keep coming back to: AI can point out something that looks wrong. It shouldn’t decide what is right.

System works mostly in retrieval — pulling the right prior note, the right protocol, the right past result, so a clinician isn’t searching for it manually. That’s still assistance.

The moment it starts deciding instead of retrieving, it needs a different level of review, not just a better prompt.