You Can't Claude Your Way Through Healthcare

Customers are becoming much more sophisticated with AI. A year ago, many organizations were still asking whether they should use large language models at all. Today, their teams are already experimenting with Claude, ChatGPT, Cursor and other AI tools themselves. And frankly, they should be.

You and I know that Claude is remarkably good. Give it a spreadsheet and it can analyze it. Give it a browser and it can navigate it. Same thing for an unfamiliar process, it can help understand it. It can write code, reconcile documents, prototype internal tools and turn an idea into something useful remarkably quickly.

For exploration, one-off analyses, prototypes and unfamiliar problems, frontier models are extraordinary.

The problem starts when the prototype becomes the process.

A useful AI result is not yet a reliable workflow

Imagine asking Claude to review a claims spreadsheet, identify missing information and update several fields. The first result may be excellent.

But tomorrow, when the column headers change or one payer uses a different identifier, it becomes a whole new ballgame. Imagine two records containing conflicting dates or a duplicate being only partially duplicated.

You've all experienced a portal timing out halfway through processing, what happens then?

Or a team member inspecting a result, changing the prompt and trying again.

A production system cannot work that way.

It needs to know what constitutes a valid input. It needs rules for conflicting information. It needs to preserve the original data, determine when to retry, recognize when it should stop, escalate exceptions and maintain a complete record of what happened.

This is where the difference between frontier AI and vertical AI infrastructure becomes important.

Frontier models provide extraordinary intelligence. Infrastructure makes that intelligence dependable. At XY, we're building that infrastructure.

The model should not become the process

There is a temptation today to keep improving the prompt with more context, more memory, tools, skills, and potentially add another agent combined with another layer of orchestration.

All of those things can make an AI agent more capable. But capability and reliability are not the same thing.

More intelligence does not automatically create more reliability.

If a process must execute thousands of times, in exactly the right sequence, across multiple systems, with financial, regulatory or patient consequences, "the model will probably follow the instructions" is not a sufficient operating model.

An agent is excellent at deciding how to approach something unfamiliar.

A workflow is excellent at ensuring that something understood happens the way it is supposed to happen. And the more highly regulated the industry, the more that distinction matters.

Better intelligence does not eliminate operational complexity

Healthcare illustrates this perfectly. A healthcare workflow might touch an EHR, payer portal, clearinghouse, spreadsheet, fax system, billing platform and CRM before it is complete. The model may understand the task perfectly but that doesn't mean the systems cooperate.

Especially when you frequently have interface changes, APIs fail, spotty data, business rules varying by payer, patient, contract and organization. And we're not even starting about human made exceptions and policies change. Someone needs to know exactly what happened six months later and this is the messy layer that general-purpose intelligence does not make disappear.

In fact, as models become better, this layer becomes more visible and the bottleneck isn't "Can the AI understand what needs to happen?" but more "Can the organization execute it reliably, repeatedly and safely?"

We encounter the same boundary ourselves

At XY.AI, we do use frontier AI extensively. Claude Code, Cursor, browser tools, scripts and internal agents help us understand problems, write software and create internal processes dramatically faster than before. They are often the fastest way to discover how something should work.

Then something interesting happens because people start depending on it and the automation runs more often. We rely on it until edge cases appear. That is why we need to know when to know whether yesterday's run succeeded, if someone changed a rule.

The question is no longer "Can Claude do this?" but "What would it take to make this dependable enough to become part of the operation?"

That's the point where an experiment needs the right infrastructure.

At XY.AI, we're not asking the model to be the workflow

We use AI to help build and operate workflows without asking the model to be the workflow.

The workflow defines the sequence, permissions, business rules, validations, retries, exception paths and completion criteria. Those elements are deterministic and versioned.

AI enters at the places where intelligence genuinely adds value: interpreting an unfamiliar document, resolving ambiguity, classifying an exception, matching inconsistent records or helping understand why something failed.

The system then validates the result before continuing.

This structure gives us something equally important: observability.

Routine work follows a routine path, unfamiliar situations receive intelligence, high-risk decisions receive appropriate human review.

And when the external environment changes, the system can detect the difference and remediate rather than silently drift.

We believe models like Claude can be a component, not the control plane

Claude can help design a workflow.

It can write the code, interpret an exception, build an integration, explain an anomaly. Those are powerful capabilities, and they'll only get better.

But it can't and shouldn't become the operational control plane for your critical workflows. That's where AI infrastructure is critical.

But there is good news here: the future isn't a choice between frontier models and vertical AI infrastructure but rather the combination.

Do you have a war story about how Claude didn't quite meet the mark that you'd like to share? Reply and tell us your adventure via eileen@xy.ai!