Real-Time AI Agents: The Architecture Behind Google AMIE
Sharan Sifat18 min read1 viewsGoogle's AMIE (Video) matched board-certified physicians across 100 clinical scenarios. The transferable lesson isn't medical — it's the asynchronous three-agent split that lets a real-time AI agent talk fast while it reasons slowly.
On this page
Google has published results from a study in which its research medical AI, AMIE, ran live video consultations with patient actors. The headline finding is that an independent panel of physicians rated it on par with board-certified primary care doctors. The finding that actually matters if you are building software is quieter: Google could not make this work with a single model. It had to split the system into three agents running asynchronously. That architectural decision is the most reusable thing in the whole study, and it applies to almost any product that puts real-time AI agents in front of a live human being.
The reason is stated plainly in Google's own write-up. A single agent cannot hold a natural conversational rhythm while simultaneously doing careful clinical reasoning and continuously parsing an audio-visual stream. Those three jobs have incompatible timing requirements. Reply latency is measured in hundreds of milliseconds. Good reasoning takes seconds. Perception never stops at all. Force them into one request/response loop and you get a system that is either fast and shallow or thoughtful and unusable.
This article is not a medical explainer. It is a look at what the AMIE architecture teaches founders and product owners about designing live AI features — voice agents, video assistants, guided onboarding, real-time triage — and how to scope one without discovering the latency problem three months into the build.
- Google's AMIE (Video) was rated on par with primary care physicians across 100 clinical scenarios — but only in a simulated study with professional actors.
- The system works because it splits into three asynchronous agents: a Talker, a Planner and a Perception agent.
- The underlying rule is general: conversation, reasoning and perception run on different clocks and must not share one loop.
- In your own product, write the latency budget before the feature list — it decides your architecture more than your model choice does.
- Google explicitly says this is not yet evidence that AMIE can safely treat real patients. Copy the architecture, not the claim.
What Google actually tested — and what it did not prove
Before borrowing any lesson from a study, it is worth being precise about what the study did. A lot of AI coverage compresses "performed well in a controlled simulation" into "as good as a doctor," and those are very different statements.
The study, in plain terms
Google ran a video-based OSCE — an objective structured clinical examination, the same simulated-patient format medical schools use to assess trainees. Fifteen trained actors performed 100 clinical scenarios across five body systems: cardiopulmonary, abdominal, HEENT (head, eyes, ears, nose and throat), neurological and psychiatric, and musculoskeletal. In total that produced 300 standardised consultations.
There were three arms. AMIE (Video), the new multimodal system. AMIE (Text), the earlier text-only version. And ten board-certified primary care physicians using the same video interface. A separate, independent panel of twenty experienced primary care physicians then scored every consultation against established clinical rubrics, without being the same people who took part.
- 100clinical scenarios tested
- 300standardised consultations
- 5body systems covered
- 30physicians involved across arms and scoring
That is a genuinely well-controlled design. Three arms, blinded scoring against fixed rubrics, and a comparison group of real clinicians using the identical interface rather than a hypothetical baseline. It is a much stronger setup than the vendor demos most AI capability claims rest on.
Where AMIE (Video) scored highest
The evaluators rated AMIE (Video) as equivalent to the physician group on the four dimensions you would expect to matter most: history-taking thoroughness, diagnostic accuracy, management appropriateness and communication quality. It also matched or exceeded the text-only version of itself on those same measures, which tells you the video channel added signal rather than just noise.
The more interesting result is where it beat the humans. Evaluators rated the video system higher than both the physicians and the text-only AMIE at eliciting physical signs and at proactively guiding actors through virtual examination manoeuvres — asking someone to turn toward the light, press a specific spot, move a joint through its range, hold the camera at a particular angle.
That is a revealing kind of win. It is not a reasoning victory. It is a patience and consistency victory. A system that never gets tired, never runs behind schedule, and never assumes it already knows the answer will ask the fifth follow-up question that a busy human skips. Patient actors also reported preferring the video interface to text chat, rating it as easier to use and more effective for communicating their concerns, and rated AMIE favourably on empathy and rapport.
The limits Google states itself
Google is unusually direct about the ceiling on these results, and it is worth repeating because it is the part that gets dropped in summaries. Professional actors cannot reproduce the variability of real patients. Scenarios were restricted to conditions an actor could authentically portray — which excludes some of the presentations where audio-visual perception would carry the most diagnostic value. Targeted automated testing still surfaced occasional perception and reasoning errors. Intermittent technical issues disrupted conversational naturalness. And Project Astra, the underlying real-time system, remains a prototype.
Google's position is that real-patient research is the next stage, not a completed one. The study provides controlled evidence about consultation behaviour and physical-examination guidance. It does not establish that AMIE can safely diagnose or manage real patients in production. If someone pitches you an AI health product citing this study as clinical validation, that is a misreading of the source.
Separately, Google notes a completed feasibility study with Beth Israel Deaconess Medical Center looking at safety and utility in practice, and an ongoing nationwide randomised study with Included Health evaluating the text-based AMIE in real virtual care. Those are the studies that will eventually answer the real-world question. This one answers an architectural one.
Why one model could not do the job
Here is the sentence from Google's write-up that should interest every product team: a single agent cannot currently sustain natural conversational response times while also conducting detailed reasoning and continuously processing audio-visual input.
Read that as an engineering constraint rather than a medical one and it describes a problem we hit on ordinary commercial builds constantly.
The latency wall
Human conversation has a tight tolerance. In natural speech, the gap between one person finishing and the next starting is short — a pause much beyond that reads as hesitation, confusion or a dropped connection. People start talking over each other, or repeat themselves, and the interaction degrades fast.
Now consider what you are asking a model to do inside that window if you build the naive version. Transcribe the audio. Interpret the video frames. Update its understanding of the situation. Reason about what is still unknown. Decide what to ask next. Generate a reply. Synthesise speech. Every one of those steps is defensible on its own. Chained inside a single turn, they blow the budget before you have added any of your own business logic.
The instinct at this point is to reach for a faster model. That helps at the margins and then stops helping, because the problem is not raw speed — it is that you have put work with a 200-millisecond deadline and work with a five-second deadline in the same queue. No model tier fixes a scheduling mistake.
Talker, Planner, Perception: who owns what
AMIE's answer is to give each clock its own agent, running asynchronously.
The Talker is the patient-facing agent. Its only job is responsive, low-latency spoken interaction — keeping the conversation flowing naturally while incorporating whatever guidance the other agents have produced so far. It is not where the deep thinking happens.
The Planner runs in the background and owns the clinical reasoning. It continuously refines the differential diagnosis and management plan, identifies what information is still missing, and re-prioritises competing goals. It is allowed to take its time, because nobody is waiting on it mid-sentence.
The Perception agent continuously reviews the audio and video streams, identifying clinically relevant non-verbal cues — visible signs of distress, physical findings, auditory signals — and contextualising them against the conversation so far. It never blocks a reply; it enriches the shared picture.
The breakthrough is not that the model got smarter. It is that the system stopped asking one model to be fast and thoughtful in the same breath.
The pattern: separate the clocks in your system
Strip out the medicine and what remains is a general design pattern for live AI. You have a fast loop that must never stall, a slow loop that must never be rushed, and a continuous loop that must never block either of them. Almost every real-time AI product has these three, whether or not the team has named them.
Fast loop, slow loop
The fast loop is the one the user experiences directly. It answers, acknowledges, holds the thread, buys time gracefully. It should run on a small, quick model with a tight prompt and a short context window, and it should be able to produce a competent turn using only what is already in shared state. If your fast loop ever has to wait for a fresh analysis before it can speak, you do not have a fast loop.
The slow loop is where judgement lives. It reads the full transcript and the accumulated observations, does the expensive reasoning, and writes its conclusions back into shared state as structured data — current hypothesis, open questions, recommended next action, confidence. It runs on a schedule or on triggers, not on every user utterance.
The continuous loop watches whatever stream your product has. Video, screen share, sensor data, a live document, a stream of events. It writes observations into the same shared state.
What is allowed to be stale
The design question that decides whether this works is deceptively simple: what is your fast loop allowed to be wrong about, and for how long?
In AMIE's case, the Talker can operate on a differential diagnosis that is a few seconds out of date, because asking one more clarifying question based on slightly old reasoning is harmless. It could not operate on a stale safety assessment, because telling someone the wrong thing about chest pain is not harmless.
Every product has this line. A sales agent can quote a slightly stale summary of the prospect's needs but must never quote a stale price. A support agent can improvise rapport but must never confirm a refund before the slow loop has checked the policy. Draw that line explicitly, in writing, before you build. It determines what the fast loop is permitted to say on its own and what it must stall on — and "stall gracefully" is a feature you design, not an error state you discover.
Give your fast loop a small set of honest holding moves: acknowledging, reflecting back what it heard, asking a low-risk clarifying question. These fill the gap while the slow loop catches up, and they are indistinguishable from good conversational manners. A system that says "let me make sure I have that right" is buying two seconds and sounding more competent while it does.
How to design a real-time AI agent this way
Here is the sequence we work through when a client asks for a live voice or video AI feature. It front-loads the decisions that are expensive to change later.
Write the latency budget before the feature list
Decide, in milliseconds, what "responsive" means for your users, then subtract everything you do not control: network round trips, speech-to-text, text-to-speech, and the media transport itself. If you are building on the browser, the WebRTC APIs handle the real-time media path, and their behaviour on poor connections is a bigger driver of perceived quality than your model choice. Whatever is left after those subtractions is your actual budget for inference. It is usually much smaller than teams expect, and seeing that number early prevents an architecture built on wishful thinking.
Name the three jobs in your own domain
Write one sentence for each agent describing what it owns and what it is explicitly not responsible for. This sounds like paperwork and it is the highest-leverage half hour in the project. Most failed live-AI builds we are asked to rescue have one prompt trying to do all three jobs, with instructions that quietly contradict each other — be concise, be thorough, respond instantly, consider everything.
Pick a model tier per agent, not per product
Once the jobs are separated, they stop needing the same model. The Talker needs speed and can be small. The Planner needs reasoning quality and can be slower and more expensive because it runs less often. The Perception agent needs multimodal input and runs constantly, which makes it the one most worth optimising for cost.
| Agent | Owns | Timing requirement | Model profile | What breaks if you get it wrong |
|---|---|---|---|---|
| Talker | The live conversation and its rhythm | Sub-second, every turn | Small, fast, tight context | Dead air; users talk over the agent and lose trust |
| Planner | Reasoning, next-best-action, open questions | Seconds; runs on triggers | Strongest reasoning tier | Confident, fluent, wrong — the worst failure mode |
| Perception | Continuous stream analysis and observations | Always on, never blocking | Multimodal, cost-optimised | Runaway inference bill, or missed context entirely |
| Shared state | The single record all three read and write | Instant reads | Not a model — a data structure | Agents contradict each other in front of the user |
That last row is the one teams skip. The three agents are not really the architecture — the shared state between them is. It needs a defined schema, clear ownership of each field, and a rule for what happens when the Planner's conclusion contradicts what the Talker already said out loud. Decide that in design, not in production.
Where this architecture pays off outside healthcare
The Talker/Planner/Perception split is not medical. It fits any interaction where someone is doing something live and an AI needs to both keep up and think ahead.
Live support and sales calls
The Talker handles the call. The Planner reads the account history, the current issue and your policy documents, and decides what the agent should be steering toward. The Perception agent listens for tone, frustration, hesitation and the moment the customer says something that should trigger escalation. The commercially valuable output is often not the conversation at all — it is the structured record the Planner leaves behind.
Inspection, claims and guided onboarding
Anywhere a user points a camera at something and needs to be walked through a process, the same shape applies. Property and vehicle inspections, insurance claims documentation, equipment troubleshooting, identity and compliance onboarding. The AMIE result about guiding people through examination manoeuvres transfers directly here: a patient, methodical guide that never skips step four is worth more than a brilliant one that assumes you already did it.
The honest caveat is that this architecture costs meaningfully more to build than a chat feature. If your interaction does not genuinely need to be live, do not make it live. Plenty of the value in these use cases can be captured with an asynchronous flow — upload the video, get an analysis back in thirty seconds — for a fraction of the engineering. We cover that trade-off in more depth in our guide to what to automate versus what to keep human.
What it takes to build one
The parts you actually pay for
In the projects we scope, the model API is rarely the largest line. The real cost sits in four places. First, the real-time media pipeline — transport, audio handling, reconnection logic, and behaving sensibly when someone's connection degrades mid-call. Second, shared state and orchestration, which is ordinary backend engineering and is where most of the subtle bugs live. Third, the evaluation harness, covered below. Fourth, observability: when a live agent behaves oddly, you need the transcript, the state at each turn and the reasoning trace, or you cannot debug it at all.
Inference cost deserves its own note. The Perception agent runs continuously, which makes it the component most likely to produce a bill nobody forecast. Sampling rate, resolution and how aggressively you skip unchanged frames are product decisions with direct financial consequences, and they belong in the scope document rather than being discovered in month two. Licensing terms matter here too — we looked at how quickly those can shift in our piece on open-source AI licensing and revenue sharing.
A phasing plan that does not blow the budget
The sequence that works is to build the slow loop first. Get the Planner producing genuinely good structured reasoning over transcripts with no live interaction at all — you can develop and evaluate that against recordings. Then add the fast loop and shared state, and confirm the conversation holds together when the Planner is deliberately delayed. Add perception last, because it is the most expensive to run and the easiest to over-engineer before you know which cues actually change the outcome.
Doing it in the other order — starting with the impressive live demo — is how teams end up with something that demos beautifully and cannot be shipped. If you are still deciding whether a live agent is the right shape at all, our overview of AI features you can add without breaking the budget covers the cheaper options honestly, and the wider AI Applications cluster goes deeper on each.
The evaluation work founders skip
Notice how much of Google's effort went into measurement rather than model work: 100 scenarios, fixed clinical rubrics, an independent panel of twenty physicians who did not take part in the consultations. That is not academic overhead. It is the only reason the results mean anything.
Build the scorecard before the demo
Decide what "good" means in your domain and write it down as dimensions you can score — the way AMIE was scored separately on history-taking, diagnosis, management and communication rather than given one overall mark. Then build a set of scenarios that covers your real distribution of cases, including the awkward ones. Score changes against that set every time you touch a prompt or swap a model.
The single most useful thing a scorecard gives you is the ability to say no to a change. Without one, every prompt tweak is an argument about vibes, and quality drifts in whichever direction the last person to edit the file was worried about.
Keep a human in the loop where it counts
AMIE is a research system operating in a domain with the strictest possible consequences, and Google's stance is still that real-patient work comes next. Your product's stakes are probably lower, but the principle scales down cleanly: identify the decisions where being confidently wrong is expensive, and route those through a person. The agent gathers, drafts and recommends; the human approves. That boundary is also what makes the system defensible when something does go wrong — and it pairs naturally with the monitoring and rollback practices we cover in our review of 2026 automation trends.
If you are building this into a commercial product, the same discipline applies to the ordinary parts of the stack — auth, billing, tenancy, uptime. A live AI feature sits on top of a normal application, and it inherits every weakness underneath it. Our SaaS development work is mostly this: making the unglamorous foundation solid enough that the interesting feature can be trusted.
Frequently asked questions
Did Google's AI actually outperform doctors?
Not in the way the phrase suggests. In a simulated study with professional patient actors, an independent panel of physicians rated AMIE (Video) as equivalent to primary care doctors on history-taking, diagnostic accuracy, management and communication, and rated it higher on eliciting physical signs and guiding virtual examination manoeuvres. Google states directly that this is not evidence it can safely diagnose or manage real patients, and that real-patient research is the next stage.
Why use three agents instead of one better model?
Because the constraint is timing, not intelligence. Conversational replies need to arrive in well under a second, careful reasoning takes several seconds, and stream perception runs continuously. Putting all three in one loop means the slowest step sets the pace for everything. Splitting them lets each run at its natural speed, with a shared state object keeping them consistent.
Do I need this architecture for a normal chatbot?
No. If your users are typing and reading, a short pause is completely acceptable and a single well-built model call is simpler, cheaper and easier to maintain. This pattern earns its extra complexity specifically when the interaction is live — voice or video — and a delay would break the experience.
What is the biggest hidden cost in a live AI agent?
Continuous perception. An agent that analyses a video or audio stream throughout a session runs inference constantly rather than once per message, and the bill scales with session length rather than with message count. Sampling rate, resolution and frame-skipping logic should be decided during scoping, because they change the unit economics of the whole feature.
How long does something like this take to build?
It depends heavily on how much of the reasoning already exists and how strict your latency target is, so any figure without a scope behind it is guesswork. What we can say is that the sequencing matters more than the estimate: teams that build the reasoning loop first and add live interaction second ship; teams that start with the live demo usually rebuild. We are happy to put real numbers against a specific scope.
Where does a human need to stay in the loop?
At any decision where being confidently wrong is expensive or irreversible — money moving, commitments being made, safety-relevant advice, anything with a regulatory dimension. The agent can gather information, draft the response and recommend an action; a person approves it. Everything else can usually run autonomously with logging.
Planning a real-time AI feature and want an honest read on whether it needs this architecture — or whether a simpler asynchronous version would do the same job for a fraction of the cost? Get a free scope and quote and we will walk through it with you.
Sources
Have a project like this in mind?
Tell us what you're building and we'll map out the scope, timeline and a fixed starting quote — no obligation.
Start your project
