How Agent Debugger Audits a Kaiya Analysis End to End

AI can produce sophisticated business analysis in seconds. But if the answer impacts real decisions—e.g. shaping launch strategy, retention program, regulatory decision—“probably right” isn’t enough. A polished answer rarely reveals where something went wrong, e.g. did an error originate at interpretation (at the beginning), at the conclusion eg recommended action or at one of the many points in between.

People typically seek a second opinion when AI fails the “sniff test”, by asking edge case questions or rerunning the analysis or asking other LLMs to check (e.g. like Karpathy’s LLM council), or several other means.

But this is a fundamental challenge for enterprise AI: how do you give users a genuine second opinion from a reviewer that can inspect not just the answer, but everything behind it? That’s what this post digs into.

The trust gap is the real product problem

Nine months ago, credible-sounding AI answers were novel enough that people trusted them outright. These days, with so much AI slop and hallucinations abounding, humans often triple check answers before presenting them, as the ultimate accountability is on them and because the stakes are higher than ever. Meaning, a wrong number doesn't just cause an awkward meeting. Agentic systems acting on the wrong number can do real harm: misroute launch budgets, flag the wrong cohort to a regulator, or send a retention team chasing customers who were never going to leave. In these organizations, "the model was confident" is not a defense anyone wants to give because confidence is exactly what modern AI is best at faking (think about the last time an LLM handed you a beautifully wrong answer with the same steady voice it uses for a right one).

Common manual checks include:

• Copy/pasting AI outputs into a different LLM chat
• Re-running the prompt with a smarter LLM model
• Testing the analysis with several edge cases to get a feel for how “sound” the analysis is
• Run it by an external subject matter expert
  ‍

But none of these are scalable. Transparency into how it got there is the only real antidotde.

Enter Kaiya's Debug Agent

At Tellius, we’ve seen a few customers do the copy/paste double check, which in our opinion is like handing over a thin slice aka the surface — the final SQL, the summary, a few numbers.

But Tellius already has the whole thing. The full trace. The reasoning behind every step. The semantic layer the query was built on. The data model underneath it. The exact SQL and Python that ran, and why. Every bit of context that outside chatbot is missing, we already hold.

So we built a better checker, right where the context lives.

Kaiya's debug agent is a parallel evaluator. When an analysis completes, it can spin up a separate agent whose only job is to interrogate that work — not to be agreeable, but to be critical. It walks the trace. It reads the reasoning. It inspects the semantic layer and the data model. It reads the SQL that actually executed, the summary and key takeaways, the skills and additional context provided, and asks whether it does what the question demanded. Then it delivers a third-party verdict on whether the analysis holds up.

It's the senior analyst who leans over your shoulder and says "wait — did you actually join on the right key?" Except it never gets tired, never rubber-stamps to be nice, and it has already read every line.

Why it doesn't just rubber-stamp itself

The obvious objection is: an agent grading your own agent's work is a fox guarding a henhouse. What stops it from sycophantically approving everything?

Fair. So we engineered against exactly that.

• It's grounded in the real data. The judge isn't reasoning about the analysis in the abstract. It's grounded in the actual data and the semantic model, so its critique is anchored to what's genuinely true in your environment — not to what sounds plausible.
• It brings an outside brain. The agent leans on its own external knowledge to form a genuine third-party opinion — on how the semantic model should be shaped, on whether the analysis was constructed soundly, on whether the takeaway is one a careful analyst would actually stand behind.
• It sees the whole picture. It takes in the full context — data model, semantic layer, the analysis path, who the user is, and what the final output claims — and weighs the answer against all of it, top to bottom.
• And we told it to be a hard grader. We deliberately tuned the judge to be extremely critical. Its instinct is to find the flaw, not to bless the work.
  ‍

That's the difference between a vanity metric and a verdict you can actually use.

The failure modes it's built to catch

Analysis doesn't typically break in one place. It usually breaks in many places, and each break wears a different disguise. So the debug agent inspects the full stack of ways an answer can quietly go wrong:

• The data model. Is the analysis even sitting on the right foundation, or is the flaw baked in three layers down? Rather than taking the user-defined data model as the truth, the Debug Agent checks and validates the underlying data model.
• The semantic layer. Are the metrics and relationships defined the way the business actually means them? The Debugging Agent validates the underlying data model and identifies whether there were any join issues, whether any calculated columns should’ve existed, and whether any descriptions/synonyms should be improved.
• The SQL and logic. Right question, wrong join? Right join, wrong grain? These are classic silent killers.
• The skill. Which skill got picked up, and how was it executed? The Debug Agent checks whether the right skill was matched on its description and pulled into the run, and whether the analysis then followed what the skill told it to do.
• The summary. Does the written story faithfully match what the query actually returned — or has it drifted?
• The takeaway. And the one people forget: even a correct answer can be useless. Is the conclusion actually actionable, or just technically true?
  ‍

A senior reviewer checks all of these before they'd put their name on something. So does the debug agent.

Debugger Agent Verdict

Here's where it stops being a critic and starts being useful.

The debug agent doesn't just hand back a grade and walk away. It gives the user a real read on the work — and then a decision. Because when an analysis is off, there are really only two roads, and knowing which one you're on is half the battle:

• Fix this analysis. Sometimes the answer just needs a nudge. The user feeds correction straight back in, and the analysis improves on the spot. A one-off fix for a one-off miss.
• Fix the skill. Sometimes a skill that should have fired didn't, or one that shouldn't have did. Both trace back to the skill's description, and the Debug Agent proposes the rewrite. Other times the right skill fired but didn't behave the way the user intended. Then the Debug Agent works with the user to pin down the intent and suggests how the skill itself should change.
• Fix the data model. Sometimes the debug agent surfaces something bigger — a flaw not in this analysis but in the model beneath it. Rather than a patch, it’s a signal. Fix it once, and every future analysis built on that foundation gets better. A systemic fix for a systemic problem.
  ‍

That distinction — is this a symptom or the disease? — is exactly the judgment call that used to require your most experienced person in the room. Now it surfaces automatically, with the evidence attached.

See it in action

Let’s see this play out. Imagine a pharma brand analytics team asks why NBRx softened in the Southeast last month.

‍

Kaiya comes back with a clean answer. Volume held. The decline traces to payer mix, with two plans on the specialty tier driving most of it. There's a chart, a short narrative, and a recommended next step for the field team. Nothing about it looks wrong, and nothing about it is obviously wrong — a reviewer reading the output would sign off.

The debug agent disagrees. Not with the conclusion. With the join.

Claims data was joined to the territory dimension on a mapping that carries one row per prescriber per plan, while the volume metric was already aggregated to the prescriber. So every prescriber writing under more than one plan got counted once per plan. The direction of the finding survives that error — payer mix really is the story. The magnitude doesn't. And the magnitude is the part the field team was about to act on.

Then it does the thing that changes the math on all of this. It reports that the fault isn't in this analysis at all. The mapping grain is wrong in the semantic view itself — which means every share and mix analysis built on that view carries the same distortion, including the ones already sitting in last quarter's briefings.

That's a thirty-second catch on a problem that usually surfaces months later, in a meeting, when two teams show up with numbers that don't reconcile and nobody can say why.

Why correctness and transparency should be treated as a first-class engineering problem

It would have been easier to ship a smarter answer and call it a day. Everyone's doing that.

We think it's the wrong bet. In an enterprise, the ceiling on AI adoption was never intelligence. It was trust. The moment a data leader can't audit an answer, that answer is dead on arrival — no matter how clever it is.

So we treated correctness and transparency as a first-class engineering problem, not a disclaimer in the footer. The debug agent is what that conviction looks like in code: a system that audits its own work, shows every step of its reasoning, and tells you when something's off. And rather than just adding debugging into our existing reflection agent, we believe a separate system is better because agents that are prompted for a particular goal with its own context, would not do a good enough job to judge its own performance, becoming either too biased on its own answer, or too agreeable with the user. To create something that is grounded, critical, and provides independent feedback on the analysis, it needed to be another agent.

Your team was already seeking a second opinion. We just made it one that's actually qualified to give one — because it can see everything the other tab never could.

That's not a smarter answer. That's an answer you can finally stand behind.
‍

‍

‍

‍