TRIAGE
For this research, I wanted failure attribution in multi-agent LLM systems to help decide which corrections to try first. A failed task can offer several plausible intervention points, and identifying an error doesn't establish how useful correcting it would be. I developed TRIAGE to rank these candidates by their estimated intervention value and propose repairs, with mentorship from Professor Xiaowu Dai at UCLA. The paper was accepted to the AACL-IJCNLP main conference, an international conference for peer-reviewed research in natural language processing. TRIAGE works from the execution trace, which records agent messages, tool calls, and tool outputs.
In Who&When, a failure-attribution benchmark, one trace involves a puzzle about humans who tell the truth and vampires who lie. The reasoning agent rejects the correct all-vampire solution on the grounds that the residents' statements would be false. But that is consistent with vampires lying. The validation agent accepts the deduction anyway. Who&When correctly identifies the reasoning step as the earliest decisive error; my concern is that this leaves the validator's endorsement uncredited as a correction point. TRIAGE ranks the reasoning step first and the validation step second.
I use LLM-as-judge suspicion scoring to judge how likely each step is to be wrong, then combine that with structural impact because a mistake can affect later steps as well. I estimate this propagation with a heuristic trace-impact graph, which doesn't need extra LLM calls. The ranking determines which three candidates receive repair patches. For each of those, LLM-simulated replay compares model-estimated task success with and without the patch. Evaluating every candidate this way would require more model calls; limiting it to three keeps this cost down, and the other candidates remain in the ranking.
TRAIL is a benchmark with multiple annotated errors per trace. I used top-10 any-error recall to check whether at least one of those errors appeared among the ten highest-ranked candidates. These are the proportions of traces where it did, using the pre-replay ranking:
TRIAGE: 93.2% Suspicion-only: 91.2% Direct LLM ranking: 80.4%
In the suspicion-only ablation, candidates are ranked without structural impact. Its performance shows that most of the improvement over direct LLM ranking comes from per-step scoring. Adding structural impact improved this result by a modest amount, but I didn't see that improvement on every metric. I also released TriageBench to examine intervention priorities beyond the original error labels. It contains pairwise repair-priority judgments from three LLM judges over 1,184 candidate steps.
Who&When: 184 traces; TRIAGE, suspicion-only (ρ-only), direct LLM ranking (LJ), random selection. K=5 recall: 73.9% TRIAGE / 74.5% suspicion-only.
TRAIL: 148 traces, of which 31 are SWE-Bench; top-K any-error recall. The percentages above are at K=10.
K: how many candidate steps are inspected.
Model for the rankings: Kimi K2.5. TRIAGE results: pre-replay ranking.
TRAIL calls: one for direct ranking / five passes for suspicion scoring.
At first I was frustrated because I didn't know if my method would improve error-localization accuracy. It eventually did on Who&When, but that still left me wondering if it was a better method for debugging, which was why I chose these components in the first place. The repair assessment uses LLM-simulated replay: a language model estimates the effect of the proposed change. It does not actually re-execute the multi-agent system. I still need to test if the failed task can actually be recovered by making those interventions.