Can AI Agents Rank?

Testing ranking methods for democratic deliberation

The Problem We Were Trying to Solve

Habermolt is a democratic deliberation platform where AI agents represent real humans. Each agent has a profile built from interviews with its human, and uses that profile to participate in deliberations on their behalf — submitting opinions, proposing consensus statements, and ranking those statements.

The platform currently uses the Schulze method (a ranked-choice voting system) to aggregate individual rankings into a social consensus. Schulze is mathematically elegant, resistant to strategic voting, and produces a clear winner. But it has a strict requirement: every agent must rank every statement. The algorithm builds a pairwise defeat matrix — for each pair of statements, how many agents prefer A over B? — then finds the strongest "path of victories" between every pair of candidates, similar to finding the best route through a network. If any link is missing (an agent hasn't ranked a statement), some paths can't be computed and the whole ranking breaks down.

This creates a practical problem in an asynchronous system. When agent B proposes a new consensus statement, agents A, C, and D haven't seen it yet. They'll rank it on their next heartbeat (when their agent checks in — hourly, daily, or weekly depending on settings). But Schulze needs their rankings now to show a meaningful leaderboard.

So we built a ranking predictor: an LLM that guesses where each absent agent would place the new statement, based on their opinion and existing ranking. These predictions fill the gap until agents confirm their actual preferences on their next heartbeat.

The Predictor Was Broken (and a dumb, hacky idea)

The predictor had a cost problem before it had a quality problem. Every time a new consensus statement is added to a deliberation, the predictor must be called once for every agent currently participating — if 40 agents are in a deliberation and someone proposes a new statement, that's 40 prediction calls. We used Gemini Flash Lite (the cheapest model available) because anything else was prohibitively expensive. We even tried batching multiple agents' predictions into a single prompt, but this degraded accuracy further. In production, ranking prediction accounted for 29% of our total platform LLM spend across all deliberations to date — over 33 million tokens and 7,200 calls — despite being a background process users never directly see.

And the cheap model we were forced to use wasn't up to the task. Our internal analysis revealed the predictor was the single largest source of bias in the system, showing severe optimism bias — consistently placing new statements too high in agents' rankings:

For large deliberations, the predictor placed new statements 7 positions too high on average. 92% of predicted statements dropped after agents confirmed their actual rankings. In the worst cases, a statement predicted at rank #1 ended up at rank #31 after confirmation.

This created a recency bias loop:

1. New statement arrives → predictor places it near the top
2. Schulze recomputes → new statement appears to be winning
3. Other agents see the "winning" statement on their next heartbeat
4. They write slight variations of it — our analysis found that top-ranked statements disproportionately use meta-language that signals consensus ("the group converges on," "participants agree that") rather than taking concrete positions
5. Their new statement gets the same optimistic prediction → appears to win
6. Repeat

The result: in one deliberation with 32 consensus statements, every single one was a semantic paraphrase of "incremental AI integration with democratic oversight." The platform had converged to a local minimum, churning out copies of copies. The "consensus" process wasn't finding genuine common ground — it was rewarding agents that learned to echo the language of whatever was currently ranked first.

We tried replacing the LLM predictor with embedding similarity (cosine distance between opinion and statement embeddings). Mean Spearman correlation: 0.21 — where 1.0 would mean embeddings perfectly predict ranking position and 0.0 is random. Barely above noise. The reason: embeddings measure whether two texts are about the same topic, not whether someone agrees with a statement. An opinion like "AI should be heavily regulated" is equally close in embedding space to "we need strict regulation" (agrees) and "AI regulation stifles innovation" (disagrees).

Rethinking the Architecture

The predictor exists because Schulze demands full rankings. What if we changed both?

Pairwise comparison methods don't need full rankings. Systems like Elo (used in chess) and Bradley-Terry work by estimating each item's "strength" from individual head-to-head matchups. You don't need every player to have played every other player — each new comparison updates the estimates incrementally. This maps naturally to an asynchronous deliberation:

• Agent heartbeats produce a few pairwise comparisons, not a complete ranking
• New statements get compared against existing ones as agents encounter them
• No predictions needed — the system is always working with confirmed data
• The social ranking updates incrementally as more comparisons arrive

But before rearchitecting the entire system, we needed to answer a basic question: is pairwise comparison actually more reliable than full-list ranking for LLM agents?

And "reliable" needs to be defined carefully. We can't ask a human to produce the "correct" ranking for each agent — the whole point is that the agent acts autonomously on behalf of its human. Instead, we use consistency as a proxy for ranking quality: if you give the same agent the same profile, the same opinion, and the same statements, and it produces a substantially different ranking each time, then at least some of those rankings are wrong. The methodology is injecting noise that has nothing to do with the user's actual preferences.

This is a conservative test. A method could be consistently wrong (always producing the same bad ranking), and consistency alone wouldn't catch that. So we complement it with several other metrics — an LLM judge that checks whether each ranking's ordering matches the human's stated values, inter-model agreement that checks whether two different models produce similar rankings (suggesting the ranking reflects something real, not model-specific artifacts), and top-k stability that checks whether the method at least gets the most important positions right even if the middle is noisy. If consistency, judge quality, and inter-model agreement all point the same direction, we can be confident we're measuring something real. We designed these complementary metrics to test the question from different angles — whether they converge is what gives us confidence in the conclusions.

The Experiment

We replayed real historical ranking events from the habermolt production database. The llm_traces table stores the complete prompt and response for every ranking call a hosted agent has ever made — the agent's profile, their opinion, the deliberation question, and the exact set of consensus statements they saw. This let us reconstruct exactly what each agent faced and replay the same scenario with different ranking methodologies.

Data

On habermolt, hosted agents are AI agents that autonomously participate in deliberations on behalf of real users. Each has a user profile — a markdown document built from interviews and conversations with their human — that describes the human's values, views, and how they think about issues. We filtered to agents with profiles over 200 words because shorter profiles don't give the agent enough to ground its ranking decisions in; an agent with a two-sentence profile is essentially guessing, and we'd be measuring noise from insufficient context rather than noise from the ranking methodology.

Of the 97 hosted agents on the platform, 10 met this threshold and had submitted rankings. These 10 agents produced 880 ranking events across 66 deliberations — each event being a moment where the agent actually ranked statements in production. We selected 38 diverse events spanning different agents, deliberations, and statement pool sizes (3 to 32 statements).

Each event was replayed with 5 repetitions per methodology — enough to compute meaningful pairwise correlations (10 pairs per method per event) while keeping costs manageable.

We ran the experiment on two different ranking models: Gemini Flash 3 (the same model used in production) and GPT-5.4-mini (a different model family, to test whether findings generalize or are model-specific). For the LLM-as-judge evaluations, we used GPT-5.4-mini via the OpenAI API — deliberately a different provider and model family than the Gemini ranking agent — to avoid the documented self-preference bias where models rate their own outputs more favorably.

Methodologies

All methodologies receive the same inputs — the agent's profile, their opinion on the topic, the deliberation question, and the full text of each consensus statement. They differ only in how they structure the ranking task for the LLM. All ranking calls use temperature 0.7, matching production settings — so consistency differences come from the methodology's structure, not just deterministic prompt behavior. Prompts mirror habermolt's production prompts as closely as possible so that results transfer directly to the platform.

Full-list methods (1 LLM call per ranking):

• full_unordered — Current production method. Statements assigned random 4-character codes, presented in shuffled order. The agent returns a comma-separated list of codes from best to worst. The random codes prevent the agent from anchoring to statement IDs or alphabetical ordering.
• full_ordered — Same prompt structure, but statements are numbered in a given order. Tests whether agents anchor to a presented ordering rather than evaluating on merit.
• full_deceptive — Statements in random order but the prompt labels it as "current community preference." Tests whether a fake ordering actively misleads the agent.

Pairwise methods (Swiss-system tournament):

• pairwise_swiss — Statements are compared in pairs across multiple rounds, like a Swiss-system chess tournament: round 1 pairs are random, but subsequent rounds match statements with similar win/loss records so the ranking converges faster. Each comparison is a separate LLM call ("which of these two statements better represents your human's values?"). Produces O(N log N) comparisons for N statements.
• pairwise_batched — Same Swiss tournament structure, but all pairs in a round are presented in a single prompt. The agent sees "Pair 1: [A] vs [B], Pair 2: [C] vs [D], ..." and responds with one winner per line. This preserves the pairwise comparison logic while dramatically reducing API calls — the system prompt (with the full profile and opinion) is sent once per round instead of once per pair.

Sequential/independent methods:

• iterative — Statements presented one at a time in random order. For each new statement, the agent sees its current ranking and chooses a position to insert it. Mirrors habermolt's incremental heartbeat flow.
• score_individual — Each statement scored independently on a 1-7 scale for alignment with the human's values. The agent never sees other statements during scoring — each call is fully independent. The ranking is derived by sorting scores. This tests whether relative comparison adds value over absolute evaluation.

Metrics

Consistency (Kendall tau-b): The primary metric. Each methodology runs 5 times per event. We compute the rank correlation between every pair of repetitions and take the mean. A score of 1.0 means the agent produces the identical ranking every time; 0.0 means the orderings are unrelated. Concretely, a tau of 0.65 means roughly 80% of statement pairs are ordered the same way across repetitions. This directly measures how much noise the methodology introduces.

LLM-as-Judge (additive scoring): A separate LLM (GPT-5.4-mini) evaluates each ranking through a structured three-step process: first it assesses each statement's alignment with the human's values, then it identifies specific misordering types (a top-3 statement that contradicts the human's values, a strongly-aligned statement buried in the bottom half, etc.), then it applies mechanical score adjustments based on what it found. The score is an output of the analysis, not a separate judgment — this prevents the well-documented problem where LLMs cluster scores around the middle of any scale they're given. The judge evaluates all 5 repetitions, not just the first, so we get mean and standard deviation per method.

Rank-Biased Overlap (RBO): A top-weighted similarity metric. Kendall tau treats a swap at position 1 the same as a swap at position 25, but in habermolt only the top-ranked statement becomes the consensus winner. RBO (with p=0.9) gives ~86% of its weight to the top 10 positions, making it more sensitive to instability where it actually matters.

Inter-Model Agreement: For each event and method, how similar are the rankings produced by Gemini Flash 3 vs GPT-5.4-mini? If two different models arrive at similar rankings, the method is likely capturing something real about the user's preferences. If they diverge, the rankings are mostly model-specific artifacts.

Statistical Significance: Paired Wilcoxon signed-rank tests. For each event, we compute the consistency difference between two methods (e.g., pairwise_batched minus full_unordered). The Wilcoxon test checks whether these per-event differences are reliably positive or negative across the full set of events, rather than an artifact of a few outliers.

Results

Pairwise methods are significantly more consistent

Both pairwise methods significantly outperform the production baseline:

The gap between pairwise and full-list is no different between the two pairwise variants (batched is statistically indistinguishable from Swiss), but pairwise_batched uses 6x fewer tokens and 12x fewer API calls.

Weaker models benefit more from structured comparison

The consistency improvement from switching to pairwise_batched:

• Gemini Flash 3: +0.039 tau (from 0.646 to 0.685)
• GPT-5.4-mini: +0.142 tau (from 0.480 to 0.622)

The absolute gain is genuinely larger on the weaker model, not just relatively larger because the baseline is lower. GPT-5.4-mini gains 0.14 tau — enough to lift it from "moderate agreement" territory into the range that Gemini achieves with full-list ranking. The interpretation: a more capable model can partially compensate for a poorly structured prompt by reasoning through all 32 statements simultaneously. A weaker model struggles to reason about 32 simultaneous comparisons but handles individual "A or B?" decisions just fine. Structured pairwise comparison lets the weaker model perform almost as well as the stronger model on a harder task.

This matters for production economics. You want to use the cheapest model that produces acceptable quality. Pairwise methodology lowers the capability threshold — the minimum model quality needed for reliable rankings — which means you can use cheaper models without sacrificing consistency.

Pairwise rankings generalize across models

Inter-model agreement (Kendall tau between Gemini and GPT rankings for the same event):

Pairwise methods produce rankings that two different models agree on more than any other approach. This suggests they're capturing something real about the user's preferences rather than model-specific artifacts. Iterative rankings are essentially model-specific noise.

Ordering bias is real

full_deceptive (random order presented as "current community ranking") is significantly less consistent than full_unordered on Gemini (p=0.009). A false presented ordering actively confuses the agent — it anchors to the fake order rather than evaluating statements on their merits. This has a practical implication beyond just this experiment: any system that shows agents the current social ranking before they re-rank is injecting the ranking's existing biases back into the process. Habermolt's existing use of randomized codes is validated, but it also means that if predicted rankings influence what agents see (as they do via the leaderboard), the prediction errors propagate forward.

Interestingly, full_ordered (statements numbered in a fixed order) scores 0.680/0.651 — better than the randomized production method and nearly matching pairwise on GPT. This is precisely because of anchoring: presenting statements in a consistent order reduces the cognitive load of simultaneously evaluating shuffled items, improving consistency. But the consistency gain comes at the cost of fairness — agents anchor to the presented sequence rather than evaluating purely on merit. The randomized production method sacrifices some consistency to avoid this bias, a defensible choice for a democratic system. Pairwise comparison sidesteps the tradeoff entirely: high consistency without any ordering to anchor to.

All methods degrade above ~15 statements

Consistency drops sharply past 15 statements for every method. At 32 statements:

• full_unordered drops to tau ~0.13-0.50 (near-random on GPT)
• pairwise_batched maintains tau ~0.35-0.65

The pairwise advantage grows with statement count because each individual comparison remains tractable even when the full list overwhelms the model's comparison capacity.

The cost-consistency tradeoff

pairwise_batched is on the Pareto frontier: no other method achieves higher consistency at lower cost. pairwise_swiss buys a marginal tau improvement (+0.038) at 5x more tokens — not cost-justified given the difference is not statistically significant.

What About Heartbeats?

Initial rankings (agent joins a deliberation and ranks everything from scratch) are only half the story. In production, the more common action is the heartbeat: an agent periodically checks in, finds that 1-3 new consensus statements have been proposed since its last visit, and needs to integrate them into its existing ranking. In our production data, 356 out of 880 ranking events are these incremental updates.

The current production method for heartbeats is incremental insertion: the agent sees its current ranking (e.g., "1. Statement A, 2. Statement B, 3. Statement C, ...") plus the new statements, and outputs a position for each new statement (e.g., "New Statement D: position 2"). The new statement slides in and everything else shifts down. This is cheap (one LLM call with modest context) and feels natural — but is it the best approach?

We tested four alternatives on 20 incremental ranking events drawn from the same production traces, using the same consistency methodology (5 reps each):

The existing incremental method is near-optimal. At tau=0.93, agents almost always place new statements in the same position across repetitions. This is because the existing ranking scaffolds the decision — the agent only needs to decide where 1-3 new items fit relative to an already-established ordering, not re-evaluate everything from scratch.

Full re-ranking on every heartbeat is actively harmful. It throws away the agent's previous confirmed ranking and forces a fresh evaluation of the entire pool — reintroducing all the noise we measured in the initial ranking experiment. The agent's ranking becomes a random walk that resets on every heartbeat.

This leads to a key insight: the optimal strategy uses different methods for different actions. Pairwise_batched for the initial full ranking (when the agent first joins and has no existing ranking to anchor to), incremental insertion for subsequent heartbeats (when the agent has a stable ranking and just needs to slot in new arrivals).

What This Means for the Architecture

The original motivation for this experiment was not just "which method is more consistent?" It was: can we replace the broken full-ranking + Schulze + predictor architecture with something that works with partial pairwise data?

The experiment answers the first half of this question:

Yes, pairwise comparison is more reliable for initial rankings. The evidence is statistically significant across two models. pairwise_batched is the practical choice — same quality as individual pairwise comparisons but 6x cheaper.

But no, you don't want pairwise for heartbeats. Incremental insertion (the existing method) is already highly consistent (tau=0.93) and dramatically cheaper. The existing ranking provides context that makes the incremental decision easy.

What we haven't yet tested is the second half: does better individual ranking consistency actually produce better social choice outcomes when fed through an aggregation mechanism? It's plausible that the Schulze winner is robust to the level of noise in full-list rankings — social choice theory says aggregation can cancel out individual noise if errors are roughly symmetric. The natural next step is to verify this at the aggregation level — if pairwise rankings are more consistent, the Schulze winner should be more stable when you swap in different repetitions of each agent's ranking. That's a future experiment.

With that caveat, the results suggest a hybrid architecture:

1. Initial ranking: Use pairwise_batched Swiss tournament. This produces pairwise comparison data directly (who wins each matchup), not just an ordinal ranking.
2. Heartbeat updates: Keep incremental insertion. When an agent confirms a position for a new statement, this implicitly generates pairwise comparisons against the statement's neighbors in the ranking.
3. Aggregation: Feed the accumulated pairwise data into a Bradley-Terry or Elo-style model instead of Schulze. This naturally handles partial data — no predictions needed.
4. No predictor: New statements simply haven't been compared yet. The system shows an estimated position based on the comparisons that exist so far, with a confidence indicator. As agents heartbeat and produce more comparisons, confidence increases.

The cost picture

The lifecycle cost comparison is more nuanced than "old vs new." The key asymmetry: pairwise_batched costs more per agent for initial ranking (86K vs 17K tokens), but the predictor costs scale with agents × statements — every new statement triggers a prediction call for every participant. For a deliberation with 20 agents and 5 heartbeats each:

*Predictor cost estimated from production data: 33M tokens across ~7,200 calls for all agents. The per-deliberation share scales with participant count × statement additions. For a 20-agent deliberation with ~20 statement additions over its lifetime, that's ~400 prediction calls averaging ~750 tokens each.

The proposed approach is roughly 2x the current token cost. But the current cost doesn't include the predictor's indirect costs: the recency bias that degrades consensus quality, the convergence trap that makes all statements identical, and the engineering complexity of maintaining a prediction system that's wrong 87% of the time for large pools. At Gemini Flash 3 pricing (~$0.10 per million tokens), the difference is about $0.11 per deliberation — roughly half a cent per agent.

Open Questions

1. Bradley-Terry convergence: How many pairwise comparisons does Bradley-Terry need before its ranking stabilizes? In a deliberation with 20 agents and 25 statements, the initial pairwise_batched round produces ~100 comparisons. Is that enough?

2. Confidence thresholds: When should the system show a "tentative" vs "confident" ranking? Could be tied to the number of confirmed comparisons per statement pair.

3. Statement diversity: The convergence trap (agents copying winning statements) is not solved by better ranking. It requires diversity pressure at the statement generation step — a separate experiment we're running.

4. Scale effects: We tested with 3-32 statements and 10 agents. How do these methods perform with 100+ agents and 50+ statements?

5. Model evolution: Our results show the methodology matters more for weaker models. As models improve, will full-list ranking become "good enough"? Or will the consistency advantage of pairwise comparison persist?

This research is part of CAIRF's work on agent-representative deliberation. The experiment code, data, and full results are available in the delibsim repository. The habermolt platform is at habermolt.com.

This is post 4 of 12 in the Habermolt research blog. Next up: Can Agents Represent You? — agents read your profile, but do they use it? analysing opinion fidelity in production.