CCT Fleet · Merge Eval V0.1.B

Hourly per-agent bullet summaries merged up a hierarchy: agent rows (4h, day) and all-agent columns (1h, 4h, day) · gamejam Day1 · 2284 source bullets · headline: agreement with Claude Haiku 4.5 merges; below: checks against the source bullets

Qwen3-8B merger Qwen3-235B-A22B merger Llama-3.1-8B merger Nemotron 3 Ultra 550B-A55B (untuned; also a LoRA teacher) merger

Agreement section: reference = Claude Haiku 4.5 merges, judge = Claude Sonnet 4.5. Source checks section: reference = each node's own input bullets, judge = Claude Haiku 4.5.

Agreement with Haiku merges

Each merger's bullets compared with Claude Haiku 4.5 merges of the same inputs; judged by Claude Sonnet 4.5 (same vendor as the GT model).

0.94

F1 vs Haiku merges, mean of row-4h, row-day, col-1h, col-4h · 235B (Qwen3-8B 0.76 · Llama-3.1-8B 0.59 · Nemotron 3 Ultra 550B-A55B (untuned; also a LoRA teacher) 0.93)

89%

Haiku bullets recovered (recall) · 235B (Qwen3-8B 63% · Llama-3.1-8B 44% · Nemotron 3 Ultra 550B-A55B (untuned; also a LoRA teacher) 87%)

unsupported candidate bullets · 235B (Qwen3-8B 5% · Llama-3.1-8B 7% · Nemotron 3 Ultra 550B-A55B (untuned; also a LoRA teacher) 0%)

Agreement with the reference (Claude Haiku merges) — F1 per level

Recall — Haiku GT bullets covered by the candidate (partly = 0.5)

Precision — candidate bullets in the GT or backed by their own sources

Agreement detail — same sampled nodes for both candidates

Merger	Level	Nodes	GT bullets	Cand. bullets	Recall	Precision	Strict precision	Unsupported	F1	Failed nodes
Qwen3-8BModal A10G	row-4h	10	151	176	0.72	0.98	0.94	0.02	0.83	—
	row-day	10	160	172	0.69	0.98	0.90	0.02	0.81	—
	col-1h	10	148	185	0.55	0.83	0.73	0.17	0.67	—
	col-4h	4	78	80	0.55	1.00	0.80	0.00	0.71	—
Qwen3-235B-A22BToken Factory	row-4h	10	151	152	0.92	1.00	0.96	0.00	0.96	—
	row-day	10	160	162	0.86	1.00	0.90	0.00	0.92	—
	col-1h	10	148	174	0.89	1.00	0.90	0.00	0.94	—
	col-4h	4	78	72	0.87	1.00	0.82	0.00	0.93	—
Llama-3.1-8BOpenRouter CoreWeave bf16	row-4h	10	151	127	0.52	0.98	0.94	0.02	0.68	3
	row-day	10	160	133	0.46	1.00	0.94	0.00	0.63	3
	col-1h	10	148	180	0.46	0.87	0.77	0.13	0.60	0
	col-4h	4	78	76	0.29	0.88	0.60	0.12	0.44	0
Nemotron 3 Ultra 550B-A55B (untuned; also a LoRA teacher)free route; list-priced on the Summary tab	row-4h	10	151	162	0.98	1.00	0.94	0.00	0.99	0
	row-day	10	160	186	0.83	0.99	0.88	0.01	0.91	0
	col-1h	10	148	183	0.86	0.99	0.89	0.01	0.92	0
	col-4h	4	78	79	0.81	0.99	0.82	0.00	0.89	0

Failed nodes = nodes where the merger returned no valid merge (scored as empty). Precision = (bullets stated in the GT merge + bullets not in GT but backed by their cited sources) / candidate bullets; strict precision counts GT matches only; F1 uses recall and the lenient precision.

Source checks

Each merged node checked against its own input bullets (the V0.1 hourly bullets it merges); judge selectable, default Claude Haiku 4.5.

Judge

71%

important facts kept, all agents · 4h · 235B (Qwen3-8B 34% · Llama-3.1-8B 19%)

86%

important-fact coverage, mean of 4 levels · 235B (Qwen3-8B 52% · Llama-3.1-8B 46%)

2 / 480

unsupported merged bullets, 3 mergers

verbatim copies · 235B (Qwen3-8B 55% · Llama-3.1-8B 66%)

Important-fact coverage by level — % of judge-flagged important source facts kept in the merged node

Grouped bars per level, one per merger. Qwen3-235B in pink. The 1h per-agent cell is the source itself and is not judged.

All-fact coverage — share of all sampled source facts kept

Faithful — merged bullets fully backed by their cited sources

Unsupported — merged bullets the sources do not back · lower is better

Near-duplicates — near-duplicate pairs per merged bullet · lower is better

Verbatim copy — merged bullets copied word for word from one source · lower is better

Compression — merged bullets / source bullets · lower = shorter

Source-check story — the 3 numbers

+37 pt
important facts kept at the 4h column
0
unsupported 235B bullets of 160
0%
235B verbatim copies (8B 55%)

Qwen3-8B mostly selects one source bullet per merged bullet, so coverage falls as the span grows. Qwen3-235B merges several sources per bullet and keeps most important facts at every level.

How we evaluated

Headline (top): reference = Claude Haiku 4.5 merges of the same inputs; judge = Claude Sonnet 4.5 (same vendor as the GT model); mergers = Qwen3-8B, Qwen3-235B-A22B, Llama-3.1-8B, Nemotron 3 Ultra 550B-A55B (untuned; also a LoRA teacher).
These source-grounded checks: judge = Claude Haiku 4.5 (cross-family); reference = each node's own input bullets (the V0.1 hourly bullets it merges), not the Haiku merges.
The judge sees each merged node's source bullets and its merged bullets, and only grades.
Coverage: up to 8 nodes per level, 40 source facts per node (1036 facts here). The judge flags important facts (deliverables, results, blockers, decisions) and marks each covered (1), partly (0.5) or missing (0).
Faithfulness: 40 merged bullets per level, each checked against the sources it cites: faithful, partial (added or shifted detail) or unsupported.
Copying: near-duplicates, single-source and verbatim-copy rates are counted on the merged tree itself.
Both mergers get the same prompt, bullet caps and judged samples. Judge: Claude Haiku 4.5 (cross-family).

55%

important facts kept, all agents · 4h · 235B (Qwen3-8B 39%)

75%

important-fact coverage, mean of 4 levels · 235B (Qwen3-8B 53%)

0 / 320

unsupported merged bullets, 2 mergers

verbatim copies · 235B (Qwen3-8B 55%)

Important-fact coverage by level — % of judge-flagged important source facts kept in the merged node

Grouped bars per level, one per merger. Qwen3-235B in pink. The 1h per-agent cell is the source itself and is not judged.

All-fact coverage — share of all sampled source facts kept

Faithful — merged bullets fully backed by their cited sources

Unsupported — merged bullets the sources do not back · lower is better

Near-duplicates — near-duplicate pairs per merged bullet · lower is better

Verbatim copy — merged bullets copied word for word from one source · lower is better

Compression — merged bullets / source bullets · lower = shorter

Source-check story — the 3 numbers

+17 pt
important facts kept at the 4h column
0
unsupported 235B bullets of 160
0%
235B verbatim copies (8B 55%)

Qwen3-8B mostly selects one source bullet per merged bullet, so coverage falls as the span grows. Qwen3-235B merges several sources per bullet and keeps most important facts at every level.

How we evaluated

Headline (top): reference = Claude Haiku 4.5 merges of the same inputs; judge = Claude Sonnet 4.5 (same vendor as the GT model); mergers = Qwen3-8B, Qwen3-235B-A22B, Llama-3.1-8B, Nemotron 3 Ultra 550B-A55B (untuned; also a LoRA teacher).
These source-grounded checks: judge = Qwen3-235B; reference = each node's own input bullets (the V0.1 hourly bullets it merges), not the Haiku merges.
The judge sees each merged node's source bullets and its merged bullets, and only grades.
Coverage: up to 8 nodes per level, 40 source facts per node (1036 facts here). The judge flags important facts (deliverables, results, blockers, decisions) and marks each covered (1), partly (0.5) or missing (0).
Faithfulness: 40 merged bullets per level, each checked against the sources it cites: faithful, partial (added or shifted detail) or unsupported.
Copying: near-duplicates, single-source and verbatim-copy rates are counted on the merged tree itself.
Both mergers get the same prompt, bullet caps and judged samples. Judge: Qwen3-235B.

Appendix (merge runs, judges)

Merge runs — from each merge tree

Merger	Calls	Fallbacks	Tokens in / out	Wall time	Caps / section
Qwen3-8BModal A10G	129	0	167k / 69k	8.4 min	6 / 6 / 4 / 4
Qwen3-235B-A22BToken Factory	129	0	165k / 78k	4.9 min	6 / 6 / 4 / 4
Llama-3.1-8BOpenRouter CoreWeave bf16	158	44	210k / 212k	5.8 min	6 / 6 / 4 / 4
Nemotron 3 Ultra 550B-A55B (untuned; also a LoRA teacher)free route; list-priced on the Summary tab	83	0	146k / 91k	48.9 min	6 / 6 / 4 / 4

Judges — same samples for every merger

Judge	Calls	Judge USD (est.)
Claude Haiku 4.5 (cross-family)	141	subscription
Qwen3-235B	94	USD 0.054