CCT Fleet · Classifier Eval loading
loading…
candidate Qwen14B-SFT (ours) Qwen3-30B labels (reference)
—
candidates vs pseudo-labels
—
Nebius spent · — headroom
—
held-out msgs (Qwen3-30B labels)
—
our Qwen14B-SFT bar
Per-field agreement with Qwen3-30B pseudo-labels (685 held-out msgs) — macro-F1 per field; Opus-labeled scores: Stage 1 table on the prose-v1 eval tab
Grouped bars per field. Each candidate's macro-F1 on that single field vs the Qwen3-30B pseudo-labels (shown as %). Qwen14B-SFT (ours) in pink.
Alignment rate — per-message exact match, agreement with Qwen3-30B pseudo-labels (685 held-out msgs)
Qwen3-30B-A3B is the labeler, so its own row scores high by construction.
Per-field macro F1 — agreement with Qwen3-30B pseudo-labels (685 held-out msgs)
Reference = Qwen3-30B-A3B labels on 685 held-out messages, not a frontier model.
Cost — $ per 1,000 messages classified
SFT'd 14B serves locally at near-zero marginal cost after setup; baselines are Nebius Token Factory per-token.
Pareto — cost vs alignment (log cost axis)
Latency — p50 per message (ms)
p50 per message on the 685 held-out messages.
Headline story — the 3 numbers
—
vs flagship baseline
—
latency speedup
—
alignment uplift vs base
Measured against Qwen3-30B-A3B pseudo-labels, not a frontier reference: a high score means close to the labeler, not correct. Opus-labeled scores: Stage 1 table on the prose-v1 eval tab.
LoRAs trained · trained tokens · Nebius serverless LoRA inference needs beta access
Appendix (training details, prose summarizer, Anthropic refs)
SFT training (v0.2 schema, Nebius LoRA)
Prose summarizer comparison — per-bucket 'started/finished/blocked/key_decisions' bullets
Bullets per section (avg)
Cost + latency per model
Judge metrics
Not yet run.