Across the pipeline
The best Artha config for each capability, against the same frontier reference.
Segregation has no published benchmark yet; it has a tab below and will fill in there first.
How this is measured
One comparison
The same document goes to both models. An LLM judge reads both outputs against the source and scores each one, then names a winner. A tie is a real outcome, not a rounding artefact — so win and loss rates are shares of the decided comparisons, with the tie count stated beside them.
Three criteria
Goodness — overall quality of the analysis.
Faithfulness — the share of assertions the source actually supports, so
100 − faithfulness is the unsupported share.
Completeness — nothing important left out.
How to read the index
The frontier reference is pinned at 100. An index of 103 means our score is 3% higher on the same measure; 91 means 9% lower. The raw scores sit beside every index — those are the honest unit, but only the index compares across tasks, because the tasks use different metrics and different graders.
full manual coverage: full = scored here against every ground-truth document; manual = an externally-computed aggregate, labelled as such.
Detail by task
How these numbers are produced
The app is the scorer. Every number on this page was computed here, from a predictions file measured against ground truth already stored in the system. Nothing is a figure copied out of someone's spreadsheet.
Fair comparison or none. A model config is only scored once its predictions cover every ground-truth document in the eval set. Incomplete submissions are blocked rather than quietly compared, and anything scored on a partial subset is excluded from this board entirely. Externally-computed aggregates appear as manual and are always labelled. The pipeline tiles prefer a full-coverage config even when a manual aggregate scores higher, so the headline figure is always one this app computed.
One family, not one model. The pipeline tiles show the best Artha config for each capability, and each tile names the config it is reporting. These are specialised configs; no single one is claimed to lead every task at once.
The head-to-head is judge-scored. For each document, our model's output and the frontier reference's production output are read by an LLM judge, which scores both on goodness, faithfulness and completeness and names a winner. Those judge scores are model-generated: they are a consistent yardstick, not ground truth. Win/loss counts are per-document verdicts, not tests of statistical significance — an analyser with a handful of judged documents is omitted rather than shown as a result, and the roster is a statement about breadth, not a significance test.
Task metrics. Segmentation leads on boundary F1 here, which balances missed boundaries against spurious ones; the operator board leads on boundary recall instead, because internally a missed boundary silently merges two documents and is the error worth hunting. Both are shown in the segmentation table. Classification leads on accuracy, extraction on field accuracy pooled across field instances, segregation on adjusted Rand index. Each board names its headline metric above the table.