JURIX 2026 · companion page for reviewers
A Thematic Layer for CLAUDETTE
Separating the cost of multi-label annotation from the effect of taxonomy granularity
The CLAUDETTE / UNFAIR-ToS corpus records whether a sentence is unfair, never what the clause is about. This page gives reviewers everything the paper releases: the thematic layer with its individual votes, the frozen gold standard, the four LLM judges and their exact prompts, the frozen taxonomies, the codebook and the protocol — and a fine-tuned Legal-BERT you can run on your own text.
Dataset fingerprint 7116e627f528c557 · 50 contracts · 9,414 sentences · 3 trained annotators · 4 LLM judges
Try the classifier
Legal-BERT fine-tuned with the paper's recipe on the 33 design contracts (11-theme taxonomy, ±1 sentence of context, 8 epochs, seed 42). The 17 contracts offered below were never seen during training. Your text is processed once and not stored.
0 / 60,000 characters · about 0 sentences
Legal-BERT (11 themes), trained on the 33 design contracts. Your text is processed once and not stored.
What we release
All files below are the ones every figure of the paper was computed from. Records are keyed by (document, index) and join directly onto the CLAUDETTE corpus, whose sentences we do not redistribute. Annotators appear as A1, A2 and A3.
Everything in one archive
thematic-layer-7116e627.zip · 383 KB · SHA-256 d6ee877cc743dae9…
Individual votes
Individual annotator votes (A1–A3): one record per (document, sentence, annotator), primary theme and secondary themes.
votes.jsonl · 28,242 records · 2.9 MB
DownloadGold standard and arbitration trail
Frozen gold standard and arbitration trail: agreement class, cascade tier, engine proposal, decided themes, vote tally, confidence.
gold.jsonl · 9,414 records · 3.4 MB
DownloadLLM judges
The four LLM judges on the same sentences and vocabulary.
judges.jsonl · 37,656 records · 4.0 MB
DownloadFrozen taxonomies
Frozen taxonomy specification: T20 and the T14 / T11 / T10 projections, design / held-out partition.
taxonomies.json · 27 KB
DownloadJudge prompts
The instruction files given to the judges (protocol v9.2) and the session protocol: inputs, isolation, validation, models, harnesses and dates for Claude Fable 5, Claude Opus 4.7, Mistral Medium 3.5, GPT-5.5.
Codebook
The 20 themes with their definitions, the segmentation and multi-label rules given to annotators.
ReadProtocol
Independent annotation, isolated judges, three-tier resolution cascade, read-time projections, document-level bootstrap.
ReadManifest, labels and splits are included in the archive. Dataset fingerprint 7116e627f528c557… · built 2026-09-20.
Key figures
Generated from the released files; open “how computed” for the source of each number.
| Measure | 20 themes | 11 themes | How computed |
|---|---|---|---|
| Krippendorff's α (MASI distance), three annotators | 0.658 | 0.725 | α computed on the individual votes (votes.jsonl) with the MASI distance on the set {primary} ∪ secondaries; the 11-theme value is the same votes projected through taxonomies.json. docs/pactiva-taxonomies/resultats/taxonomy_matrix.json (population all, raw) |
| Nominal α on primary themes | 0.732 | 0.794 | Same votes, primary theme only, nominal distance; the gap with α-MASI is the cost of the multi-label format. taxonomy_matrix.json (raw) |
| Gain from consolidating 20 → 11 themes (α-MASI) | — | 0.067 | Paired difference of α-MASI between the T20 votes and their T11 projection (bootstrap CI over documents in the paper). taxonomy_matrix.json |
| Human ceiling, κ (leave-one-annotator-out against the gold) | 0.859 | — | Each annotator scored against a gold rebuilt without her votes; mean Cohen's κ over the three annotators (accuracy 0.8705). frontend/src/features/paper/campaign.json (E1.5) |
| Legal-BERT fine-tuned, κ (5-fold, grouped by document) | 0.695 | 0.720 | nlpaueb/legal-bert-base-uncased, 8 epochs, ±1 sentence of context, weighted cross-entropy, seed 42, 5-fold cross-validation grouped by document on Grid'5000; macro-F1 0.612707 / 0.734652. research/runs/legalbert_T20/results.json, research/runs/legalbert_T11/results.json |
| Best LLM judge (Claude Fable 5), κ | 0.581 | 0.591 | judges.jsonl against the gold; the other judges: Claude Opus 4.7 0.5147 / 0.5308, Mistral Medium 3.5 0.3272 / 0.3533, GPT-5.5 0.2659 / 0.3016. campaign.json (E3.3) |
| Served model: 33 design → 17 held-out contracts, macro-F1 / κ | — | 0.729 / κ 0.711 | The exact weights behind “Try the classifier”: same recipe, trained once on the 33 design contracts, evaluated on the 17 held-out contracts (design_holdout split). research/runs/demo_legalbert_T11_holdout/results/results.json |
The model behind “Try the classifier”
nlpaueb/legal-bert-base-uncased, fine-tuned once on the design set (33 contracts) with the paper's recipe (epochs 8, batchSize 16, learningRate 0.00002, maxLength 128, context ±1 sentence, seed 42, loss weighted_ce) and evaluated on the held-out set (17 contracts, never seen in training): macro-F1 0.729 [0.694; 0.761], κ 0.711.
Protocol and codebook
All 50 English contracts of CLAUDETTE (9,414 sentences) were labelled independently by three trained annotators working from a written codebook, with one primary theme and zero or more secondary themes from a closed 20-theme vocabulary. Four large language models labelled the same sentences in isolated sessions and were kept out of the reference. The gold standard is resolved strictly between annotators by a three-tier cascade: strict agreement, majority of two, human arbitration for the remainder. Coarser taxonomies are deterministic projections applied at read time, so every comparison is paired by construction; confidence intervals come from bootstrap resampling over documents. The merges were designed on 33 contracts and validated on the 17 held out from that design.
Twenty themes, one primary theme per sentence, secondary themes when a sentence genuinely does two things. A theme change starts a new clause.
The 11-theme consolidation
The annotation platform
Pactiva is the platform the layer was produced, arbitrated and exported with. Public projects are open to everyone; the annotation workshop, the gold arbitration cockpit and the experiment lab require an account.
Reviewer access
A shared reviewer account gives read access to the 150 annotation sessions of the campaign (frozen, with annotators shown as A1–A3), the human–judge comparison views and the concordance figures, plus a sandbox project on the same 50 contracts where you can annotate yourself; your sessions stay yours and never enter the released layer. The account has no access to configuration, experiments or exports.