EGC 2027 · companion page for reviewers
Executing the grey list
Explicit legal knowledge and the price of interpretability in unfair-term detection
This page gives reviewers everything the paper releases: the thematic layer annotated on the 50 CLAUDETTE contracts, with the individual votes of the three annotators, the gold standard and the arbitration trail of every sentence, the four LLM judges and their exact prompts, the frozen taxonomies, the codebook and the annotation protocol. The platform that produced the layer is open for reading: you can browse the annotations, see how every label was reached and annotate yourself in a sandbox.
Dataset fingerprint 7116e627f528c557 · 50 contracts · 9,414 sentences · 3 trained annotators · 4 LLM judges
Try the classifier
Legal-BERT fine-tuned with the paper's recipe on the 33 design contracts (11-theme taxonomy, ±1 sentence of context, 8 epochs, seed 42). The 17 contracts offered below were never seen during training. Your text is processed once and not stored.
0 / 60,000 characters · about 0 sentences
Legal-BERT (11 themes), trained on the 33 design contracts. Your text is processed once and not stored.
What we release
All files below are the ones every figure of the paper was computed from. Records are keyed by (document, index) and join directly onto the CLAUDETTE corpus, whose sentences we do not redistribute. Annotators appear as A1, A2 and A3.
Everything in one archive
thematic-layer-7116e627.zip · 383 KB · SHA-256 c8d3170c508c7b70…
Individual votes
Individual annotator votes (A1–A3): one record per (document, sentence, annotator), primary theme and secondary themes.
votes.jsonl · 28,242 records · 2.9 MB
DownloadGold standard and arbitration trail
Frozen gold standard and arbitration trail: agreement class, cascade tier, engine proposal, decided themes, vote tally, confidence.
gold.jsonl · 9,414 records · 3.4 MB
DownloadLLM judges
The four LLM judges on the same sentences and vocabulary.
judges.jsonl · 37,656 records · 4.0 MB
DownloadFrozen taxonomies
Frozen taxonomy specification: T20 and the T14 / T11 / T10 projections, design / held-out partition.
taxonomies.json · 27 KB
DownloadJudge prompts
The instruction files given to the judges (protocol v9.2) and the session protocol: inputs, isolation, validation, models, harnesses and dates for Claude Fable 5, Claude Opus 4.7, Mistral Medium 3.5, GPT-5.5.
Codebook
The 20 themes with their definitions, the segmentation and multi-label rules given to annotators.
ReadProtocol
Independent annotation, isolated judges, three-tier resolution cascade, read-time projections, document-level bootstrap.
ReadManifest, labels and splits are included in the archive. Dataset fingerprint 7116e627f528c557… · built 2026-10-18.
Key figures
Generated from the released files; open “how computed” for the source of each number.
| Measure | 20 themes | 11 themes | How computed |
|---|---|---|---|
| Krippendorff's α (MASI distance), three annotators | 0.658 | 0.725 | α computed on the individual votes (votes.jsonl) with the MASI distance on the set {primary} ∪ secondaries; the 11-theme value is the same votes projected through taxonomies.json. docs/pactiva-taxonomies/resultats/taxonomy_matrix.json (population all, raw) |
| Nominal α on primary themes | 0.732 | 0.794 | Same votes, primary theme only, nominal distance; the gap with α-MASI is the cost of the multi-label format. taxonomy_matrix.json (raw) |
| Gain from consolidating 20 → 11 themes (α-MASI) | — | 0.067 | Paired difference of α-MASI between the T20 votes and their T11 projection (bootstrap CI over documents in the paper). taxonomy_matrix.json |
| Human ceiling, κ (leave-one-annotator-out against the gold) | 0.859 | — | Each annotator scored against a gold rebuilt without her votes; mean Cohen's κ over the three annotators (accuracy 0.8705). frontend/src/features/paper/campaign.json (E1.5) |
| Legal-BERT fine-tuned, κ (5-fold, grouped by document) | 0.695 | 0.720 | nlpaueb/legal-bert-base-uncased, 8 epochs, ±1 sentence of context, weighted cross-entropy, seed 42, 5-fold cross-validation grouped by document on Grid'5000; macro-F1 0.612707 / 0.734652. research/runs/legalbert_T20/results.json, research/runs/legalbert_T11/results.json |
| Best LLM judge (Claude Fable 5), κ | 0.581 | 0.591 | judges.jsonl against the gold; the other judges: Claude Opus 4.7 0.5147 / 0.5308, Mistral Medium 3.5 0.3272 / 0.3533, GPT-5.5 0.2659 / 0.3016. campaign.json (E3.3) |
| Served model: 33 design → 17 held-out contracts, macro-F1 / κ | — | 0.729 / κ 0.711 | The exact weights behind “Try the classifier”: same recipe, trained once on the 33 design contracts, evaluated on the 17 held-out contracts (design_holdout split). research/runs/demo_legalbert_T11_holdout/results/results.json |
The model behind “Try the classifier”
nlpaueb/legal-bert-base-uncased, fine-tuned once on the design set (33 contracts) with the paper's recipe (epochs 8, batchSize 16, learningRate 0.00002, maxLength 128, context ±1 sentence, seed 42, loss weighted_ce) and evaluated on the held-out set (17 contracts, never seen in training): macro-F1 0.729 [0.694; 0.761], κ 0.711.
Protocol and codebook
All 50 English contracts of CLAUDETTE (9,414 sentences) were labelled independently by three trained annotators working from a written codebook, with one primary theme and zero or more secondary themes from a closed 20-theme vocabulary. Four large language models labelled the same sentences in isolated sessions and were kept out of the reference. The gold standard is resolved strictly between annotators by a three-tier cascade: strict agreement, majority of two, human arbitration for the remainder. Coarser taxonomies are deterministic projections applied at read time, so every comparison is paired by construction; confidence intervals come from bootstrap resampling over documents. The merges were designed on 33 contracts and validated on the 17 held out from that design.
Twenty themes, one primary theme per sentence, secondary themes when a sentence genuinely does two things. A theme change starts a new clause.
The 11-theme consolidation
The annotation platform
Pactiva is the platform the layer was produced, arbitrated and exported with. It is open to reviewers so that the annotation work can be checked at its source.
Reviewer access
- browse the 150 annotation sessions of the campaign, frozen, with annotators shown as A1–A3;
- trace, for every sentence, how its label was reached: the three annotators' votes, the tally, the agreement class and, where needed, the arbitration;
- open the human–judge comparison views and the concordance figures;
- annotate yourself in a sandbox project on the same 50 contracts: your sessions stay yours and never enter the released layer.
The reviewer account has no access to configuration, experiments or exports.