Upload training data
JSONL, JSON, Excel or CSV. Three label shapes are recognised:
a scores object per record, one spreadsheet column per
entity, or a single class column. The mapping is detected and you can
correct it below.
How this file is labelled
What the file contains
Can each class be measured?
The bar shows the credible interval you can expect on that class's MCC, given how many examples land in the test fold. Hatched area is below the resolution floor — a result in there cannot support a conclusion.
Languages
A single aggregate score hides a language that is failing. Runs report metrics per language wherever there are enough documents to measure one.
Findings
Errors block the run. Fix them at source where you can — every one of these is a defect in the export, not in the model.
Choose what to train
Settings
DeBERTa-v3 produces NaN gradients
under transformers 5. Run diagnose.py after changing this
if a run aborts. fp32 is roughly twice as slow
but DeBERTa-v3 diverges under mixed precision when a class weight is
large. If a run aborts with "training diverged", lower the learning
rate or the weight cap rather than retrying. A single seed cannot be
compared to anything — split-to-split swing on data this size reaches
0.09 MCC. Repeating across seeds reports a mean and spread instead.
Runs
Training continues if you close this tab. Progress updates every two seconds.
Label review
Every case the model got wrong on held-out data. Judge each one: if the gold label is wrong, that is a correction for the annotation set. If the guidelines never covered it, that is a guideline gap. Both are more valuable than a model tweak.
Your work
Local LLM
Labelling runs against an Ollama server on this machine. Nothing leaves the box.
What the corpus calls each entity
The regex is the BASELINE this project's uplift is measured against, so a pattern that under-matches flatters the model - and the same patterns decide `silent_on_mention`, where a missed variant reads as the model ignoring something nobody mentioned. Read the misses: they say what the corpus actually calls the thing.
Project and taxonomy
The class descriptions are your annotation guidelines — the LLM and your annotators must work from the same words. Editing one changes the standard, so prompts are versioned and earlier labels are flagged stale.
Draft a taxonomy with the model
The model proposes; you accept. Generated definitions become the annotation manual for both the model and the humans, so a vague one is not noise - it is a systematic error every later metric reports as a real finding.
Facets to choose from
Taxonomy in force
The guideline text sent to the LLM, and the prefilter that selects candidates. Editing a description changes the annotation manual for both the model and the humans — and every metric measured under the old wording is pooled with the new one until you re-verify.
Generated prompt
Built from the descriptions above and sent to Ollama with a JSON schema, so the response is parsed rather than scraped.
The prompt in force — what the next batch will send
JSON schema
Post pool
Unlabelled posts to draw batches from. CSV, Excel or JSONL — only a text column is needed. Exact duplicates are linked, not stored twice, so you never pay for a copy.
Candidate selection
Random sampling is fine at 30% prevalence and hopeless at 3%. Selection happens before the LLM, so the model only labels posts worth labelling. Whether it pays on THIS corpus is measured below — the “Is targeting paying off?” table is the figure to act on.
Where the 7.8x figure comes from (a different pool)
On a separate 5,000-post pool with a 2.9% class: a keyword prefilter reached 22.6% precision (7.8x), embedding retrieval 10.9% (3.8x), the two combined 28.0% (9.7x) — turning ~50,700 posts of labelling into ~4,900. Not a prediction for your corpus. A class that is common, or one your keywords do not separate, can measure below 1x here.
Label a batch
Draws unlabelled posts from the pool, labels them, and queues the results for checking. Classes that have hit their target stop being served, apart from a 5% trickle so drift stays visible.
Feedback from labellers
What they wrote while deciding, for this project. Notes on cells they AGREED with are included — a near-miss is where a guideline is about to fail. This lives here rather than on Label review because most labelling happens long before any run exists, and a note is about the guidelines, not about a model.
Verify
One decision per cell, not per post — on a 3-D project a cell is one entity and one aspect, and a single yes/no across several tells you nothing about which was wrong. Each card names the pair it is asking about. Some items arrive blind, with the model's answer hidden, so you label from scratch; those give the only unbiased accuracy estimate. Keys: 1 agree, 2 reject, 3 fix, ← back.
Coverage
Synthetic examples
Which aspects can the detector not be measured on?
Export and train
Assessment
Measures a fine-tuned model on data it has never seen. An assessment is a (dataset, model, prompt) triple — each named explicitly, because a number that does not say which three it came from cannot be compared with another.
New assessment dataset
Upload a fresh pull with the model's predictions in it. 1k–5k records: the file caps how many rare-class cells exist to draw from, while the target per stratum sets how many you spend — so a larger file costs no more human time.
Report
Tools
Diagnostics you can run from here, and the ones that belong in a shell. Nothing on this tab writes to a project.
Dry run shows the counts and changes nothing. Applying needs the phrase typed, and the server also requires that a dry run of the same target was shown in the last ten minutes.
Calibrate against labels you already trust
Run the prompt over a stratified sample of an already-labelled file and compare with the human gold. You are about to distil this LLM into a smaller model, so it cannot beat the LLM minus noise. This is the cheapest way to find out whether the rest is worth doing — and which classes need humans regardless.
Deploy
1 · Take a run
2 · What was built, and what was refused
4 · Against human, LLM and production
Users
A new user gets a single-use invite link and chooses their own password on first sign-in — no password is ever set for them or sent to them. Disable rather than delete: past verification decisions are attributed to a user, and removing the account would orphan them.
Annotator agreement
Per-person agreement, and where two people judged the same cell, agreement between them — the human ceiling. Without it a model score cannot be told from parity with your own annotators.
Defaults for new runs
Each run records the config it used, so changing these never alters a past result.
Activity
Logins, user changes, settings changes, and refused requests.