Nobody benchmarks German PII detection. I trained my own.

Every PII detector you can buy or download fails German text in a testable way. The public benchmarks measure English, the vendors promise 99 percent, and the identifiers a German firm actually redacts, SVNR, Steuer-Identifikationsnummer, KVNR, sit in nobody's training data. So I trained my own detector. It measures F1 0.916.

What do the public benchmarks measure?#

Start with PIIBench, the broadest benchmark corpus published so far: ten datasets merged into 2.37 million annotated sequences across 48 PII types. Its authors ran eight published systems against it, Microsoft Presidio among them. "All systems achieve span-level F1 below 0.14," the paper reports, and the best of them, Presidio at 0.1385, shows "zero recall on most entity types."

The strongest open model does better. GLiNER2-PII, a 0.3-billion-parameter model trained on 4,910 synthetic texts, reaches an average F1 of 0.471 on the SPY benchmark and beats OpenAI's privacy filter, which sits at 0.373. Both numbers are a long way from the 99 percent the marketing pages promise.

Read the fine print and German disappears. The GLiNER2-PII authors state their own limitation: "The evaluation covers only legal and medical documents," both English. PIIBench merges multilingual corpora but reports no per-language results. Nobody has published a German PII benchmark. The German text inside those merged corpora never received an evaluation of its own.

What the vendors promise instead#

Limina, the company until recently known as Private AI, states that its engine "misses between 0.2% and 7%" of PII entities where general-purpose solutions "miss between 13.8% and 46.5%." The site cites "99.5% Accuracy on physician conversations," names 52 languages, and lists Germany among its deployments. What the site does not name is a price. The call to action reads "Talk to an Expert."

Even taken at face value, those numbers tell you two things. First, the vendor's own marketing puts the miss rate of general-purpose tools at up to 46.5 percent, which matches what I saw running English-trained detectors over German client documents: the structured identifiers went through untouched. Second, serious redaction software in this market quotes its price after a sales call, if it quotes at all.

One more signal sits in the infrastructure. Presidio, the default answer for years, is no longer a Microsoft product. Its documentation moved to the community-run Data Privacy Stack organization. The standard free baseline is a volunteer project now.

The formats are public, so regex goes first#

A German identifier is not free text. The Rentenversicherungsnummer has a published shape: two digits, six digits, one letter, three digits. The Steuer-Identifikationsnummer is eleven digits and never starts with zero. Personalausweis and Reisepass numbers separate by letter pool under the rules in force since November 2021. Fourteen of these formats are encoded as regex patterns in my build, each pattern checked by hand against the official specification.

I would underline one sentence for anyone budgeting an LLM project: the language model decided nothing in this system. It helped around the edges, mapping formats, drafting generator variants, paraphrasing templates. It also confused the Steuernummer with every other digit run on the page. My first version, model only, scored F1 0.452 on tax identifiers. In the shipped build the regex layer decides on overlap, and the model gets what has no format: names, addresses, passwords in context.

Why the training data is synthetic#

The one material this project refuses to train on is real client documents, because redacting exactly that material is the point of the tool. So the corpus is synthetic, more than 50,000 examples generated from pattern families: addresses in German, Austrian and Swiss spellings, Steuernummer in the old regional and the unified form, plates, IBANs with plausible banks, plus sentences that contain no PII at all.

The sentences without PII are the point. A version trained without negatives saw personal data in every sentence, and its precision collapsed. The generators also track rule changes; the Ausweis and Reisepass letter pools are the current example.

Five epochs on a free T4#

The model layer is a fine-tuned GLiNER2, 307 million parameters on an mDeBERTa backbone. Training runs on Kaggle's free T4 tier, a 16 GB GPU, and a full fine-tune fits once the batch is sized for it. One epoch is one pass over the training set. My runs take five or more, early stopping holds back the best checkpoint, and a run costs about half an hour: fp16, cosine schedule, warmup. The GPU bill is zero.

The bugs were worth more than the hardware. An earlier version shipped learning rates ten to a hundred times below the GLiNER2 recommendations and trained on German descriptions while asking for English labels at inference. Both fixes were documented in the repo, and neither cost money.

What it measures now#

The 0.916 comes from my own benchmark: 188 German examples covering all 17 PII types, name-like strings in non-PII roles, partial IBANs, Austrian and Swiss formats. It is my benchmark, tuned to what firms actually send, ticket text, letters, forms. That is both its strength and the caveat. The public German benchmark does not exist, and building it was part of the work. The target for the next data revision is 0.95, with the fix list already written: learning rates, more negatives, English label descriptions.

The system runs in production at firms as a CLI over documents and ticket text. It has not been released as open source, so commercial use goes through info@dimitripisarev.com. The full architecture, the format tables and the eval protocol sit in the Schwärzwerk case study.

If you handle German customer text, ask any vendor one question: which benchmark measured your F1 on German? A percentage without a dataset is marketing. I hit the same wall with outreach tooling and built my own engine there too. The parts that touch your data should be the parts you can measure.