IrishCore-DiffMask-135M-v1-rc5

• PPSN
• ACCOUNT_NUMBER
• BANK_ROUTING_NUMBER
• CREDIT_DEBIT_CARD
• PASSPORT_NUMBER
• POSTCODE
• PHONE_NUMBER
• EMAIL
• FIRST_NAME
• LAST_NAME
• SWIFT_BIC

What "DiffMask" Means Here

• Base OpenMed: plain BIO token classification
• DiffMask: token-span extraction with token-presence and boundary heads
• DiffMask training: repeated masked denoising over the same sentence
• DiffMask inference: one forward pass, no iterative refinement, no text generation

• The encoder starts from the DistilBERT-family weights inside OpenMed/OpenMed-PII-mLiteClinical-Base-135M-v1.
• The model adds three task heads over the encoder hidden states:
  • a per-label token-presence head
  • a typed start-boundary head
  • a typed end-boundary head
• During training, each input sentence is corrupted multiple times by replacing a random fraction of visible tokens with [MASK].
• The corruption level follows a short noise schedule from heavy masking to light masking.
• The same gold spans are learned at every noise level, and the losses are averaged across the denoising passes.
• At inference time there is no diffusion loop and no rewrite step: the model runs once and a score-only span decoder reconstructs spans from token scores plus typed boundaries.

What It Is Not

• timestep or noise conditioning inside the model
• iterative denoising at inference time
• multi-step sequence refinement at runtime
• text generation or full-sequence reconstruction as a first-class objective

• corrupt the sentence with [MASK] at several noise levels
• train on the same target spans each time
• average those losses

Architecture

• Encoder: DistilBERT-size encoder from the OpenMed mLiteClinical 135M base
• Heads:
  • token presence per released label
  • typed start boundary per released label
  • typed end boundary per released label
• Decoder:
  • score-only span decoding from offsets, token continuity, label-specific thresholds, and typed boundaries
  • email spans do not bridge across whitespace
  • structured labels can bridge over a simple hyphen token when needed
  • name spans containing digits are rejected during decoding
  • PPSN whitespace bridging is restricted to short suffix tokens only
  • structured IBAN and card spans use conservative start/tail rescue logic for fragmented tokenization
  • no regex candidate extractor
  • no checksum validator
  • no scanner layer

Training And Inference Flow

1. tokenize a sentence with gold BIO spans
2. convert spans into:
   • token-presence targets
   • typed start targets
   • typed end targets
3. create several noised copies of the same tokenized sentence by masking random visible tokens
4. run the same encoder+heads on each noised copy
5. average the losses across those denoising passes

1. tokenize the raw text once
2. run a single forward pass
3. predict:
   • which labels are present on each token
   • where each labeled span starts
   • where each labeled span ends
4. decode spans with label-aware thresholds and boundary rules
5. replace the detected spans with placeholders such as [PII:PPSN]

How It Differs From The Original OpenMed Model

• one encoder
• one token-classification head
• BIO labels such as B-email, I-email, B-phone_number
• generic token aggregation to recover spans

1. Different supervision
   • base OpenMed learns only BIO token labels
   • DiffMask learns token presence plus typed span boundaries
2. Different training recipe
   • base OpenMed is trained as a standard token classifier
   • DiffMask is trained on multiple masked-noised views of the same sentence

How It Differs From rc5 And rc8

Why This Exists

References

• Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  https://arxiv.org/abs/1810.04805
• Sanh et al., DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
  https://arxiv.org/abs/1910.01108
• Fu et al., Boundary Smoothing for Named Entity Recognition
  https://aclanthology.org/2022.acl-long.490/
• Wang et al., SPANNER: Named Entity Re-/Recognition as Span Prediction
  https://aclanthology.org/2021.acl-long.558/

• Nie et al., LLaDA 2.0: Scaling Up Diffusion Language Models to 100B
  https://arxiv.org/abs/2512.15745
• Gong et al., Scaling Diffusion Language Models via Adaptation from Autoregressive Models
  https://arxiv.org/abs/2410.17891

Included Artifacts

• Full transformers checkpoint in the repo root
• Dynamic q8 ONNX export in onnx/model_quantized.onnx
• Unquantized ONNX export in onnx/model.onnx
• inference_mask.py for the full checkpoint
• inference_mask_onnx.py for the ONNX q8 path
• common.py, model.py, and multitask_model.py implementing the release decoder
• benchmark files in eval/

• Full checkpoint: 514 MB (model.safetensors)
• Dynamic q8 ONNX: 393 MB (onnx/model_quantized.onnx)

How To Use It

Q8 Comparison

• The rc5 speed references come from its published q8 end-to-end inference stack, which includes its older repair decoder.
• The rc8 and IrishCore-DiffMask-135M-v1-rc5 numbers use the same raw-only token-span ONNX path.
• A weight-only q4 ONNX experiment was also tried during development, but it was slower than q8 on this CPU and is not shipped.

Limits

• This is still a compact model. The hardest remaining errors are multilingual PPSN near-miss cases rather than Irish core numeric formats.
• The release path is intentionally scanner-free. If you need deterministic validation of individual identifier types, add that in your application layer.
• If you rely on release behavior, use the bundled inference scripts or import decode_token_presence_segments from common.py.
• rc5 resolves the post-rc3 QA feedback suite, but it still has known misses on a few longer UAT-style messages:
  • the second phone number in a two-phone support sentence
  • one multiline address block with R93 EC57
  • EPStamp4@enterprise.gov.ie in the longer employment-permit example

License And Attribution

• Release license: Apache-2.0
• Base model: OpenMed/OpenMed-PII-mLiteClinical-Base-135M-v1
• The derivative release remains subject to the attribution terms of the upstream datasets listed above.
• See NOTICE, training_sources.json, and eval/benchmark_summary.json for provenance and benchmark details.

Portfolio Comparison

• The first core table only includes public checkpoints that ship both comparable q8 accuracy and q8 CPU throughput.
• The first PPSN table only includes public artifacts that ship comparable PPSN accuracy and CPU throughput.
• Missing cells in the archive tables mean the older release did not ship that metric in its public bundle.
• DiffMask rows use the reconciled clean_single_pass harness that matches the deployed runtime.
• GlobalPointer rows use the public raw-only span-matrix release bundle and its packaged q8 ONNX artifact.
• The same content is shipped as PORTFOLIO_COMPARISON.md inside each public model repo.

Irish Core PII: Comparable Public Checkpoints

Irish Core PII: Other Public Checkpoints

PPSN-Only: Comparable Public Artifacts

PPSN-Only: Historical Public Checkpoints