LiLTInvoiceCzech (V3 – Full Pipeline with Real Data Fine-Tuning)
This model is a fine-tuned version of
SCUT-DLVCLab/lilt-roberta-en-base for structured information extraction from Czech invoices.
It achieves the following results on the evaluation set:
- Loss: 0.0358
- Precision: 0.8840
- Recall: 0.8976
- F1: 0.8908
- Accuracy: 0.9910
Model description
LiLTInvoiceCzech (V3) is the final and best-performing model in the experimental pipeline.
The model performs token-level classification using both textual and spatial (bounding box) information to extract structured invoice fields:
- supplier
- customer
- invoice number
- bank details
- totals
- dates
By combining layout-aware architecture with progressively more realistic data, this version achieves strong performance on real-world-like documents.
Training data
The dataset used in this stage combines:
- Synthetic template-based invoices (V0)
- Synthetic invoices with randomized layouts (V1)
- Hybrid invoices with real layouts and synthetic content (V2)
- Real annotated invoices
Real data fine-tuning
The final stage introduces:
- real invoice documents
- annotated entity spans
- natural linguistic variability
- real formatting inconsistencies and layout noise
This enables the model to:
- adapt to real-world distributions
- leverage both spatial and textual patterns
- achieve high robustness and generalization
Role in the pipeline
This model corresponds to:
V3 – Full pipeline (synthetic + hybrid + real data fine-tuning)
It represents:
- the final model in the LiLT branch
- the best-performing configuration
- a production-ready layout-aware solution
Intended uses
- Real-world invoice information extraction
- Document AI systems with layout awareness
- OCR post-processing pipelines with spatial features
- Benchmarking layout-aware architectures
Limitations
- Depends on quality of OCR and bounding box extraction
- May struggle with:
- extremely noisy scans
- highly non-standard invoice formats
- Domain-specific (Czech invoices)
- Requires structured input (tokens + bounding boxes)
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 16
- eval_batch_size: 2
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 0.1
- num_epochs: 10
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1 | Accuracy |
|---|
| No log | 1.0 | 12 | 0.0636 | 0.7820 | 0.8140 | 0.7977 | 0.9819 |
| No log | 2.0 | 24 | 0.0472 | 0.8499 | 0.8020 | 0.8253 | 0.9855 |
| No log | 3.0 | 36 | 0.0446 | 0.8293 | 0.8874 | 0.8574 | 0.9873 |
| No log | 4.0 | 48 | 0.0393 | 0.8555 | 0.8788 | 0.8670 | 0.9891 |
| No log | 5.0 | 60 | 0.0359 | 0.8872 | 0.8720 | 0.8795 | 0.9905 |
| No log | 6.0 | 72 | 0.0366 | 0.8870 | 0.8840 | 0.8855 | 0.9905 |
| No log | 7.0 | 84 | 0.0358 | 0.8826 | 0.8976 | 0.8900 | 0.9909 |
| No log | 8.0 | 96 | 0.0360 | 0.8822 | 0.8942 | 0.8881 | 0.9907 |
| No log | 9.0 | 108 | 0.0374 | 0.8696 | 0.8993 | 0.8842 | 0.9904 |
| No log | 10.0 | 120 | 0.0366 | 0.8783 | 0.8993 | 0.8887 | 0.9908 |
Framework versions
- Transformers 5.0.0
- PyTorch 2.10.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2
