Data-to-Text Executive Report Generator — FLAN-T5 Base
A controlled table-to-text generation model for converting CSV/JSON KPI tables into concise executive reports.
The system combines a fine-tuned FLAN-T5 Base generator with deterministic KPI analysis, row-level source references, structural normalization, and a numerical claim verifier. Unsupported model claims are blocked or deterministically replaced before they are displayed.
Published artifact: merged FLAN-T5 Base fine-tuned model used by Project 04.
Live Resources
Evaluation Overview
The final PyTorch model was evaluated on 250 held-out KPI-to-report examples.
The application does not trust raw generated claims blindly. Model-generated claim plans pass through a deterministic verifier that checks row references and numerical support before final display.
Final 250-Example Test Results
| Metric | Result |
|---|
| BLEU | 71.46 |
| ROUGE-L | 0.8125 |
| BERTScore F1 | 0.9714 |
| JSON validity | 100.00% |
| Claim F1 | 98.72% |
| Row-ID F1 | 98.72% |
| Structured-field F1 | 98.85% |
| Exact plan match | 0.40% |
| Structured composite score | 99.19% |
| Final section coverage | 100.00% |
| Composite portfolio score | 84.78% |
Factuality and Verification Results
| Metric | Result | Interpretation |
|---|
| Raw model claim support rate | 67.25% | Claims supported before verifier intervention |
| Raw numerical factuality | 56.33% | Numerical correctness before final verification |
| Raw unsupported / hallucination rate | 32.75% | Unsupported raw model claims detected |
| Verifier block rate | 32.75% | Unsupported claims intercepted by verifier |
| Model claim retention rate | 67.25% | Generated claims retained without replacement |
| Deterministic replacement rate | 19.17% | Unsupported content replaced with supported facts |
| Final displayed claim support rate | 100.00% | All claims shown to the user were source-supported |
| Final section coverage | 100.00% | Required executive-report sections remained covered |
What These Results Mean
The raw generator is not treated as a standalone factual authority. It is intentionally deployed behind a numerical/source verifier.
The most important deployment result is:
100% final displayed claim support on the 250-example evaluation after verifier gating.
The verifier prevented unsupported raw claims from reaching the final report while preserving complete section coverage.
ONNX Optimization
The final model was exported to ONNX and an INT8 quantized version was produced.
Runtime Smoke Evaluation
| Runtime | Eval Examples | BLEU | ROUGE-L | BERTScore F1 | JSON Validity | Claim F1 | Final Claim Support | Avg. Generation |
|---|
| PyTorch / CUDA | 250 | 71.46 | 0.8125 | 0.9714 | 100% | 98.72% | 100% | 3.62 s/example |
| FP32 ONNX | 5 | 66.90 | 0.7972 | 0.9679 | 100% | 100% | 100% | 9.08 s/example |
| INT8 ONNX | 5 | 64.26 | 0.7781 | 0.9656 | 100% | 100% | 100% | 8.91 s/example |
The PyTorch result is the full 250-example quality evaluation. The ONNX rows are deliberately small 5-example runtime/parity smoke tests, not statistically equivalent quality benchmarks.
Quantization
| Artifact | Result |
|---|
| FP32 ONNX size | ~2.33 GB |
| INT8 ONNX size | ~1.07 GB |
| Size reduction | 54.03% |
| FP32 runtime generation | Passed |
| INT8 runtime generation | Passed |
In this local evaluation environment, ONNX CPU-style generation was slower than CUDA PyTorch generation. ONNX is therefore retained as an export/optimization artifact rather than being presented as a latency improvement.
Training Summary
| Setting | Value |
|---|
| Base model | google/flan-t5-base |
| Architecture | T5 encoder-decoder |
| Fine-tuning | LoRA / PEFT |
| LoRA rank | 32 |
| LoRA alpha | 64 |
| LoRA dropout | 0.05 |
| Training examples used | 6,000 |
| Epochs | 2 |
| Learning rate | 3e-4 |
| Max source length | 1,280 tokens |
| Max target length | 512 tokens |
| Precision | BF16 |
| GPU | NVIDIA GeForce RTX 5090 |
| Final training loss | 1.0247 |
| Final evaluation loss | 0.8505 |
| Generation beams during final training profile | 1 |
The final published checkpoint is the merged model and can be loaded directly with transformers.
Dataset and Verification
The project generated and validated a synthetic business-KPI corpus designed for controlled table-to-text generation.
| Dataset Check | Result |
|---|
| Total generated examples | 20,000 |
| Training split | 16,000 |
| Validation split | 2,000 |
| Test split | 2,000 |
| Cross-split table-ID overlap | 0 |
| Duplicate prompts across splits | 0 |
| Verified target claims | 159,327 |
| Target claim support rate | 100% |
| Challenge cases | 60 |
| Challenge target support rate | 100% |
The final fine-tuning run used a 6,000-example subset of the verified training split.
System Architecture
1CSV / JSON KPI Table
2 ↓
3Parsing + Validation
4 ↓
5Schema Detection
6 ↓
7Deterministic KPI Analysis
8 ↓
9Controlled Prompt + Fact Index
10 ↓
11Fine-Tuned FLAN-T5 Base
12 ↓
13Lean Structured Claim Plan
14 ↓
15Structural Normalization
16 ↓
17Deterministic Fact Hydration
18 ↓
19Numerical / Source Claim Verifier
20 ↓
21Unsupported Claim Blocking / Replacement
22 ↓
23Verified Executive Report
The model generates a compact claim plan containing section codes, claim types, source row IDs, and report text. Numerical values and factual fields can then be reconstructed from verified source rows before final claim validation.
Why a Verifier-Gated Architecture?
A fluent generated sentence can still contain an unsupported number or attribution.
For executive KPI reporting, the project therefore separates:
- Language generation — FLAN-T5 determines how to express the report.
- Deterministic analysis — KPI movements and source facts are calculated outside the model.
- Evidence mapping — generated claims reference source rows.
- Verification — unsupported claims are blocked.
- Hydration/replacement — supported factual content is reconstructed when necessary.
This design prioritizes final report factuality over trusting unconstrained generation.
Example Usage
1from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
2
3repo_id = "anmol-unitmole/data-to-text-executive-report-generator-flan-t5-base"
4
5tokenizer = AutoTokenizer.from_pretrained(repo_id)
6model = AutoModelForSeq2SeqLM.from_pretrained(repo_id)
7
8prompt = """
9Generate a structured executive KPI claim plan from the supplied
10table analysis and source-row references.
11"""
12
13inputs = tokenizer(
14 prompt,
15 return_tensors="pt",
16 truncation=True,
17 max_length=1280,
18)
19
20outputs = model.generate(
21 **inputs,
22 max_new_tokens=512,
23 num_beams=1,
24)
25
26print(tokenizer.decode(outputs[0], skip_special_tokens=True))
The full Project 04 application performs additional deterministic hydration and claim verification after model generation. For production-style use, use the complete GitHub pipeline rather than treating the raw model output as the final verified report.
Model Selection and Deployment Decision
The final candidate was selected because it demonstrated:
- strong text similarity and semantic quality,
- near-perfect structured claim-plan extraction,
- complete final section coverage,
- deterministic source-row traceability,
- verifier interception of unsupported claims,
- 100% final displayed claim support in the 250-example evaluation,
- successful FP32 ONNX export,
- successful INT8 ONNX quantization and runtime smoke validation.
The project does not claim that raw FLAN-T5 generations are perfectly factual. The deployment design explicitly relies on the verifier.
Experimental Environment
Training and evaluation were executed locally using:
| Component | Environment |
|---|
| GPU | NVIDIA GeForce RTX 5090 |
| VRAM | ~31.84 GB |
| Python | 3.12 |
| PyTorch | 2.11.0 + CUDA 12.8 |
| Transformers | 4.57.6 |
| PEFT | 0.20.0 |
| Mixed precision | BF16 |
Quality Gates
| Validation | Status |
|---|
| Python unit tests | ✅ 19 / 19 passed |
| Dataset validation | ✅ Passed |
| Challenge-set verification | ✅ Passed |
| Source release validation | ✅ Passed |
| PyTorch final evaluation | ✅ Completed |
| FP32 ONNX export | ✅ Passed |
| FP32 ONNX generation smoke | ✅ Passed |
| INT8 quantization | ✅ Passed |
| INT8 ONNX generation smoke | ✅ Passed |
| GitHub Actions CI | ✅ Passing |
Intended Use
This model and project are suitable for:
- KPI table summarization experiments
- executive-report generation demonstrations
- controlled table-to-text generation research
- source-grounded generation workflows
- numerical-verification experiments
- ONNX export and quantization experiments
- machine-learning portfolio demonstrations
- educational use
Limitations
- Raw model generations can contain unsupported claims before verifier gating.
- Raw numerical factuality is substantially lower than final verifier-gated factuality.
- The current dataset is synthetic/artificial business KPI data and does not represent every real enterprise reporting format.
- ONNX quality was smoke-tested on 5 examples; the 250-example full evaluation was performed with the PyTorch model.
- ONNX generation was slower than CUDA PyTorch generation in the measured local environment.
- The model should not be used as a standalone authority for financial, legal, compliance, medical, safety-critical, or other high-stakes decisions.
- Human review remains appropriate for consequential reporting.
Responsible Use
The model is intended for educational, portfolio, research, analytics-assistance, and general business-reporting experiments.
Generated reports should be reviewed before consequential use. The complete application includes verification specifically because fluent language generation alone is not sufficient evidence of factual correctness.
Reproducibility
The GitHub project contains code for:
- synthetic KPI dataset generation,
- leakage and overlap validation,
- challenge-set generation,
- FLAN-T5 LoRA fine-tuning,
- structured claim-plan generation,
- structural normalization,
- numerical/source verification,
- unsupported-claim blocking,
- content-selection evaluation,
- numerical factuality evaluation,
- hallucination-rate evaluation,
- BLEU / ROUGE-L / BERTScore evaluation,
- ONNX export,
- INT8 quantization,
- runtime validation,
- automated Python tests,
- Next.js application code,
- GitHub Actions validation.
Complete implementation:
License
Apache License 2.0.
