TELEN introduces a novel embedding architecture designed specifically for Vietnamese legal text retrieval in RAG (Retrieval-Augmented Generation) systems. Unlike conventional static embedding models, TELEN generates embeddings that adapt dynamically to the current state of the legal corpus — enabling seamless integration of new laws without retraining.
Key Innovations
HyperNetwork-Driven Projection — Instead of fixed projection weights, a HyperNetwork generates the embedding projection function from the current legal corpus state. When new laws are published, the embedding space adapts automatically.
Legal Concept Graph (LCG) — An evolving knowledge graph where nodes represent legal entities (laws, key terms) and edges encode cross-references, agency hierarchy, temporal sequences, and semantic similarity.
State-Adaptive Embeddings — Embeddings are not static vectors but are modulated by a learned "legal state vector" that summarizes the entire legal landscape at any point in time.
Architecture
Legal Text
↓
Bi-Encoder (bkai-foundation-models/vietnamese-bi-encoder)
↓
Raw Representation [768-dim]
↓
┌─────────────────────────────────────┐
│ HyperNetwork(state_vector) → ΔW, Δb │ ← Generated, not learned!
│ Adapted Projection = Base + ΔW·x + Δb │
└─────────────────────────────────────┘
↓
Legal Concept Graph (GNN)
↓ state_vector
State Encoder ← current legal corpus
↓
L2-Normalized Embedding [768-dim]
Benchmark Results
Test set: 1,406 Vietnamese legal articles from 2021 (held-out, unseen during training)
Model
NDCG@3
NDCG@5
NDCG@10
MRR@3
MRR@5
MRR@10
BM25 (lexical)
0.6753
0.7173
0.7250
0.6683
0.6928
0.6990
PhoBERT-base-v2 (monolingual dense)
0.5866
0.6360
0.6505
0.5657
0.5970
0.6059
multilingual-E5-base (multilingual dense)
0.4675
0.4888
0.5157
0.4327
0.4452
0.4573
BAAI/bge-m3 (multilingual dense, 1024d)
0.4668
0.5129
0.5452
0.4407
0.4657
0.4802
DEk21 (legal dense)
0.7900
0.8127
0.8344
0.7660
0.7785
0.7865
TELEN (adaptive dense)
0.9036
0.9138
0.9132
0.8830
0.8878
0.8878
TELEN + CE re-rank (adaptive dense)
0.9346
0.9339
0.9238
0.9199
0.9223
0.9223
Key insight: Multilingual SOTA models (multilingual-E5, BGE-M3) score below even BM25 on Vietnamese legal text, confirming that domain and language specialization trumps generic multilingual pre-training for legal retrieval.
Relative Improvement
Baseline
NDCG@3
NDCG@5
NDCG@10
MRR@10
vs multilingual-E5
+93.3%
+86.9%
+77.1%
+94.1%
vs BGE-M3
+93.6%
+78.2%
+67.5%
+84.9%
vs PhoBERT
+59.3%
+46.8%
+42.0%
+52.2%
vs DEk21
+18.3%
+14.9%
+10.7%
+17.3%
Quick Start
Installation
pip install -r requirements.txt
Inference
python
1from inference import TELENInference
23# Load model4model = TELENInference()56# Encode legal texts7texts =[8"Điều 1: Thông tư này quy định về quản lý thuế giá trị gia tăng...",9"Điều 2: Đối tượng áp dụng là các tổ chức, cá nhân kinh doanh...",10]11embeddings = model.encode(texts)# → [2, 768] normalized vectors1213# Compute similarity14similarity = model.similarity(texts[0], texts[1])15print(f"Cosine similarity: {similarity:.4f}")1617# Retrieve similar documents18results = model.retrieve(texts[0], corpus, top_k=10)
Format: Word-segmented Vietnamese text (underscore-separated compound words)
Training Pipeline
Stage
Description
Epochs
Trainable Params
1. Contrastive Pretraining
Triplet + InfoNCE loss on same-law article pairs
5
~1M (projection head)
2. Meta-Training
HyperNetwork learns to adapt embedding space for future laws
50 (early stop)
~4M (HyperNetwork + State Encoder)
Hyperparameters
Parameter
Value
Backbone
bkai-foundation-models/vietnamese-bi-encoder
Embedding dimension
768
Adaptation rank
64
GNN layers
3
Meta N-way, K-shot
16-way, 5-shot
Negatives per query
256 (50% hard + 50% random)
Temperature
0.05
Optimizer
AdamW + CosineAnnealingWarmRestarts
Hardware
GPU: NVIDIA RTX 5070 Ti (16GB VRAM)
Training time: ~8 hours (5 contrastive + 50 meta epochs)
Continuous Adaptation
When a new law is published, TELEN adapts without retraining:
python
1# New law arrives2new_articles =[3"Điều 1: Luật mới về trí tuệ nhân tạo...",4"Điều 2: Các nguyên tắc áp dụng AI trong xét xử...",5]67# Update concept graph (milliseconds)8model.add_new_law("123/2025/l-ai", new_articles)910# Embedding space automatically adapts via HyperNetwork11# All subsequent query embeddings reflect the new legal landscape12embeddings = model.encode(["Điều 1: ..."])
