You can deploy this model and start earning money today!

JohnMarble/
vi-en-glm

Name: vi-en-glm
Availability: InStock

safetensors

glm

omarkamali/wikipedia-monthly

unknown

Views

No views yet

vi-en-glm: Bilingual Small Language Model (English/Vietnamese)

This model is a Small Language Model (SLM) based on the GLM (General Language Model) architecture, trained from scratch to support both English and Vietnamese. It was developed as part of a NLP project to explore bilingual text generation and cross-lingual understanding in a resource-constrained environment.

Model Description

Language(s): Vietnamese, English
Model Type: Causal Language Model (GLM Architecture)
Parameters: ~200 Million
Vocabulary Size: 32,000 (BPE Tokenizer)

Training Details

Training Data

The model was trained on an interleaved dataset consisting of 52,000 English and 52,000 Vietnamese samples from Wikipedia. The datasets were cleaned to remove Wikipedia markdown and filtered for high-quality content longer than 100 characters.

Training Procedure

The training was conducted on 2 Kaggle T4 GPU utilizing fp16 precision and tf32 for optimized performance.

Epochs: 10
Batch Size: 64 (Effective Batch Size: 256 via Gradient Accumulation)
Learning Rate: 5e-4 with Cosine Scheduler
Optimizer: AdamW
Masking Strategy: GLM span masking (15% probability)

How to Use

python

from transformers import GlmForCausalLM, PreTrainedTokenizerFast

model = GlmForCausalLM.from_pretrained("JohnMarble/vi-en-glm")
tokenizer = PreTrainedTokenizerFast.from_pretrained("JohnMarble/vi-en-glm")

prompt = "Việt Nam là một quốc gia"
inputs = tokenizer(prompt, return_tensors="pt", return_token_type_ids=False)

outputs = model.generate(
**inputs,
max_length=50,
do_sample=True,
top_p=0.9,
temperature=0.7
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))