inacio-sft-d16v1
inacio-sft-d16v1 is the first instruction-tuned model of the nanochatpt project. It is obtained by supervised fine-tuning (SFT) of the base model inacio-d16v1, with the objective of improving conversational abilities while maintaining European Portuguese (PT-PT) as the primary language.
This release represents the first end-to-end demonstration of the complete nanochatpt training pipeline, from tokenizer training to pretraining and supervised fine-tuning.
Base Model
This model is fine-tuned from:
The base model was pretrained from scratch on a corpus primarily composed of European Portuguese text, together with a smaller proportion of English.
Model Architecture
- Architecture: NanoChat
- Decoder-only Transformer
- Parameters: 872,416,058
- Layers: 16
- Hidden size: 1024
- Attention heads: 8
- KV heads: 8
- Context length: 2048 tokens
- Vocabulary: 65,536 tokens
Supervised Fine-Tuning Dataset
The instruction tuning stage uses NanoChat's TaskMixture framework.
Training data consists of:
- CAIS MMLU (English)
- OpenAI GSM8K (English)
- inaciose/smol-smoltalk-pt-v1 (Mix European Portuguese 80%, English 20%)
The Portuguese conversational dataset smol-smoltalk-pt-v1 was created specifically for this project as a temporary PT-PT replacement for HuggingFaceTB/smol-smoltalk.
It is derived from:
- AMALIA-LLM-0626-SFT-Dataset
Future releases will progressively replace the remaining English instruction datasets with Portuguese equivalents.
Training
Supervised fine-tuning was performed on the Modal cloud platform using:
- 4 × NVIDIA A100 SXM4 80GB
The model was initialized directly from the pretrained checkpoint of inacio-d16v1.
Intended Uses
This model is intended for:
- conversational assistants
- experimentation
- educational purposes
- research on European Portuguese language models
Current Capabilities
The model is capable of:
- maintaining multi-turn conversations
- following simple instructions
- generating structured answers
- responding primarily in Portuguese
Compared to the base model, it produces considerably more natural conversational responses and follows instruction formats learned during supervised fine-tuning.
Current Limitations
This is the first public SFT release of nanochatpt.
The model still presents significant limitations, including:
- factual hallucinations
- weak mathematical reasoning
- inconsistent scientific explanations
- occasional repetition
- occasional generation of special conversation tokens (e.g.
<|assistant_end|>)
These behaviours are expected for a first-generation instruction-tuned model trained on a relatively small Portuguese conversational corpus.
Future releases may improve both the instruction dataset and the amount of Portuguese pretraining data.
Requeriments
nanochatpt or nanochat cloned to your machine, install requirements.
https://github.com/inaciose/nanochatptcopy the model to your local folder
copy the tokenizer to your local folder
cache/modal
├── chatsft_checkpoints
│ └── d16
│ ├── meta_0XXXXX.json
│ ├── model_0XXXXX.pt
│ ├── optim_0XXXXX_rank0.pt
│ ├── optim_0XXXXX_rank1.pt
│ ├── optim_0XXXXX_rank2.pt
│ └── optim_0XXXXX_rank3.pt
└── tokenizer
├── token_bytes.pt
└── tokenizer.pkl
export NANOCHAT_BASE_DIR=cache/modal
run the python -m scripts/chat_cli.py
Project
nanochatpt is an personal open research project whose objective is to train compact language models from scratch with European Portuguese as the primary language.
Future work includes:
- larger pretraining corpora
- improved Portuguese instruction datasets
- Portuguese evaluation benchmarks
- additional supervised fine-tuning
- reinforcement learning alignment
