Model Card for SmolLM-135M-de
A german version of
HuggingFaceTB/SmolLM-135M, trained to speak German by applying CPT for about 6 billion tokens.
If you are looking for a chat model, try
this fine tune or the
corresponding adapter model.
Model Details
Model Description
The base model is
HuggingFaceTB/SmolLM-135M, which I further trained on about 6 billion German-language tokens.
- Model type: Large Language Model (Llama architecture)
- Language(s) (NLP): German
- License: Apache 2.0
- Finetuned from model: HuggingFaceTB/SmolLM-135M
Uses
I mainly made this as a small experimentation model to quickly benchmark datasets etc. - since the model is so small, I am unsure about its usefulness for any real-world scenarios.
This is a base model without any chat fine tuning etc. and thus should not be used as-is. It outputs mostly correct German, which is what I tried to achieve.
If you are looking for a chat model, try
this adapter.
Bias, Risks, and Limitations
This is a very small model and will output blatantly wrong information. I have not done any further filtering on the source datasets, so it is possible that the model will generate lewd or otherwise inappropriate content. Use with care.
I would strongly recommend against using this model in a production setting, at least without further fine tuning and preference optimization.
How to Get Started with the Model
Use the code below to get started with the model.
1# adapted from the original SmolLM repo
2# pip install transformers
3from transformers import AutoModelForCausalLM, AutoTokenizer
4checkpoint = "LemiSt/SmolLM-135M-de"
5device = "cuda" # for GPU usage or "cpu" for CPU usage
6tokenizer = AutoTokenizer.from_pretrained(checkpoint)
7# for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
8model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
9inputs = tokenizer.encode("Rezept für einen leckeren veganen Schokokuchen:\n", return_tensors="pt").to(device)
10outputs = model.generate(inputs, max_new_tokens=256)
11print(tokenizer.decode(outputs[0]))
Training Details
Training Data
Training Procedure
This was trained with axolotl, using full fine tuning (no LoRA etc). I used a sequence length of 2048 with an effective batch size of 512, learning rate of 0.003 with the adamw_bnb_8bit optimizer and a cosine scheduler.
Due to an error I made in calculating the token count, I accidentally trained for nearly 2 epochs, with the learning rate not reaching its proper minimum.
