This document describes cucafera (Base Model), a tiny Catalan Large Language Model (LLM) developed for experimental and educational purposes. The model has been pre-trained exclusively on Catalan text using the pauhidalgoo/patufet-pretrain dataset, making it a specialized tool for a variety of Catalan NLP applications.

Model Details

Model Description

cucafera (Base Model) is a 244-million parameter transformer-based language model inspired by the LLAMA architecture (notably LLAMA3). Despite its relatively small size compared to many contemporary models, it is optimized for generating coherent and contextually relevant text in Catalan.

Model Size: 244M parameters
Architecture: Transformer-based (LLAMA-inspired) with 30 layers
Embedding Size: 768
Attention Mechanism: 4 key/value heads and 8 query heads (using Grouped Query Attention - GQA)
Context Length: 2048 tokens
Tokenizer: Byte-Pair Encoding (BPE) with a vocabulary size of 65,536
Activation Function: GeGLU

Training Data

The base model was pre-trained using the patufet-pretrain dataset. This dataset consists of high-quality Catalan text curated specifically for pre-training language models, ensuring rich linguistic and cultural representation and high-quality text sources.

Training Procedure

The model was trained using the LLAMA3-like architecture with the following setup:

Total training steps: 11,007
Batch size: 524288
Sequence Length: 2048
Learning rate: 3e-4 with 200 warmup steps and cosine decay up to 3e-5
Optimizer: AdamW
Hardware used: Single A100 80GB GPU from vast.ai
Training duration: 1.5 days
Total tokens processed: 5.5 billion
Hardware and Efficiency:
Training was executed on a single A100 80GB GPU, showcasing that effective LLM training can be achieved with accessible resources while maintaining cost efficiency.

Direct Use

The base model is suitable for general text generation tasks in Catalan, including but not limited to:

Creative writing
Summarization

Note that this is the base model, so it is trained on a text completion task.

However, due to its limited size, it is not able to provide correct factual information and you must be aware of this fact when using this model.

Downstream Applications

Users may fine-tune cucafera (Base Model) for domain-specific tasks such as:

General purpose conversational agents
Catalan language question answering
Chatbots and interactive applications tailored for Catalan speakers

Out-of-Scope Uses

High-Stakes Applications:
The model is not recommended for uses where extremely high factual accuracy is required or where outputs could have significant real-world consequences.
Non-Catalan Tasks:
Since the model is exclusively trained on Catalan text, it is not suited for tasks in other languages without further training or fine-tuning.
It does not have any fine-tuning for safety, so it may generate incorrect or biased outputs.
It should not be used for generating misinformation or offensive content.

Bias, Risks, and Limitations

The model has no instruction tuning, so it may not follow prompts effectively.
It only understands Catalan, meaning it is unsuitable for multilingual applications.
Due to its small size (244M parameters), its knowledge and reasoning capabilities are limited.
It was trained on a limited dataset, which may introduce biases in its outputs.

Recommendations

The goal of this model is educational. You are encouraged to train your own model.
If used in production, human review of its outputs is recommended.
Fine-tuning on task-specific data can improve accuracy and mitigate biases.
Users should be cautious when using it in sensitive or high-stakes applications.

How to Get Started

Below is an example code snippet to load and utilize cucafera (Base Model) using Hugging Face's Transformers library:

python

from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "pauhidalgoo/cucafera"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

input_text = "Història sobre un drac i un cavaller. Hi havia una vegada"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Acknowledgements

This model was developed as an experimental project, inspired by Karpathy's NanoGPT Series. My colleague Roger Baiges also trained his own CatGPT.

For more details, updates, or to contribute to the project, please visit the GitHub repository