Model Card for CatGPT
CatGPT is a Catalan natural language model inspired by GPT-2. It is designed to generate coherent and contextually relevant text in Catalan. The model is intended primarily for educational and experimental purposes, providing a lightweight tool for exploring natural language processing in Catalan.
Model Details
Model Description
CatGPT follows the architecture of GPT-2 but is trained from scratch with a specific focus on the Catalan language. The model's smaller size makes it accessible and easy to deploy, though it does not aim for high-performance text generation. Its design choices ensure it can be used efficiently for training and inference within the Catalan language context.
- Developed by: Roger Baiges
- Model type: Causal Language Model (GPT-2 based)
- Language(s): Catalan
- License: MIT
- Finetuned from model: Trained from scratch
Model Sources
Uses
Direct Use
CatGPT can be used as a text generator in Catalan. It's suitable for creating educational content, generating sample text, or experimenting with language modeling in Catalan.
Downstream Use
The model can be fine-tuned for specific tasks like text completion, dialogue systems, or creative writing in Catalan.
Out-of-Scope Use
This model is not suitable for tasks requiring high accuracy or dealing with complex language understanding, such as legal or medical text generation. It is also not recommended for use in generating content that requires a deep understanding of context or nuance.
Bias, Risks, and Limitations
Biases
As with most language models, CatGPT may reflect biases present in the training data. Given the training datasets are primarily web-scraped data, the model might inadvertently generate biased or inappropriate content.
Limitations
- The model's small size limits its ability to generate high-quality text.
- It may not perform well in generating text that requires nuanced understanding or in contexts outside the training data.
- The model may struggle with certain dialects or less common expressions in Catalan.
Recommendations
Users should monitor outputs for bias and inappropriate content. Fine-tuning with carefully curated data can help mitigate some biases.
How to Get Started with the Model
To use CatGPT, you can load the model and tokenizer as follows:
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("baiges/CatGPT")
4model = AutoModelForCausalLM.from_pretrained("baiges/CatGPT")
5
6input_text = "La intel·ligència artificial"
7inputs = tokenizer(input_text, return_tensors="pt")
8outputs = model.generate(inputs.input_ids, max_length=100, num_return_sequences=1)
9
10generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
11print(generated_text)
