TinyPoliticaLlama-1.1B is a SFT fine-tune of TinyLlama/TinyLlama-1.1B-Chat-v1.0 and a sister model of h4rz3rk4s3/TinyParlaMintLlama-1.1B and h4rz3rk4s3/TinyNewsLlama-1.1B. This model was fine-tuned for ~24h on one A100 40GB on ~225M tokens on the training corpora of both her sister models.

The goal of this project is to study the potential for improving the domain-specific (in this case political) knowledge of small (<3B) LLMs by concentrating the training datasets TF-IDF in respect to the underlying Topics found in the origianl Dataset.

The used training data contains speeches from the Austrian, Danish, French, British, Hungarian, Dutch, Norwegian, Polish, Swedish and Turkish Parliament, as well as political news articles from The New York Times, USA Today and The Washington Times.

💻 Usage

python

!pip install -qU transformers accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM
from accelerate import Accelerator
import transformers
import torch
model = "h4rz3rk4s3/TinyPoliticaLlama-1.1B"
messages = [
    {
        "role": "system",
        "content": "You are an experienced journalist in the political domain and an expert of European politics.",
    },
    {"role": "user", "content": "Write a short article explaining how the French yellow vest protests started, how they developed over time and how the French Government reacted to the protests."},
]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
model = AutoModelForCausalLM.from_pretrained(
    model, trust_remote_code=True, device_map={"": Accelerator().process_index}
)

pipeline = transformers.pipeline(
    "text-generation",
    tokenizer=tokenizer,
    model=model,
    torch_dtype=torch.float16,
    device_map={"": Accelerator().process_index},
)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])