Supra-50M Instruct is a compact 50M-parameter chat/instruct causal language model built by SupraLabs, trained from scratch using a Llama-style architecture on 20 billion tokens of high-quality educational web text. Despite being significantly smaller than comparable open models, it achieves competitive or superior results on several key benchmarks. It's our first SupraLabs Scaling Up Plan model.

📚 Training Data

For the SFT (supervised finetuning) we used the full Alpaca-Cleaned dataset for 4 epochs. See the full SFT-code in sft.py

Benchmarks

Task	Metric	Value
arc_easy	acc,none	0.4659
arc_easy	acc_stderr,none	0.0102
arc_easy	acc_norm,none	0.4423
arc_easy	acc_norm_stderr,none	0.0102
arc_challenge	acc,none	0.2287
arc_challenge	acc_stderr,none	0.0123
arc_challenge	acc_norm,none	0.2756
arc_challenge	acc_norm_stderr,none	0.0131
hellaswag	acc,none	0.2794
hellaswag	acc_stderr,none	0.0045
hellaswag	acc_norm,none	0.2922
hellaswag	acc_norm_stderr,none	0.0045
winogrande	acc,none	0.5154
winogrande	acc_stderr,none	0.0140
piqa	acc,none	0.5558
piqa	acc_stderr,none	0.0114
piqa	acc_norm,none	0.5952
piqa	acc_norm_stderr,none	0.0115
openbookqa	acc,none	0.1580
openbookqa	acc_stderr,none	0.0163
openbookqa	acc_norm,none	0.2860
openbookqa	acc_norm_stderr,none	0.0202
boolq	acc,none	0.4205
boolq	acc_stderr,none	0.0086

⚙️ Pretraining details

For more details, the full code, configs and weights, please refer to https://huggingface.co/SupraLabs/Supra-50M-Base

🚀 Inference

python

import os
import warnings
import time

os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
warnings.filterwarnings("ignore", category=UserWarning, module="transformers")

import torch
from transformers import pipeline, AutoTokenizer, logging

logging.set_verbosity_error()

# ── Global variables ──────────────────────────────────────────────────────────

end = time.time()
start = time.time()
tokens = []


# ── Config ────────────────────────────────────────────────────────────────────

MODEL_ID = "SupraLabs/Supra-50M-Instruct"  
MAX_NEW_TOKENS = 512

# ── Load pipeline directly from HF ────────────────────────────────────────────

print(f"[*] Loading SFT model and tokenizer from HF Hub ({MODEL_ID})...")

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, clean_up_tokenization_spaces=False)

pipe = pipeline(
    "text-generation",
    model=MODEL_ID,
    tokenizer=tokenizer,
    device_map="auto",
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32
)

print(f"[+] Pipeline ready — Model loaded using {pipe.model.device}")

# ── Prompt template (must match sft.py exactly) ───────────────────────────────

def build_prompt(instruction: str, input_text: str = "") -> str:
    if input_text.strip():
        return (
            "Below is an instruction that describes a task, paired with an input "
            "that provides further context. Write a response that appropriately "
            "completes the request.\n\n"
            f"### Instruction:\n{instruction}\n\n"
            f"### Input:\n{input_text}\n\n"
            "### Response:\n"
        )
    return (
        "Below is an instruction that describes a task. Write a response that "
        "appropriately completes the request.\n\n"
        f"### Instruction:\n{instruction}\n\n"
        "### Response:\n"
    )

# ── Generate ──────────────────────────────────────────────────────────────────

def generate(instruction: str, input_text: str = "", max_new_tokens: int = MAX_NEW_TOKENS) -> str:
    prompt = build_prompt(instruction, input_text)
    start = time.time()
    result = pipe(
        prompt,
        max_new_tokens=max_new_tokens,
        do_sample=True,
        temperature=0.7,
        top_k=50,
        top_p=0.9,
        repetition_penalty=1.15,
        pad_token_id=pipe.tokenizer.pad_token_id,
        eos_token_id=pipe.tokenizer.eos_token_id,
        return_full_text=False,
        generation_config=None
    )
    end = time.time()
    
    generated_text = result[0]["generated_text"]
    tokens = pipe.tokenizer(generated_text)["input_ids"]
    
    return generated_text, tokens, end, start

# ── Interactive loop ──────────────────────────────────────────────────────────

if __name__ == "__main__":
    print("\n[+] Ready. Type 'quit' to exit.\n")

    while True:
        instruction = input("Instruction: ").strip()
        if instruction.lower() == "quit":
            break

        inp = input("Input (optional, Enter to skip): ").strip()

        print("-" * 50)
        text, tokens, end, start = generate(instruction, inp)
        print(text)
        print()
        
        print(f"Generated tokens: {len(tokens)}")
        print(f"Time: {end - start:.2f}s")
        print(f"Speed: {len(tokens) / (end - start):.2f} tokens/sec")

💬 Sample Outputs

These outputs were generated with these samplings: temperature=0.7, top_k=50, top_p=0.9, repetition_penalty=1.15

(The model still hallucinates, but can respond a big quantity of questions correctly)

User: What is AI?
Supra-50M Instruct::
AI refers to the simulation of human intelligence and creativity in machines, which are designed to mimic or surpass human cognition and reasoning. It is the process by which AI systems can solve problems, make predictions, or automate processes that typically require human-level thinking and problem solving. AI technology has been rapidly growing and evolving over time, with many potential benefits for society as a whole. Some of these include:

Improved Accuracy: AI algorithms have demonstrated great accuracy in predicting outcomes, making accurate predictions on new data sets, and even answering real-world questions. This improves our ability to learn from experience and adapt to changing situations.
Personalization: AI is used to personalize user experiences, such as recommending products based on their previous interactions and preferences. This helps people to stay motivated and maintain high levels of satisfaction.
Better Decision Making: AI enables systems to analyze large amounts of data quickly and accurately, allowing us to make informed decisions more easily and

User: What is the capital of Japan?
Supra-50M Instruct::
The capital of Japan is Tokyo.

User: Hi!
Supra-50M Instruct::
Hello! How can i help you today?

User: What is the capital of United States?
Supra-50M Instruct::
The capital of United States is New York City.

User: What is the capital of Russia?
Supra-50M Instruct::
The capital of Russia is Moscow.

📄 License

This model is released under the Apache 2.0 License.