You can deploy this model and start earning money today!

merterbak/
Seed-0.5B

Name: Seed-0.5B
Availability: InStock

HuggingFaceFW/fineweb-edu

apache-2.0

Views

No views yet

Seed-0.5B

0.5B decoder only dense model trained from scratch with 6B tokens.
Model and tokenizer is trained on fineweb-edu sample-10BT for achieving good results with low token size.
As it isn't instruction finetuned, the model performs document completion, not conversational generation.
While 6B tokens are enough to learn reasonable grammar and structure, this amount is insufficient for strong factual grounding, and model can generate inaccurate or imprecise informations.
Model is released primarily for educational, research, and experimental purposes.

GitHub: merterbak/llm-from-scratch

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("merterbak/Seed-0.5B")
model = AutoModelForCausalLM.from_pretrained(
    "merterbak/Seed-0.5B",
    trust_remote_code=True,
    dtype="auto"
)
prompt = "Climate change can affect"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    temperature=0.3,
    top_k=40,
    do_sample=True,
    repetition_penalty=1.2,
    no_repeat_ngram_size=3,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))