You can deploy this model and start earning money today!

qrk-labs/
dabe-stage-b-gpt2-0_5b-instruct-sft

Name: dabe-stage-b-gpt2-0_5b-instruct-sft
Availability: InStock

qrk-labs/dabe-stage-b-gpt2-0_5b-stage-a

finetune

cc-by-sa-3.0

Views

No views yet

DABE Stage B GPT-2 0.5B Instruct SFT

This is an experimental instruction-tuned checkpoint derived from the DABE Stage B GPT-2-tokenized 0.5B base run. It is published as a validation artifact for the DABE training pipeline, not as a production assistant.

Training

Base checkpoint: frozen-checkpoint-step-0005000 from qrk-labs/dabe-stage-b-gpt2-0_5b-stage-a.
Parameters: 505,977,856.
Tokenizer: GPT-2 tokenizer. No new special tokens were added.
Format: answer-only loss masking over GPT-2 token IDs.
SFT steps: 1000.
Batch size: 64.
Final SFT validation loss: 2.9926.
Final SFT validation perplexity: 19.94.

Training examples by source:

json

{
  "arc_easy": 2251,
  "commonsenseqa": 9741,
  "dolly": 15011,
  "oasst1": 34795,
  "openbookqa": 4957,
  "tinystories_lm_mix": 3000,
  "winogrande": 40398
}

Prompt format used for tuning:

text

<|endoftext|>Instruction:
{instruction}

Input:
{optional input}

Answer:
{response}<|endoftext|>

Text Completion Benchmarks

Task	Split	Metric	Value	Examples
hellaswag	validation	accuracy_norm	0.2584	10042
openbookqa	validation	accuracy_norm	0.2480	500
winogrande	validation	accuracy_norm	0.5012	1267
commonsenseqa	validation	accuracy_norm	0.2621	1221
arc_easy	validation	accuracy_norm	0.3351	570
lambada	test	next_word_exact_match	0.0968	5153

Known Limitations

This model is still very weak as an assistant. Direct instruction probes show that it can sometimes follow simple answer style and terminate, but it fails many basic tasks such as arithmetic, translation, structured JSON/tool-call output, and robust summarization. Treat it as a research checkpoint showing that the GPT-2-tokenized 0.5B base can be SFT-trained, not as a deployable chat model.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "qrk-labs/dabe-stage-b-gpt2-0_5b-instruct-sft"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

prompt = "<|endoftext|>Instruction:\nWrite a short thank-you note to a teacher.\n\nAnswer:\n"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=80, do_sample=False, pad_token_id=tok.eos_token_id)
print(tok.decode(out[0]))