This is a full-parameter supervised fine-tune of EleutherAI/pythia-410m at pretraining revision step50000. It was trained for one epoch on the usable portion of yahma/alpaca-cleaned to study how instruction tuning changes an intermediate Pythia checkpoint.

Result: the experiment's primary objective was achieved. Alpaca fine-tuning successfully taught the intermediate checkpoint to behave like a recognizable instruction-following chatbot. Where the untouched model often echoed the question, repeated phrases, produced mismatched code, or fell into a loop, the fine-tuned model consistently produced direct, answer-shaped assistant responses.

This is a meaningful behavioral success even though factual correctness remains limited by the small 410M model, its intermediate pretraining checkpoint, and the amount and quality of supervised data. The supplied evaluation still contains a wrong arithmetic result and a broken prime-number function, so the model should not be treated as a dependable source of facts or code. The achievement is narrower and clearly demonstrated: the model learned the instruction-response behavior taught by Alpaca instead of continuing with the base checkpoint's repetitive or incoherent raw-text behavior.

Experiment objective and outcome

The objective was to test whether one epoch of full-parameter Alpaca fine-tuning could turn Pythia-410M step50000 into a recognizable chatbot-style instruction follower. Success was defined behaviorally:

respond to an instruction instead of merely echoing or extending it;
produce a coherent, answer-shaped response in the requested domain or format;
avoid the base checkpoint's obvious repetition loops; and
improve loss on a fixed held-out Alpaca validation set.

All four success criteria were met. Across the three supplied comparisons, the base model loops, repeats the question, or emits mismatched Go-like code. The fine-tuned model instead gives a two-sentence explanation, a structured calculation, and a Python-shaped function. Validation loss also declines at every reported evaluation point. Alpaca therefore succeeded at teaching the model the behavioral form of a chatbot, even though it did not supply enough model capacity, pretraining, or supervised signal to make every answer correct.

The underlying architecture is still an autoregressive causal language model trained through token prediction. Here, “learned to behave like a chatbot” means that fine-tuning changed the conditional behavior of that token predictor: given the Alpaca prompt format, it now generates relevant assistant-style responses rather than the base checkpoint's gibberish-like continuations and loops.

Model summary

Property	Value
Model repository	shehryars715/pythia-410m-step50000-alpaca
Base model	EleutherAI/pythia-410m
Base revision	step50000
Architecture	GPT-NeoX causal language model
Parameters	405,334,016 total; 302,311,424 non-embedding
Language	English
Fine-tuning method	Full-parameter supervised fine-tuning
Adapter or quantization	None
Training data	yahma/alpaca-cleaned
Upstream rows	51,760
Validation split	256 fixed shuffled examples
Training split	Remaining usable examples after empty-output filtering
Epochs	1
Training objective	Completion-only causal-language-model loss
Fine-tuning sequence limit	384 tokens
Framework	Transformers and TRL

This repository contains complete model weights and a tokenizer. It is not a LoRA or PEFT adapter and does not require the base checkpoint at inference time.

Intended uses

Suitable uses include:

demonstrating a successful behavioral conversion from raw continuation to chatbot-style instruction following;
studying instruction tuning at an intermediate language-model checkpoint;
reproducing or extending small-model supervised fine-tuning experiments;
qualitative comparisons with the untouched Pythia step50000 checkpoint;
educational demonstrations of completion-only loss and full fine-tuning.

Out-of-scope uses

Do not use this checkpoint as:

a source of factual, mathematical, medical, legal, financial, or safety-critical advice;
an autonomous agent or production chatbot;
a secure code generator;
a replacement for human review;
a multilingual model;
a system expected to refuse unsafe requests.

No dedicated safety tuning, red-teaming, RLHF, DPO, tool-use training, or production evaluation was performed.

Training data

The upstream cleaned Alpaca dataset reports 51,760 English instruction examples. Before splitting, rows with blank outputs were removed. The usable rows were shuffled with seed 42; the first 256 became a fixed validation set, and all remaining rows were used for training. No 10,000-example cap was applied.

Each example was converted to one of the original Alpaca-style templates.

Without additional input:

text

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:
{completion}

With additional input:

text

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input}

### Response:
{completion}

An EOS token was appended to every completion. Prompt tokens supplied context but were masked from the loss; loss was calculated only on completion and EOS tokens.

Training configuration

Hyperparameter	Value
Random seed	42
Epochs	1
Per-device training batch size	2
Per-device evaluation batch size	2
Gradient accumulation	8
Effective batch per GPU	16
Learning rate	5e-5
Scheduler	Cosine
Warmup ratio	0.03
Weight decay	0.01
Optimizer	AdamW (PyTorch)
Maximum gradient norm	1.0
Maximum sequence length	384
Packing	Disabled
Gradient checkpointing	Enabled
Precision	BF16 on supported GPUs, otherwise FP16
Evaluation interval	250 optimizer steps
Checkpoint interval	250 optimizer steps
Checkpoints retained	2
Trainable parameters	Approximately 100%

The checkpoint was loaded in the selected half precision for baseline evaluation. Immediately before optimization, trainable weights were promoted to FP32 master weights while mixed precision was retained for forward and backward computation.

Training results

The dashed red line in the supplied plot is the untouched step50000 model's fixed-validation loss. The exact numeric baseline was not supplied with the logged table, so it is not invented here. At every reported checkpoint, the fine-tuned model is below that baseline.

Step	Training loss	Validation loss	Entropy	Tokens processed	Mean token accuracy
250	1.722400	1.746238	2.189269	1,540,590	0.581343
500	1.672100	1.692939	2.166639	3,066,300	0.594109
750	1.623800	1.643273	2.199533	4,595,545	0.599544
1,000	1.604500	1.607838	2.160674	6,130,144	0.607807
1,250	1.610100	1.580659	2.137191	7,669,953	0.611636
1,500	1.560100	1.571541	2.133488	9,196,660	0.612945

Across the reported evaluations, validation loss decreased from 1.746238 at step 250 to 1.571541 at step 1500. Mean token accuracy rose from 0.581343 to 0.612945. These metrics show improved fit to the held-out Alpaca distribution; they do not establish factual correctness, reasoning ability, or safety.

Qualitative comparison

The comparison used deterministic decoding with a maximum of 128 new tokens and repetition penalty 1.05. The untouched base model received each question as raw text. The fine-tuned model received the same question inside the full Alpaca prompt template. This tests each model through its intended interface, but it is not a template-controlled benchmark.

Only the question, base response, and fine-tuned response from the supplied CSV are reproduced below.

Explain photosynthesis in exactly two sentences.

Base model response

text

The first sentence is a description of the photosynthesis process. The second sentence is a description of the photosynthesis process.

The first sentence is a description of the photosynthesis process. The second sentence is a description of the photosynthesis process.

The first sentence is a description of the photosynthesis process. The second sentence is a description of the photosynthesis process.

The first sentence is a description of the photosynthesis process. The second sentence is a description of the photosynthesis process.

The first sentence is a description of the photosynthesis process. The second sentence is a description of the photosynthesis

Fine-tuned model response

Photosynthesis is the process by which plants, algae, and some bacteria convert sunlight, water, and carbon dioxide into glucose, oxygen, and other organic compounds. It is a vital process for life on Earth, as it provides the primary source of energy for all living organisms.

Assessment: The fine-tuned response is coherent, answer-shaped, and follows the two-sentence constraint. It is much more useful than the looping base response.

Calculate the average speed for 180 km in 3 hours.

Base model response

text

The average speed for 180 km in 3 hours.

The average speed for 180 km in 3 hours.

The average speed for 180 km in 3 hours.

The average speed for 180 km in 3 hours.

The average speed for 180 km in 3 hours.

The average speed for 180 km in 3 hours.

The average speed for 180 km in 3 hours.

The average speed for 180 km in 3 hours.

The average speed for 180 km in 3 hours.

The average speed for 180 km in 3 hours.

The average speed for 180 km

Fine-tuned model response

text

The average speed of a car on a road is calculated by multiplying the distance traveled by the time taken to cover that distance. In this case, the average speed is calculated as follows:

Average speed = Distance / Time
Average speed = 180 km / 3 hours
Average speed = 0.5 miles per hour (mph)

Assessment: The fine-tuned response adopts a useful answer structure and states the correct formula, but its final result is wrong. The correct answer is 60 km/h. This is a clear factual-reasoning failure.

Write a Python function that checks whether an integer is prime.

Base model response

text

func IsPrime(n int) bool {
    return n % 10 == 0
}

// IsPrime returns true if the given integer is prime.
func IsPrime(n int) bool {
    return n % 10 == 0
}

// IsPrime returns true if the given integer is prime.
func IsPrime(n int) bool {
    return n % 10 == 0
}

// IsPrime returns true if the given integer is prime.
func IsPrime(n int) bool {
    return n % 10 == 0
}

// IsPrime returns

Fine-tuned model response

text

Here's a simple Python function that checks whether an integer is prime or not:

def is_prime(num):
    if num < 2:
        return False
    else:
        return num % num == 0

This function takes in an integer num as an argument and returns True if num is divisible by 2, False otherwise. If num is divisible by 1, True will be returned, otherwise False.

Assessment: The fine-tuned response switches to the requested language and looks like a Python answer, but its logic is invalid: for every nonzero number, num modulo num is zero. The explanation also contradicts the code. Do not trust generated code without testing and review.

What the examples establish

The primary chatbot-behavior objective was achieved. The untouched step50000 model does not give a usable direct answer to any of the three questions: it repeats itself, echoes the input, or produces looping code in the wrong language. In contrast, every fine-tuned output is recognizably an assistant response aimed at completing the instruction. It explains, shows a calculation, or supplies Python-shaped code instead of falling into the base model's continuation loops.

This demonstrates successful learning of instruction-response structure, prompt adherence, answer formatting, and chatbot-like conversational behavior. The change is particularly clear because the same small architecture produces qualitatively different behavior after Alpaca fine-tuning.

Behavioral success and factual success are separate. Of the three examples, one is broadly correct while two contain decisive correctness failures. The model has learned how to respond like an assistant, but it has not reliably learned how to produce a correct answer. Three hand-selected prompts are also far too few to estimate general model quality.

No standard benchmark, safety suite, bias evaluation, contamination analysis, or statistically powered human evaluation has been reported.

Usage

The model is a plain causal language model and does not define a chat template. Format requests with the Alpaca prompt used during fine-tuning.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "shehryars715/pythia-410m-step50000-alpaca"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype="auto",
    device_map="auto",
)

instruction = "Explain why leaves appear green in simple language."
prompt = (
    "Below is an instruction that describes a task. Write a response that "
    "appropriately completes the request.\n\n"
    f"### Instruction:\n{instruction}\n\n"
    "### Response:\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
prompt_length = inputs["input_ids"].shape[1]

with torch.inference_mode():
    generated = model.generate(
        **inputs,
        do_sample=False,
        max_new_tokens=128,
        repetition_penalty=1.05,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

answer = tokenizer.decode(
    generated[0, prompt_length:],
    skip_special_tokens=True,
).strip()
print(answer)

For requests with extra context, use the template shown in the training-data section and add an Input block before Response.

Limitations and risks

Intermediate base checkpoint: step50000 is far earlier than Pythia's step143000 final checkpoint. Instruction tuning cannot replace the language pretraining that had not yet occurred.
Small model: 410M-scale models have limited knowledge, reasoning, long-context behavior, and instruction reliability.
Demonstrated correctness failures: the supplied arithmetic and code generations are wrong despite looking confidently formatted.
English only: neither the base model nor this fine-tune is intended for reliable multilingual use.
Sequence truncation: supervised fine-tuning used a 384-token maximum sequence length, which may weaken behavior on longer prompts and responses.
Synthetic training data: Alpaca responses were generated by another model and can contain errors, biases, stereotypes, or unsafe content.
Inherited base-model risks: Pythia was pretrained on the Pile and may reproduce offensive, private, copyrighted, or otherwise undesirable text.
No safety alignment: the model was not trained to refuse harmful requests.
No production validation: latency, robustness, calibration, security, privacy, and adversarial behavior were not evaluated.

Always review outputs before use. Execute generated code only in an isolated environment after inspection and testing.

Licensing and data-use notice

The Pythia base model is published under Apache-2.0. The Hugging Face metadata for yahma/alpaca-cleaned has displayed CC BY 4.0, while text in its dataset card and the original Stanford Alpaca repository state CC BY-NC 4.0 and describe the data as research/non-commercial. Because those upstream signals conflict, this card uses the Hugging Face metadata value other instead of asserting a single license for the fine-tuned weights.

Review the current base-model terms, dataset terms, and Stanford Alpaca usage notice before using or redistributing the model. A conservative interpretation is to limit use to research and non-commercial purposes unless the applicable rights have been independently clarified.

Reproducibility notes

Dataset shuffle seed: 42
Fixed validation set: first 256 usable rows after seeded shuffle
Comparison decoding: greedy, 128 new tokens, repetition penalty 1.05
Baseline comparison: EleutherAI/pythia-410m at revision step50000
Fine-tuning implementation: Transformers 4.57.1 and TRL 0.26.2
Reported metrics and generations: supplied directly from the completed run

Citation

If this checkpoint is useful, cite the Pythia and Alpaca projects:

bibtex

@article{biderman2023pythia,
  title={Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling},
  author={Biderman, Stella and Schoelkopf, Hailey and Anthony, Quentin and others},
  journal={Proceedings of the 40th International Conference on Machine Learning},
  year={2023}
}

@misc{alpaca,
  author={Taori, Rohan and Gulrajani, Ishaan and Zhang, Tianyi and Dubois, Yann and Li, Xuechen and Guestrin, Carlos and Liang, Percy and Hashimoto, Tatsunori B.},
  title={Stanford Alpaca: An Instruction-following LLaMA Model},
  year={2023},
  howpublished={https://github.com/tatsu-lab/stanford_alpaca}
}

Acknowledgements

Thanks to EleutherAI for the Pythia checkpoint suite, the Stanford Alpaca authors for the original instruction dataset and methodology, and the maintainers of yahma/alpaca-cleaned.