Hello Agent 500M — M1 Research Pretrain 5B Clean
This repository contains the Hugging Face export of the 500M-parameter
Hello Agent causal language model trained during the How Agentic M1 research
stage.
Training status
This export is the completed pretraining checkpoint trained on the cleaned
5B-token-class M1 corpus. The local package also contains SFT data and an SFT
configuration, but no completed SFT checkpoint was present alongside this
export. This model is therefore intentionally described as pretrained rather
than instruction-tuned.
Architecture
- Decoder-only causal language model
- 24 layers
- Hidden size 1,024
- 16 attention heads and 4 key/value heads
- Vocabulary size 32,000
- Maximum position length 8,192
- RoPE and QK normalization
- Tied input/output embeddings
The repository includes custom configuration and modeling code. Load it only
after reviewing that code.
Usage
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_id = "jjyaoao/hello-agent-500m-m1-pretrain-5b-clean"
4tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
5model = AutoModelForCausalLM.from_pretrained(
6 model_id,
7 trust_remote_code=True,
8 device_map="auto",
9)
10
11inputs = tokenizer("Research question:", return_tensors="pt").to(model.device)
12outputs = model.generate(**inputs, max_new_tokens=128)
13print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Use a recent Transformers 5.x release; the tokenizer export uses the
TokenizersBackend introduced in that generation.
Training summary
The final pretraining checkpoint was saved at step 100,459. Its recorded final
training loss was approximately 3.126. These training metrics are descriptive
and are not a benchmark of downstream capability.
An independent local evaluation on 192 packed validation batches (2,855,903
supervised tokens at sequence length 2,048) reproduced a token-weighted loss of
3.00968 and perplexity of 20.2808. All 387 exported tensors were checked
against the native training checkpoint and were exactly equal, with no missing
or extra tensors and a maximum absolute difference of 0.
A one-sample SFT readiness check completed a BF16 forward/backward pass with
finite loss and finite gradients. No optimizer step was performed. This
confirms that the checkpoint is usable as initialization for post-training,
but does not establish instruction-following capability.
Limitations
This is a small research model and has not been established as safe or reliable
for production use. It may hallucinate, reproduce biases or unsafe material,
and follow instructions poorly because this uploaded checkpoint is not
instruction-tuned. Do not use it for high-stakes decisions without extensive
independent evaluation and safeguards.
License
No license is asserted in this model card. Users must review the provenance and
licenses of the training data and project code before use or redistribution.
