Trained on raw text with no chat template, so it is a style continuation model, not an assistant. Prompt it with a fragment and it continues in horoscope register.

Results

epoch	eval_loss	perplexity
1	2.3419	10.40
2	2.0541	7.80
3	2.0292	7.61

Perplexity fell 27% from epoch 1 to epoch 3. Eval loss decreased monotonically - no overfitting observed.

Before / after

Same prompt, same seed (42), same sampling (temperature=0.8, top_p=0.9); only the weights differ.

Prompt: Leo, the stars suggest

Base: "...that the galaxy is not just a flat plane, but a sphere, and that the universe is expanding..."
Fine-tuned: "...that you are going to find someone who is very attractive. They may be a friend or a romantic partner..."

The base model read "stars" astronomically; the fine-tuned model reads it astrologically.

Hyperparameters


epochs	3
effective batch size	16 (per_device 16 x grad_accum 1)
learning rate	2e-5, linear decay, no warmup
optimizer	adamw_torch_fused
precision	bf16
max_length	512 (longest example was 197 tokens; truncation never triggered)
optimizer steps	3708
hardware	1x RTX 4000 Ada (20 GB)
wall time	22.4 min

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("VizRohan/qwen3-0.6b-horoscope")
model = AutoModelForCausalLM.from_pretrained("VizRohan/qwen3-0.6b-horoscope", dtype="auto")

enc = tok("Aries, today", return_tensors="pt")
out = model.generate(**enc, max_new_tokens=60, do_sample=True,
                     temperature=0.8, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

Entertainment only. Trained on ~20k short texts, so output is formulaic and reproduces the genre's cliches. Low perplexity partly reflects that horoscope text is deliberately vague and therefore easy to predict - it is not a measure of quality.