Training Framework: Megatron-LM
Runtime: .venv-apex
Cluster: CINECA Leonardo. We acknowledge the CINECA award under the ISCRA initiative, for the availability of high-performance computing resources and support.
Topology: 3 nodes / 12 GPUs
Sequence length: 2048
Micro-batch size: 3
Global batch size: 72
Tokens per step: 147.456
Target steps: 1.104.964
Checkpoint interval: 50.000

Final successful completion occurred on Sunday, July 26, 2026.

Evaluation

End-of-training validation:

iteration: 1.104.964
validation loss: 2.910531E+00
validation perplexity: 1.836655E+01

This is the validated Megatron end-of-training metric for the completed full-pass run.

Release validation and benchmark highlights:

Hugging Face controlled-generation summary:
- empty_generation_rate: 0.0
- avg_repeated_bigram_fraction: 0.1148
- avg_repeated_trigram_fraction: 0.0809
- avg_topic_keyword_overlap: 0.23
stronger benchmark groups (BLiMP-IT):
- verbal_class_and_argument_structure: 0.9500
- pronouns: 0.7188
- agreement_and_inflection: 0.7454 with appended EOS

Usage

Transformers example:

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Gpeik/Sophira-360M-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Il governo ha annunciato che"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Pipeline example:

python

from transformers import pipeline
import torch

generator = pipeline(
    "text-generation",
    model="Gpeik/Sophira-360M-base",
    tokenizer="Gpeik/Sophira-360M-base",
    model_kwargs={"dtype": torch.bfloat16},
    device_map="auto",
)

result = generator(
    "Milano, 19 luglio - La Borsa europea ha chiuso",
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
)

print(result[0]["generated_text"])

Evaluation Snapshot

Completed:

end-of-training Megatron validation
checkpoint save/resume validation across multiple Slurm jobs
quota-recovery and checkpoint-pruning procedure
Hugging Face export
native vs Hugging Face first-token parity validation
corrected Hugging Face controlled-generation rerun
corrected controlled-generation analysis rerun
BLiMP-IT benchmark
BLiMP-IT appended-EOS benchmark

Areas for Improvement

This is a base model, not an instruction-tuned model.
Controlled generation still shows visible topic drift and clause repetition on some prompts.
BLiMP-IT performance is only modestly above chance overall, especially on non-local dependency phenomena.
Broader robustness, bias, and safety evaluation is still pending.
This release should be treated as an open Italian base-model baseline, not as a polished assistant model.

Release Artifacts

The canonical release candidate checkpoint is:

iter_1104964

The published Hugging Face repository is intended to contain:

the frozen Hugging Face model weights
the tokenizer files needed for same-repo loading
this model card as the public README.md
the release image asset used for repository presentation

License

Code: Apache-2.0
Tokenizer: Apache-2.0
Model weights target: Apache-2.0

Source dataset licenses remain governed by their upstream terms and by the repository tracking in DATA_LICENSES.md.

Citation

If you use this model in academic work, please cite it as follows:

bibtex

@misc{peikos2026sophira360mbase,
  title        = {Sophira-360M-base},
  author       = {Peikos, Georgios},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Gpeik/Sophira-360M-base}},
}