The first release is a continued-pretraining checkpoint trained only to improve a compatible base model's understanding of Pharo syntax, source-code structure, message-passing conventions, and common language patterns.

It is not yet fine-tuned for code generation, code completion, test generation, bug fixing, code repair, or instruction following. Those capabilities may be introduced in later checkpoints within the collection.

Pharo is a low-resource programming language with syntax and conventions that differ significantly from mainstream languages. The purpose of this first checkpoint is language adaptation only: it teaches the model what Pharo code looks like before any downstream task-specific fine-tuning.

Collection Overview

The pharo-one collection may eventually contain multiple checkpoints trained for different tasks and model sizes.

The first checkpoint is:

Continued-pretrained on raw Pharo source code
Trained with a causal language-modeling objective
Intended only for Pharo language and syntax adaptation
Distributed as a LoRA or PEFT adapter, when applicable

Future checkpoints may add supervised fine-tuning for code generation, completion, testing, repair, or other downstream tasks.

Each checkpoint should be used according to the task and training configuration described in its repository.

The underlying base model may differ between checkpoints. Consult the model configuration, adapter configuration, or repository metadata to identify the compatible base checkpoint.

Model Details

Collection name: pharo-one
Model family: Decoder-only causal language models
Programming language: Pharo / Smalltalk
Primary domain: Source-code generation and understanding
Framework: Transformers
Adapter framework: PEFT / LoRA, when applicable
Training method: Continued pre-training
Current purpose: Pharo syntax and language adaptation
Current task scope: Causal language modeling only

Motivation

Large language models perform well on popular programming languages such as Python, Java, JavaScript, and C++, but their performance is often weaker on low-resource languages such as Pharo.

Pharo inherits distinctive features from Smalltalk, including:

Keyword-based message syntax
Message-oriented control flow
Dynamic typing
Reflective APIs
Image-based development
Compact method definitions
Test-driven development conventions

General-purpose code models may not represent these characteristics accurately. The pharo-one collection adapts coding models to Pharo syntax, APIs, idioms, and development patterns.

Training Overview

The first model in this collection uses the following training stage.

Continued Pre-Training

A compatible causal language model is adapted to Pharo source code using next-token prediction.

This stage helps the model learn:

Pharo syntax
Common selectors
Standard library APIs
Class and method conventions
Message-passing patterns
Naming and formatting conventions
Typical project structures

Training examples are tokenized, concatenated, and packed into fixed-length causal language-modeling blocks.

Current Scope

The first pharo-one checkpoint is intended only to improve the model's internal representation of Pharo code.

It is trained to learn:

Pharo syntax
Message-send structure
Keyword, unary, and binary messages
Method and class-definition patterns
Common selectors and naming conventions
Typical source-code formatting
Recurrent Pharo language patterns

It should not be presented as a task-specialized model for code generation, test generation, bug fixing, or fill-in-the-middle completion.

Installation

Install the required libraries:

pip install -U transformers accelerate torch peft

Some checkpoints may require additional packages. Refer to the corresponding model repository for checkpoint-specific requirements.

Usage

Loading a Standalone Model

Use this pattern for checkpoints distributed as complete models:

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "<PHARO_ONE_MODEL_ID>"

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
).eval()

Loading a LoRA or PEFT Adapter

Use this pattern for checkpoints distributed as adapters:

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_name = "<COMPATIBLE_BASE_MODEL_ID>"
adapter_name = "<PHARO_ONE_ADAPTER_ID>"

tokenizer = AutoTokenizer.from_pretrained(
    base_model_name,
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(
    model,
    adapter_name,
).eval()

The base model must match the checkpoint used to train the adapter. Check adapter_config.json or the checkpoint repository for the required base-model identifier.

Basic Causal-Language-Modeling Example

python

prompt = """Collection >> selectEven

    ^ self select: [ :each |
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
    )

generated_text = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)

print(generated_text)

The model may continue the Pharo source sequence with text such as:

smalltalk

Collection >> selectEven

    ^ self select: [ :each | each even ]

Future Test-Generation Models

python

prompt = """Generate a Pharo unit test for this method:

Calculator >> add: firstNumber to: secondNumber
    ^ firstNumber + secondNumber
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=192,
        do_sample=False,
    )

completion = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)

print(completion)

A future task-specific checkpoint may produce results such as:

smalltalk

CalculatorTest >> testAdd

    self
        assert: (calculator add: 2 to: 3)
        equals: 5

Future Bug-Fixing Models

python

prompt = """Fix the following Pharo method:

Collection >> firstEven
    ^ self detect: [ :each | each odd ]
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
    )

completion = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True,
)

print(completion)

A future task-specific checkpoint may produce corrections such as:

smalltalk

Collection >> firstEven

    ^ self detect: [ :each | each even ]

Prompt Formats

The first checkpoint uses ordinary causal-language-model input rather than a task-specific instruction format.

A checkpoint may use:

Plain causal-language-model prompts
Instruction-style prompts
Chat templates
Fill-in-the-middle tokens
Structured task prefixes
Repository-specific templates

Always consult the individual checkpoint model card for the recommended prompt format.

Training Configuration

Training settings may differ between checkpoints. A typical language-adaptation run may use:

LoRA or full-model fine-tuning
Causal language modeling
Packed source-code sequences
Automatic mixed precision
Gradient checkpointing
Multi-GPU distributed training
Periodic evaluation and checkpointing
Automatic checkpoint resumption
Validation loss and perplexity tracking

The exact values for context length, learning rate, batch size, LoRA rank, number of epochs, and target modules are checkpoint-specific.

Evaluation

Models in the collection should be evaluated on task-appropriate Pharo datasets.

Possible metrics include:

Language Adaptation

Validation loss
Perplexity

Code Completion

Exact match
Edit similarity
Token accuracy
CodeBLEU
Compilation or parsing success

Test Generation

Test compilation rate
Test execution rate
Assertion correctness
Mutation score
Code coverage

Bug Fixing

Exact patch match
Plausible patch rate
Test-suite pass rate
Functional correctness

Evaluation results should be reported in each checkpoint repository when available.

Intended Use

The first pharo-one checkpoint is intended for:

Continued pre-training research
Pharo syntax adaptation
Low-resource programming-language research
Parameter-efficient language adaptation
Use as a starting point for later supervised fine-tuning
Local experimentation with Pharo source-code modeling

It is not intended to be evaluated or advertised as a specialized code-generation, test-generation, bug-fixing, or code-completion model without additional task-specific fine-tuning.

Limitations

Generated code may be incorrect, incomplete, insecure, or non-compiling.
The models do not execute or validate generated Pharo code.
Generated selectors, classes, methods, or APIs may not exist.
Performance varies across checkpoints and base-model families.
Some models are specialized for a single task and may perform poorly on others.
LoRA adapters require the exact compatible base model.
FIM tokens and chat templates may differ between model families.
The models may reproduce errors, biases, insecure patterns, or licensing characteristics found in their training data.
Repository-level reasoning may be limited when the required context exceeds the model context window.
Generated tests may pass without adequately validating program behavior.
Bug-fixing outputs should be reviewed and verified with the project test suite.

Generated code should be reviewed, executed, and tested before use in production.

Responsible Use

Users should:

Review generated code manually
Run relevant tests
Validate API and selector availability
Check licensing requirements
Avoid using generated code without verification in critical systems
Avoid submitting private or sensitive source code to untrusted inference services

Reproducibility

Exact results may vary across:

Base-model versions
Tokenizers
Dataset revisions
Hardware
Precision modes
PyTorch versions
Transformers versions
PEFT versions
CUDA versions
Distributed-training configurations
Decoding parameters

For reproducible experiments, record:

The full checkpoint identifier
The model revision
The dataset revision
The training command
The random seed
Dependency versions
Hardware information
Prompt templates
Generation parameters

License

This repository is released under the MIT License.

Individual checkpoints may also be subject to the license and usage terms of their underlying base models and training datasets. Users are responsible for reviewing all applicable licenses before use, redistribution, or deployment.

pharo-llm/pharo-one-qwen2-5-1-5B

pharo-one

Collection Overview

Model Details

Motivation

Training Overview

Continued Pre-Training

Current Scope

Installation

Usage

Loading a Standalone Model

Loading a LoRA or PEFT Adapter

Basic Causal-Language-Modeling Example

Future Test-Generation Models

Future Bug-Fixing Models

Prompt Formats

Training Configuration

Evaluation

Language Adaptation

Code Completion

Test Generation

Bug Fixing

Intended Use

Limitations

Responsible Use

Reproducibility

License

pharo-llm/
pharo-one-qwen2-5-1-5B