pharo-one is a collection of language models and adapters for Pharo.
The first release is a continued-pretraining checkpoint trained only to improve a compatible base model's understanding of Pharo syntax, source-code structure, message-passing conventions, and common language patterns.
It is not yet fine-tuned for code generation, code completion, test generation, bug fixing, code repair, or instruction following. Those capabilities may be introduced in later checkpoints within the collection.
Pharo is a low-resource programming language with syntax and conventions that differ significantly from mainstream languages. The purpose of this first checkpoint is language adaptation only: it teaches the model what Pharo code looks like before any downstream task-specific fine-tuning.
Collection Overview
The pharo-one collection may eventually contain multiple checkpoints trained for different tasks and model sizes.
The first checkpoint is:
Continued-pretrained on raw Pharo source code
Trained with a causal language-modeling objective
Intended only for Pharo language and syntax adaptation
Distributed as a LoRA or PEFT adapter, when applicable
Future checkpoints may add supervised fine-tuning for code generation, completion, testing, repair, or other downstream tasks.
Each checkpoint should be used according to the task and training configuration described in its repository.
The underlying base model may differ between checkpoints. Consult the model configuration, adapter configuration, or repository metadata to identify the compatible base checkpoint.
Model Details
Collection name: pharo-one
Model family: Decoder-only causal language models
Programming language: Pharo / Smalltalk
Primary domain: Source-code generation and understanding
Framework: Transformers
Adapter framework: PEFT / LoRA, when applicable
Training method: Continued pre-training
Current purpose: Pharo syntax and language adaptation
Current task scope: Causal language modeling only
Motivation
Large language models perform well on popular programming languages such as Python, Java, JavaScript, and C++, but their performance is often weaker on low-resource languages such as Pharo.
Pharo inherits distinctive features from Smalltalk, including:
Keyword-based message syntax
Message-oriented control flow
Dynamic typing
Reflective APIs
Image-based development
Compact method definitions
Test-driven development conventions
General-purpose code models may not represent these characteristics accurately. The pharo-one collection adapts coding models to Pharo syntax, APIs, idioms, and development patterns.
Training Overview
The first model in this collection uses the following training stage.
Continued Pre-Training
A compatible causal language model is adapted to Pharo source code using next-token prediction.
This stage helps the model learn:
Pharo syntax
Common selectors
Standard library APIs
Class and method conventions
Message-passing patterns
Naming and formatting conventions
Typical project structures
Training examples are tokenized, concatenated, and packed into fixed-length causal language-modeling blocks.
Current Scope
The first pharo-one checkpoint is intended only to improve the model's internal representation of Pharo code.
It is trained to learn:
Pharo syntax
Message-send structure
Keyword, unary, and binary messages
Method and class-definition patterns
Common selectors and naming conventions
Typical source-code formatting
Recurrent Pharo language patterns
It should not be presented as a task-specialized model for code generation, test generation, bug fixing, or fill-in-the-middle completion.
Installation
Install the required libraries:
pip install -U transformers accelerate torch peft
Some checkpoints may require additional packages. Refer to the corresponding model repository for checkpoint-specific requirements.
Usage
Loading a Standalone Model
Use this pattern for checkpoints distributed as complete models:
The base model must match the checkpoint used to train the adapter. Check adapter_config.json or the checkpoint repository for the required base-model identifier.
A future task-specific checkpoint may produce corrections such as:
smalltalk
1Collection >> firstEven
23^self detect:[:each| each even ]
Prompt Formats
The first checkpoint uses ordinary causal-language-model input rather than a task-specific instruction format.
A checkpoint may use:
Plain causal-language-model prompts
Instruction-style prompts
Chat templates
Fill-in-the-middle tokens
Structured task prefixes
Repository-specific templates
Always consult the individual checkpoint model card for the recommended prompt format.
Training Configuration
Training settings may differ between checkpoints. A typical language-adaptation run may use:
LoRA or full-model fine-tuning
Causal language modeling
Packed source-code sequences
Automatic mixed precision
Gradient checkpointing
Multi-GPU distributed training
Periodic evaluation and checkpointing
Automatic checkpoint resumption
Validation loss and perplexity tracking
The exact values for context length, learning rate, batch size, LoRA rank, number of epochs, and target modules are checkpoint-specific.
Evaluation
Models in the collection should be evaluated on task-appropriate Pharo datasets.
Possible metrics include:
Language Adaptation
Validation loss
Perplexity
Code Completion
Exact match
Edit similarity
Token accuracy
CodeBLEU
Compilation or parsing success
Test Generation
Test compilation rate
Test execution rate
Assertion correctness
Mutation score
Code coverage
Bug Fixing
Exact patch match
Plausible patch rate
Test-suite pass rate
Functional correctness
Evaluation results should be reported in each checkpoint repository when available.
Intended Use
The first pharo-one checkpoint is intended for:
Continued pre-training research
Pharo syntax adaptation
Low-resource programming-language research
Parameter-efficient language adaptation
Use as a starting point for later supervised fine-tuning
Local experimentation with Pharo source-code modeling
It is not intended to be evaluated or advertised as a specialized code-generation, test-generation, bug-fixing, or code-completion model without additional task-specific fine-tuning.
Limitations
Generated code may be incorrect, incomplete, insecure, or non-compiling.
The models do not execute or validate generated Pharo code.
Generated selectors, classes, methods, or APIs may not exist.
Performance varies across checkpoints and base-model families.
Some models are specialized for a single task and may perform poorly on others.
LoRA adapters require the exact compatible base model.
FIM tokens and chat templates may differ between model families.
The models may reproduce errors, biases, insecure patterns, or licensing characteristics found in their training data.
Repository-level reasoning may be limited when the required context exceeds the model context window.
Generated tests may pass without adequately validating program behavior.
Bug-fixing outputs should be reviewed and verified with the project test suite.
Generated code should be reviewed, executed, and tested before use in production.
Responsible Use
Users should:
Review generated code manually
Run relevant tests
Validate API and selector availability
Check licensing requirements
Avoid using generated code without verification in critical systems
Avoid submitting private or sensitive source code to untrusted inference services
Reproducibility
Exact results may vary across:
Base-model versions
Tokenizers
Dataset revisions
Hardware
Precision modes
PyTorch versions
Transformers versions
PEFT versions
CUDA versions
Distributed-training configurations
Decoding parameters
For reproducible experiments, record:
The full checkpoint identifier
The model revision
The dataset revision
The training command
The random seed
Dependency versions
Hardware information
Prompt templates
Generation parameters
License
This repository is released under the MIT License.
Individual checkpoints may also be subject to the license and usage terms of their underlying base models and training datasets. Users are responsible for reviewing all applicable licenses before use, redistribution, or deployment.
