You can deploy this model and start earning money today!

jalasoft/
smollm3-3B-it-ft-typ

Name: smollm3-3B-it-ft-typ
Availability: InStock

Views

No views yet

SmolLM3-3B-IT Fine-Tuned for Typst (Experimental)

License: MIT

An experimental QLoRA fine-tuned model for Typst code generation - shared for research and educational purposes

Warning: This is an experimental model that did not achieve production-quality results. It is shared for research transparency and educational purposes only.

SmolLM3-3B-IT Fine-Tuned for Typst (Experimental)

Model Summary

This model is a QLoRA fine-tuned version of HuggingFaceTB/SmolLM3-3B trained on the jalasoft/typst-instruct dataset for generating Typst markup language documents from natural language instructions.

Attribute	Value
Base Model	HuggingFaceTB/SmolLM3-3B
Fine-tuning Method	QLoRA (4-bit quantization)
Training Framework	Axolotl
Dataset	jalasoft/typst-instruct (1,016 examples)
Task	Text-to-Typst code generation
Status	Experimental (not production-ready)

Intended Use Cases (Not Achieved)

The model was developed with the goal of:

Document generation: Creating complete Typst documents from natural language descriptions
Template creation: Generating reusable Typst templates for various document types

However, the model did not meet the quality threshold required for these use cases due to the limitations described below.

Training Details

Training Procedure

Method: QLoRA (Quantized Low-Rank Adaptation)
Framework: Axolotl
Precision: BF16 with 4-bit quantization
Flash Attention: Enabled

Training Hyperparameters

Parameter	Value
LoRA Rank (r)	32
LoRA Alpha	96
LoRA Dropout	0.05
LoRA Target Modules	q_proj, k_proj, v_proj, o_proj
Learning Rate	2e-4
LR Scheduler	Cosine
Optimizer	AdamW (fused)
Epochs	3
Sequence Length	8192
Micro Batch Size	2
Gradient Accumulation	4
Warmup Ratio	0.1
Weight Decay	0.0
Sample Packing	Enabled
Gradient Checkpointing	Enabled

Training Data

Dataset: jalasoft/typst-instruct
Size: 1,016 instruction-completion pairs
Content: Natural language instructions paired with Typst code and feature summaries
Coverage: Academic documents, technical tutorials, reports, tables, figures, and advanced typography

Known Limitations

This model is marked as experimental due to significant issues observed during evaluation.

Observed Issues

Issue	Description
Hallucinations	The model frequently generates incorrect or non-existent Typst syntax and features
Code Quality	Generated Typst code often contains syntax errors and fails to compile

These limitations make the model unsuitable for production use in document generation workflows.

Recommendations

If you experiment with this model:

Always validate generated Typst code before use
Expect compilation errors in most complex task outputs
Use for research purposes to understand fine-tuning challenges
Consider larger models or more training data for production use cases

Evaluation

Performance was evaluated qualitatively across multiple model versions (v1, v2, v3) using 10 diverse Typst generation test cases. Results consistently fell below the quality threshold required for the intended use cases.

How to Use

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "jalasoft/smollm3-3B-it-ft-typ"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="float16")

messages = [
    {
        "role": "system",
        "content": "You are an expert in Typst markup language. Generate clean, well-formatted Typst code based on user instructions:"
    },
    {
        "role": "user",
        "content": "Create a simple document with a title and two paragraphs"
    }
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=1000)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation and Licensing

License

This model is released under the MIT License. You are free to use, modify, and distribute this model for any purpose, including commercial applications.

Citation

If you use this model in your research, please cite:

bibtex

@misc{smollm3-typst-experimental,
  author = {{Jala R\&D}},
  title = {SmolLM3-3B Fine-Tuned for Typst (Experimental)},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/jalasoft/smollm3-3B-it-ft-typ}}
}