JL-Code-Python-97M is an experimental, Python-specialized decoder-only language model developed by JumpLander.
The checkpoint contains exactly 97,536,768 trainable parameters—approximately 97.54 million parameters. It was trained from randomly initialized weights and does not inherit pretrained weights from Qwen, Llama, DeepSeek, Code Llama, GPT, or another external model family.
The release focuses on a narrow technical domain:
Python source-code continuation
Python function completion
technical English associated with Python code
docstring-to-code patterns
short English-to-Python programming tasks
fill-in-the-middle code reconstruction
This is a base research checkpoint, not a general chat model and not a production coding assistant.
Compatibility notice: this release uses a custom PyTorch architecture and a custom .pt checkpoint. It is not currently loadable through AutoModelForCausalLM.from_pretrained(...). Use the included jumplander_python_100m.py file for loading and inference.
JL-Code model family
JL-Code is JumpLander's language-specific code-model line. Each branch is intended to focus on one programming language rather than mixing every language into a single small checkpoint.
Purpose: Python source-code pretraining and code-structure learning
Source type: deduplicated Python files collected from public GitHub repositories
The upstream dataset includes per-file license metadata and contains code under multiple software licenses. Users should review the dataset card and original repository licenses before downstream commercial use.
Purpose: short technical-English-to-Python examples and held-out programming tasks
Content: natural-language tasks, Python reference solutions, and automated tests
MBPP examples assigned to test or validation splits were kept outside the main training split by the project pipeline.
Data preparation pipeline
The dataset was not passed directly into training. The local pipeline applied additional filtering and transformation:
stream Python records from Hugging Face;
normalize line endings and remove null bytes;
reject very short or oversized files;
detect likely credentials, keys, and private-key blocks;
reject generated or minified files;
validate Python using ast.parse;
split large source files at Python function/class boundaries where possible;
remove exact duplicates using SHA-256;
create deterministic train, validation, and test splits;
extract selected function/docstring pairs as English-to-Python instruction rows;
add MBPP examples according to their upstream split;
generate deterministic fill-in-the-middle variants for base training rows;
tokenize and pack the corpus into contiguous uint16 token streams.
Prepared dataset statistics
Statistic
Count
Accepted base samples
20,000
Final training rows
35,855
Final validation rows
389
Final test rows
793
Exact duplicates rejected
391
Generated files rejected
246
Secret-like samples rejected
75
Syntax-invalid samples rejected
1,523
Oversized samples rejected
268
Source records read
10,809
A single accepted source file can produce more than one training row—for example, a base-code sample plus extracted function/docstring instruction samples. This is why the final training-row count is larger than the accepted-base-sample count.
Training configuration
The model was trained using next-token prediction over packed token sequences.
Setting
Value
Optimizer
AdamW
Optimizer betas
(0.9, 0.95)
Optimizer epsilon
1e-8
Weight decay
0.1
Peak learning rate
3e-4
Minimum learning rate
3e-5
Scheduler
Cosine decay
Warmup
200 optimizer steps
Total optimizer steps
10,000
Micro-batch size
1 sequence
Gradient accumulation
32
Effective batch
32 sequences
Sequence length
1,024 tokens
Tokens per optimizer step
32,768
Approximate tokens processed
327,680,000
Gradient clipping
1.0
Evaluation interval
250 steps
Checkpoint interval
500 steps
Random seed
1,337
Precision
BF16 mixed precision
Hardware
NVIDIA RTX 3060 12GB
Observed during training:
approximately 8,400–9,000 tokens/second;
approximately 1.86 GB peak allocated VRAM in the reported run;
stable gradient norms during the observed training window.
Do not upload .venv, __pycache__, Hugging Face caches, tokenized .bin training files, or private training logs unless they are intentionally part of the release.
Because the checkpoint includes optimizer state, it can be larger than a weights-only release.
A future release should also provide:
a weights-only checkpoint;
model.safetensors;
native Hugging Face Transformers integration;
AutoModelForCausalLM loading support.
Evaluation status
This release documents the completed training run, but it does not yet claim a verified HumanEval, MBPP pass@1, or production code-generation score.
Training loss alone does not establish code correctness. Proper evaluation should include:
Python AST parse rate;
executable completion rate;
unit-test pass rate;
MBPP pass@1 and pass@k;
HumanEval evaluation after contamination review;
repetition and memorization checks;
security-oriented code review.
Until those evaluations are published, treat this checkpoint as an experimental base model.
Intended uses
Appropriate uses:
research on small code language models;
experiments with Python code completion;
educational study of from-scratch Transformer training;
tokenizer and data-pipeline research;
local inference experiments;
continued pretraining and instruction tuning;
analysis of consumer-GPU model development.
Out-of-scope uses
This model is not intended for:
production-critical code generation;
security-sensitive implementation without review;
autonomous deployment of generated code;
legal, medical, financial, or safety-critical systems;
generating or executing untrusted code without sandboxing;
replacing human code review and automated testing.
Limitations
The model is small and narrowly trained.
It has limited general-world knowledge.
It may generate invalid, incomplete, insecure, or fabricated Python code.
English understanding is primarily tied to technical programming patterns.
The context window is limited to 1,024 tokens.
The training corpus is much smaller than corpora used for leading code models.
GitHub-derived data can contain bugs, insecure patterns, biases, or licensing constraints.
The current architecture is custom and is not yet integrated with Transformers.
Generated code must be reviewed, sandboxed, and tested before use.
Safety and responsible use
Never execute generated code directly in a privileged environment.
Recommended controls:
run generated code in a restricted sandbox;
disable network access where possible;
apply CPU, memory, and execution-time limits;
scan generated code for secrets and unsafe operations;
require unit tests and human review;
avoid exposing private repositories or credentials to untrusted systems.
Licensing and data notice
The repository declares the Apache License 2.0 for JumpLander-authored code and released model artifacts.
That license declaration does not replace or override the licenses attached to upstream source files used by the training datasets. codeparrot/codeparrot-clean includes per-file license metadata from many public repositories. Users are responsible for reviewing the upstream dataset cards, source-code licenses, attribution requirements, and applicable law before commercial or regulated use.
MBPP is commonly distributed under CC BY 4.0; consult the current dataset card for attribution and reuse requirements.
This section is informational and is not legal advice.
Roadmap
Potential follow-up work:
publish a weights-only safetensors checkpoint;
add Transformers-compatible configuration and model classes;
publish reproducible evaluation scripts;
evaluate on MBPP and HumanEval;
expand high-quality Python training data;
create a stronger instruction-tuned Python variant;
train larger Python-specific models;
extend the JL-Code family to additional programming languages.
