Qwen3 4B Thinking 2507 Heretic CodeFeedback — Agentic Tessa 8K 2048 Experimental LoRA
This repository contains an experimental LoRA adapter trained on top of:
This adapter is a larger continuation experiment using:
This is not recommended as the main version.
Status
This is an experimental adapter kept for transparency, comparison, and future analysis.
In a small local Python coding benchmark, this adapter regressed compared with the previous CodeFeedback checkpoint.
| Model | Adapter | Passed | Pass rate | Avg tokens/s |
|---|
| Before | heretic_F_lora_python5000_codefeedback5000 | 9/10 | 90.00% | 9.54 |
| After | NOITE_3090_TESSA_8000_2048 | 7/10 | 70.00% | 9.28 |
Delta:
| Metric | Value |
|---|
| Passes | -2 |
| Pass rate | -20.00% |
| Avg tokens/s | -0.26 |
Observed behavior
The adapter did not fail completely, but it became worse at strict executable-code output.
Observed regressions:
flatten failed with a type error.valid_parentheses failed to output executable code.lru_cache remained incomplete.- The model showed more explanatory / agentic behavior instead of always returning compact executable code.
This suggests that the larger Agentic Tessa continuation pushed the model toward a more verbose agentic-assistant style, which may be useful for some workflows but is worse for strict code-output benchmarks.
Training configuration
| Item | Value |
|---|
| Base model | JoaoZaokk/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback |
| Input adapter | heretic_F_lora_tessa_agentic_1000_test |
| Dataset | smirki/Agentic-Coding-Tessa |
| Samples used | 8,000 |
| Sequence length | 2048 |
| Epochs | 1 |
| Learning rate | 7e-7 |
| Training method | QLoRA / LoRA |
| Quantized loading during training | 4-bit NF4 |
| Trainable parameters | ~33M |
| Trainable percentage | ~0.81% |
Training result
| Metric | Value |
|---|
| Train runtime | 9421 seconds |
| Runtime | 2h 37m 00s |
| Samples/second | 0.849 |
| Steps/second | 0.106 |
| Final train loss | 1.178 |
| First logged loss | 1.509 |
| Last logged loss | 1.072 |
Benchmark files
Benchmark artifacts are included under:
Files:
1benchmark/before_summary.md
2benchmark/after_summary.md
3benchmark/COMPARISON.md
4benchmark/comparison.json
5benchmark/before_results.jsonl
6benchmark/after_results.jsonl
Intended use
This adapter is intended for:
- comparison against the previous CodeFeedback checkpoint
- studying regression from larger agentic fine-tuning
- analyzing output-style drift
- future experiments with smaller learning rates or filtered datasets
It is not recommended for strict agentic coding workflows that require compact executable code output.
For the stronger current baseline, prefer:
Loading example
1from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
2from peft import PeftModel
3import torch
4
5base_model = "JoaoZaokk/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback"
6adapter = "JoaoZaokk/Qwen3-4B-Thinking-2507-Heretic-CodeFeedback-Agentic-Tessa-8K-2048-Experimental-LoRA"
7
8tokenizer = AutoTokenizer.from_pretrained(base_model, trust_remote_code=True)
9
10bnb_config = BitsAndBytesConfig(
11 load_in_4bit=True,
12 bnb_4bit_quant_type="nf4",
13 bnb_4bit_compute_dtype=torch.float16,
14 bnb_4bit_use_double_quant=True,
15)
16
17model = AutoModelForCausalLM.from_pretrained(
18 base_model,
19 quantization_config=bnb_config,
20 device_map="auto",
21 trust_remote_code=True,
22)
23
24model = PeftModel.from_pretrained(model, adapter)
25model.eval()
Important notes
This is an experimental LoRA adapter.
It should not be treated as a universal improvement over the previous CodeFeedback model.
The benchmark used here is small and should not be treated as a formal coding leaderboard. It is mainly useful for local before/after regression testing.
