An offensive-security / agentic tool-calling LoRA merge on top of
hotdogs/Qwen3.8-27B-abliterated
— the training-free abliterated build of
Qwen/Qwen3.8-27B. The LoRA is
merged into the base at scale 1.0 (PEFT alpha/r = 64/32 = 2.0, i.e.
the exact strength the scale sweep below found optimal), and the MTP
(Multi-Token Prediction) head is preserved, so the GGUF build supports
self-speculative decoding.
This is a preview: the LoRA was trained on an agentic dataset that mixes
penetration-testing tool calls (<tool_call><function=...> XML) with
security QA. This model will not refuse. It is published for authorized
security research, red-team exercises, and studying LLM-driven tool use —
pentesting only systems you own or have explicit permission to test. You are
responsible for your use of it and for complying with all applicable laws.
Inherits the Apache-2.0 license of the base.
Quick Results
All numbers below were measured on our own reproduction pipeline (see
Method and Evaluation). The merged model adds a reliable, correctly
formatted tool-call pathway while leaving general capability essentially
unchanged.
Reading the numbers: the merge shifts the model's next-token distribution by a
tiny amount on general prompts (KL ≈ 0.04 → base knowledge is preserved), while
shifting it ~20× more on security/tool prompts (KL ≈ 0.81 → the intended
behavioral re-target to emit tool calls). The scale sweep and MTP numbers are
detailed below.
What changed
The base model already refuses nothing (abliterated). This repo adds a
tool-calling capability: given a pentest scenario, the model now emits a
structured, well-formed agentic call instead of free-form text.
User: Port scan the host 203.0.113.10 and identify which services are exposed.
<tool_call>
<function=nmap>
<parameter=target>
203.0.113.10
</parameter>
<parameter=ports>
-top 1000
</parameter>
</function>
</tool_call>
The tool-call format follows the Qwen3.5 native <tool_call>/<function>/<parameter>
schema (also exercised through the model's chat template). At scale 1.0 the
model picked the correct real tool per scenario in 6/6 cases.
Method
Base
hotdogs/Qwen3.8-27B-abliterated (λ = 1.2), a 27B native vision-language
hybrid (full-attention + GDN linear-attention) with a frozen MTP head.
Target modules: standard q/k/v/o/gate/up/down+ GDN
in_proj_qkv/out_proj/in_proj_z/a/b.
Merge
Full LoRA (incl. GDN) merged into the base via PEFT merge_and_unload() at
default scale (alpha/r = 2.0, ≡ llama.cpp --lora-scaled :1.0).
MTP head recovered after merge: merge_and_unload() + save_pretrained()
drops the mtp.* tensors, so the 15 MTP tensors (849 MB) were copied back
from the base into the merged safetensors (the LoRA never touched mtp.*,
so these weights are byte-correct). Final model = 1,199 tensors, verified
loading via unsloth, MTP physically present.
A small gap (≤ 0.15) means the model generalizes rather than memorizes.
The training loss plateau (~0.44 median, only ~3.8 % of points near zero) is a
healthy learning floor for a diverse 8.4K dataset, not overfitting.
2. Scale sweep (llama.cpp --lora-scaled, no merge needed)
6 tool prompts + 7 base prompts, temperature 0, identical config across scales:
scale
tool-call format (6)
correct real tool (6)
base capability (7)
baseline (no LoRA)
0/6
0/6
6/7
0.25
1/6
0/6
7/7
0.5
6/6
2/6
7/7
0.75
6/6
4/6
7/7
1.0
6/6
6/6
7/7
1.5
6/6
6/6
7/7
Scale 1.0 is the sweet spot: every tool prompt emits a well-formed
<tool_call>and picks the correct real tool, while base capability stays
at 7/7. Damping below 1.0 hurts tool selection (hallucinated tools like
web_fetch, wp_recon); going above ~2.0 degrades parameter quality (e.g.
ports="-s -p 1-65535"). The merge therefore uses the default
alpha/r = 2.0, which corresponds to this optimal 1.0 point.
3. KL divergence (base ‖ merged)
Measured on a shared continuation that the merged model sampled, comparing
top-25 next-token log-probs from the base vs the merged GGUF (both via
llama.cpp llama-server):
prompt
KL (base ‖ merged)
17 × 43
0.0066
Python reverse-string
0.1559
Capital of France
0.0206
stack vs queue
0.0556
15 % of 200
0.0030
boiling point
0.0041
mean (base prompts)
0.0410
port-scan tool prompt
0.7332
SQLi tool prompt
0.8833
mean (tool prompts)
0.8083
overall mean
0.2328
The ~20× gap between base-prompts (0.04) and tool-prompts (0.81) is the
quantitative signature of a surgically targeted LoRA: it re-wires
tool-calling behavior without disturbing general knowledge. This matches the
abliterated base's own near-zero first-token KL (0.0001) — the merge adds
capability, not drift.
4. MTP (Multi-Token Prediction)
metric
value
MTP draft acceptance rate
0.77 (51 / 66)
mean draft length
2.55
head location
in-file (bf16 GGUF)
Well above the ~0.3 threshold where speculative decoding pays off — MTP is
active and beneficial.
This is a preview built for authorized security research and red-teaming.
It will not refuse and may emit instructions for exploiting systems. Use only
on systems you own or are explicitly authorized to test. Not for
misuse/harmful activity. Apache-2.0, inherited from the base.
