NEXS LoRA Adapters — Serving & Evaluation Guide
Rank-128 LoRA adapters extracted with
mergekit from domain fine-tunes and
sanitized for vLLM serving (pure
lora_A/
lora_B weights only;
modules_to_save tensors stripped,
modules_to_save: null in configs; no
resize_token_embeddings() needed). Two families:
- Llama-3.1-8B — five adapters (finance, legal, medical, toxicity,
truthfulness), 224 A/B pairs each (32 layers × q/k/v/o/gate/up/down).
- Qwen3-32B — six adapters (instruction-following, medical ×2,
theorem proving, Russian ×2), 448 A/B pairs each (64 layers × 7 projections).
Llama-3.1-8B adapters on the Hub
Qwen3-32B adapters on the Hub
Serve the five Llama adapters with vLLM
Directly from the Hub (vLLM downloads the adapters itself; requires
hf auth login or HF_TOKEN for the gated base model):
1python -m vllm.entrypoints.openai.api_server \
2 --model meta-llama/Llama-3.1-8B \
3 --enable-lora \
4 --lora-modules \
5 finance=anjohn0077/NEXS-finance-lora \
6 legal=anjohn0077/NEXS-legal-lora \
7 medical=anjohn0077/NEXS-medical-lora \
8 toxicity=anjohn0077/NEXS-toxicity-lora \
9 truthfulness=anjohn0077/NEXS-truthfulness-lora \
10 --port 8000 \
11 --max-loras 5 \
12 --max-lora-rank 128 \
13 --gpu-memory-utilization 0.85
Or from the local sanitized copies (on this cluster):
1python -m vllm.entrypoints.openai.api_server \
2 --model meta-llama/Llama-3.1-8B \
3 --enable-lora \
4 --lora-modules \
5 finance=/scratch/shared_dir/lora/llama/finance_sanitized \
6 legal=/scratch/shared_dir/lora/llama/legal_sanitized \
7 medical=/scratch/shared_dir/lora/llama/medical_sanitized \
8 toxicity=/scratch/shared_dir/lora/llama/toxicity_sanitized \
9 truthfulness=/scratch/shared_dir/lora/llama/truthfulness_sanitized \
10 --port 8000 \
11 --max-loras 5 \
12 --max-lora-rank 128 \
13 --gpu-memory-utilization 0.85
To fetch the Hub copies to a local directory instead:
1for d in finance legal medical toxicity truthfulness; do
2 hf download "anjohn0077/NEXS-${d}-lora" --local-dir "./adapters/${d}"
3done
Serve the six Qwen3-32B adapters with vLLM
The base model is ~65 GB in bf16; one 141 GB H200 fits the model, all six
adapters, and KV cache at --gpu-memory-utilization 0.85.
1python -m vllm.entrypoints.openai.api_server \
2 --model Qwen/Qwen3-32B \
3 --enable-lora \
4 --lora-modules \
5 IF=anjohn0077/NEXS-qwen3-32b-IF-lora \
6 medical_openmedzoo=anjohn0077/NEXS-qwen3-32b-medical-openmedzoo-lora \
7 medical_tachyhealth=anjohn0077/NEXS-qwen3-32b-medical-tachyhealth-lora \
8 prover=anjohn0077/NEXS-qwen3-32b-prover-lora \
9 russian_refalmachine=anjohn0077/NEXS-qwen3-32b-russian-refalmachine-lora \
10 russian_t_tech=anjohn0077/NEXS-qwen3-32b-russian-t-tech-lora \
11 --port 8000 \
12 --max-loras 6 \
13 --max-lora-rank 128 \
14 --gpu-memory-utilization 0.85
Or from the local sanitized copies (on this cluster), replace each repo id
with /scratch/shared_dir/lora/qwen/<dir>_sanitized (dirs: IF_sanitized,
medical_OpenMedZoo_sanitized, medical_TachyHealth_sanitized,
prover_sanitized, russian_RefalMachine_sanitized,
russian_t-tech_sanitized).
Evaluate with lm-evaluation-harness
With a server running on port 8000, each adapter is addressed by the name
given in --lora-modules. Llama family:
1lm_eval --model local-completions \
2 --model_args model=finance,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
3 --tasks mmlu_econometrics \
4 --output_path results/vllm_finance
5
6lm_eval --model local-completions \
7 --model_args model=legal,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
8 --tasks mmlu_professional_law \
9 --output_path results/vllm_legal
10
11lm_eval --model local-completions \
12 --model_args model=medical,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
13 --tasks mmlu_professional_medicine \
14 --output_path results/vllm_medical
15
16lm_eval --model local-completions \
17 --model_args model=toxicity,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
18 --tasks sst2 \
19 --output_path results/vllm_toxicity_sst2
20
21lm_eval --model local-completions \
22 --model_args model=truthfulness,base_url=http://localhost:8000/v1/completions,tokenizer=meta-llama/Llama-3.1-8B,num_concurrent=10 \
23 --tasks truthfulqa_mc2 \
24 --output_path results/vllm_truthfulness
Qwen3-32B family (note tokenizer=Qwen/Qwen3-32B):
1lm_eval --model local-completions \
2 --model_args model=IF,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
3 --tasks ifeval \
4 --output_path results/vllm_qwen_IF
5
6lm_eval --model local-completions \
7 --model_args model=medical_openmedzoo,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
8 --tasks medqa_4options \
9 --output_path results/vllm_qwen_medical_openmedzoo
10
11lm_eval --model local-completions \
12 --model_args model=medical_tachyhealth,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
13 --tasks medqa_4options \
14 --output_path results/vllm_qwen_medical_tachyhealth
15
16lm_eval --model local-completions \
17 --model_args model=russian_refalmachine,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
18 --tasks m_mmlu_ru \
19 --output_path results/vllm_qwen_russian_refalmachine
20
21lm_eval --model local-completions \
22 --model_args model=russian_t_tech,base_url=http://localhost:8000/v1/completions,tokenizer=Qwen/Qwen3-32B,num_concurrent=10 \
23 --tasks m_mmlu_ru \
24 --output_path results/vllm_qwen_russian_t_tech
(The prover adapter's intended minif2f evaluation has not produced results
yet, so it ships unbenchmarked.)
Benchmark results
Accuracy of each LoRA served on the base model via vLLM, compared against the
plain base model and the original full fine-tune it was extracted from.
Llama-3.1-8B family:
| Domain | Benchmark | Base | LoRA (vLLM) | Full fine-tune |
|---|
| finance | mmlu_econometrics | 0.4825 | 0.4561 | 0.5526 |
| legal | mmlu_professional_law | 0.4941 | 0.4915 | 0.4948 |
| medical | mmlu_professional_medicine | 0.7169 | 0.7059 | 0.7721 |
| toxicity | sst2 | 0.6732 | 0.8991 | 0.8899 |
| truthfulness | truthfulqa_mc2 | 0.4416 | 0.7164 | 0.7307 |
Note: toxicity and truthfulness recover nearly all (or more than) the
fine-tune's gains; finance/medical/legal LoRAs land close to base — most of
those fine-tunes' improvement lived in the full-rank embedding/head deltas
that vLLM cannot serve and were stripped during sanitization.
Qwen3-32B family (from qwen32b_adapter_acc.csv):
| Adapter | Benchmark | Base | LoRA (vLLM) | Full fine-tune |
|---|
| IF | ifeval | 0.8336 | 0.2737 | 0.8669 |
| medical-openmedzoo | medqa_4options | 0.7494 | 0.7604 | 0.7596 |
| medical-tachyhealth | medqa_4options | 0.7494 | 0.7455 | 0.7478 |
| russian-refalmachine | m_mmlu_ru | 0.7517 | 0.7524 | 0.7370 |
| russian-t-tech | m_mmlu_ru | 0.7517 | 0.7542 | 0.7679 |
| prover | minif2f | — | — | — |
Notes: the IF LoRA scores far below base on ifeval (0.2737 vs 0.8336) — the
low-rank approximation appears to break the fine-tune's instruction-following
behavior rather than approximate it; it is published for completeness with
these honest numbers. The medical and russian adapters track their full
fine-tunes closely. prover has not been benchmarked (its minif2f run produced
no results), but it passes structural validation and serves through vLLM.
Dependencies (for replication)
All extraction, sanitization, serving, and evaluation ran in a single Python
virtualenv on Linux with NVIDIA H200 GPUs (driver 595.71.05). Exact versions:
| Component | Version | Notes |
|---|
| Python | 3.12.3 | |
| torch | 2.9.1 | CUDA 12.8 build (2.9.1+cu128) |
| mergekit | 0.1.4 | fork: ikhyunAn/mergekit @ e85a454, editable install — not upstream arcee-ai/mergekit |
| vllm | 0.15.1 | |
| transformers | 4.57.1 | |
| peft | 0.15.2 | |
| huggingface-hub | 0.35.3 | pinned <1.0 (required by transformers 4.57.1) |
| safetensors | 0.5.3 | |
| accelerate | 1.6.0 | |
| tokenizers | 0.22.1 | |
| xformers | 0.0.32.post1 | |
| numpy | 2.2.6 | |
| lm_eval | 0.4.9.1 | EleutherAI/lm-evaluation-harness @ ad3f4d0c, editable install |
To recreate the core environment:
1python3.12 -m venv lora-env && source lora-env/bin/activate
2pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu128
3pip install vllm==0.15.1 transformers==4.57.1 peft==0.15.2 \
4 "huggingface_hub==0.35.3" safetensors==0.5.3 accelerate==1.6.0
5pip install -e "git+https://github.com/ikhyunAn/mergekit.git@e85a454#egg=mergekit"
6pip install -e "git+https://github.com/EleutherAI/lm-evaluation-harness.git@ad3f4d0c#egg=lm_eval"
Caveats:
- Do not upgrade
huggingface_hub past 1.0 in this environment —
transformers 4.57.1 pins huggingface-hub<1.0 and both transformers and
mergekit break at import time with 1.x.
- Extraction used a single GPU (
CUDA_VISIBLE_DEVICES=<idx> with
mergekit-extract-lora --cuda); each 8B extraction takes ~4 minutes on an
H200 and peaks around 2×16 GB of downloaded weights in the HF cache.
Provenance / pipeline
- Extraction: `mergekit-extract-lora --model --base-model
--out-path --cuda` with base `meta-llama/Llama-3.1-8B` for the
llama family (see `extract_and_upload_loras.sh`; raw outputs in
`/scratch/shared_dir/lora/llama//`) and `Qwen/Qwen3-32B` for the
qwen family (raw outputs in `/scratch/shared_dir/lora/qwen/`).
- Sanitization (same script set in
/scratch/shared_dir/lora/llama/ and
/scratch/shared_dir/lora/qwen/): fix_adapters.py (drop norm layers) →
fix_vllm_weights.py / final_clean.py (drop all non-lora_* keys, null
out modules_to_save) → verify_all_clean.py. For llama, toxicity's
extended vocab (128258) was truncated to the base 128256 first via
fix_toxicity_vocab.py.
- Upload:
upload_sanitized_loras.py (llama) and upload_qwen_loras.py
(qwen), both in this directory, push each sanitized adapter to its Hub
repo with an updated README.
