1,060 knowledge-distilled QA pairs from 381 SEC summaries (10-K / 10-Q / 8-K) covering 69 S&P 500 companies. Knowledge distillation done with Gemini 3.1 Pro.

Hyperparameters

LoRA rank=8, alpha=16, dropout=0.05
Targets: q/k/v/o/gate/up/down_proj (regex-scoped to .language_model. for multimodal Gemma 4)
bs=4 × grad_accum=2, AdamW lr=1e-4, 2 epochs, bf16
FSDPv2 mesh (8, 1) over ("fsdp", "tensor")

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-31B-it",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("google/gemma-4-31B-it")
model = PeftModel.from_pretrained(base, "Srx7703/gemma-4-31b-financial-adapter")

prompt = "What are the principal risk factors disclosed in NVIDIA's most recent 10-K?"
inputs = tok.apply_chat_template(
    [{"role": "user", "content": prompt}],
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)
out = model.generate(inputs, max_new_tokens=256)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

License

Use of this adapter is subject to the Gemma Terms of Use. The base model weights are not included.

Srx7703/gemma-4-31b-financial-adapter

gemma-4-31B-it — SEC Financial QA LoRA Adapter (Phase 2)

Result (n=20 held-out)

Training data

Hyperparameters

Usage

See also

License

Srx7703/
gemma-4-31b-financial-adapter