This is a LoRA adapter for Qwen3-8B with HiCI (Hierarchical Construction-Integration) architecture, trained for long-context understanding up to 48K tokens.

Paper: HiCI (arXiv 2603.20843)

HiCI Architecture

Three-stage hierarchy per transformer layer:

Local Construction — M learnable query slots attend to each segment via bottleneck cross-attention → local summary L_i
Global Integration — multi-view statistics (mean/max/min/std/ℓ2-norm) → shared compression → attention-based selection → gated expansion → G
Top-down Broadcast — per-segment attention with augmented KV=[G, L_i, segment tokens]; queries from segment tokens only

Input (48K tokens) → 8 segments × 6K
  Stage 1: 8 local slots per segment → L_i
  Stage 2: multi-view stats → K=4 global slots G
  Stage 3: Q=[chunk], KV=[G, L_i, chunk] → Flash Attention

Trainable Components

adapter_model.safetensors  (27 MB)
└── LoRA Adapters (r=8, alpha=16): q_proj, k_proj, v_proj, o_proj

trainable_params.bin  (~4 GB)
├── global_memory.*            — Local Construction modules (36 layers)
├── hierarchical_aggregator.*  — Global Integration modules (36 layers)
├── self_attn.q_norm / k_norm  — QK-Norm weights (Qwen3-specific, 36 layers)
├── input_layernorm / post_attention_layernorm — LayerNorm weights (36 layers)
├── model.embed_tokens.weight  — Token embeddings
└── model.norm.weight          — Final LayerNorm

Training Details

Base Model: Qwen/Qwen3-8B
Context Length: 49,152 tokens (48K)
Segments: 8 × 6,144 tokens
Local Memory Slots (M): 8 per segment
Global Memory Slots (K): 4
Memory Heads: 8, Bottleneck dim: 512
LoRA: r=8, alpha=16, target: q/k/v/o_proj
Checkpoint: step 500 / 1000
Batch: per_device=1, grad_accum=8 (effective batch=8)
LR: 2e-5 (LoRA), 2e-4 (memory modules), grad clip=0.3
Precision: bf16
Hardware: 8× H200 141GB, DeepSpeed Stage 2

Usage

Requires qwen3_attn_hici.py from this repo.

python

import torch
import transformers
from peft import PeftModel
import qwen3_attn_hici as hici_attn

# 1. Replace attention with HiCI BEFORE loading model
hici_attn.MIXED_GROUP_TRAINING = False
hici_attn.replace_qwen3_attn(use_flash_attn=True, use_full=False, use_hierarchical_forward=True)

# 2. Load base model
base_model = transformers.AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B", torch_dtype=torch.bfloat16, device_map="auto",
)

# 3. Register HiCI modules (must match training config)
hici_attn.register_hici_to_model(base_model, num_memory_slots=8, global_slots=4, num_heads=8, bottleneck_dim=512)

# 4. Load LoRA adapter + trainable_params
model = PeftModel.from_pretrained(base_model, "ZengXiangyu/Qwen3-8b-HiCI-48k-1000steps")

# 5. Tokenizer
tokenizer = transformers.AutoTokenizer.from_pretrained("ZengXiangyu/Qwen3-8b-HiCI-48k-1000steps")

Citation

bibtex

@article{zeng2026hici,
  title={HiCI: Hierarchical Construction-Integration for Long-Context Attention},
  author={Zeng, Xiangyu and Xu, Qi and Wang, Yunke and Xu, Chang},
  journal={arXiv preprint arXiv:2603.20843},
  year={2026}
}

License

Apache 2.0 (follows Qwen3 license)