Common hyperparameters: β=4.0, distill T=2.0, forget_retain_ratio=1:2, lr=1e-2 cosine, 3 epochs, bf16, init_logit_std=0.1, use_chat_template=True, enable_thinking=False.

Gate Statistics

Metric	Value
Total gates	172,032 (28 layers × 6144 intermediate)
Mean sigmoid gate	≈ 0.501
Std	≈ 0.030
Min / Max	0.321 / 0.672

Selected thresholds (chat_energy)

Threshold	Off-fraction
0.451	4.9%
0.482	25.4%
0.501	49.5%
0.521	75.1%
0.551	95.0%

Per-variant thresholds are in each folder's thresholds.txt / gate_stats.json.

Contents per variant

<variant>/
  lambda_logits.pt     # 172,032 per-neuron logits
  neuron_indices.json  # Knowledge neurons at threshold 0.5
  gate_stats.json      # Statistics + selected thresholds
  thresholds.txt       # Comma-separated thresholds

Usage

python

import torch, json
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-1.7B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-1.7B", torch_dtype=torch.bfloat16)

messages = [{"role": "user", "content": "What is 2+2?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)

gate_state = torch.load("chat_energy_optA/lambda_logits.pt", map_location="cpu")

See the 0.6B chat README for the complete gating recipe and scale-mode notes.

hyunseoki/qwen3-1.7b-lambda-gates-chat

Qwen3-1.7B Lambda Gates — Chat-Template Variants

Variants

Gate Statistics

Selected thresholds (chat_energy)

Contents per variant

Usage

Related Checkpoints

hyunseoki/
qwen3-1.7b-lambda-gates-chat