This model is fine-tuned on a textual dataset for analog circuit knowledge learning. The training dataset is constructed from high-quality textbooks using a knowledge distillation approach to extract structured question-answer pairs. The model achieves 85.04% accuracy on the AMSBench-TQA benchmark, showing a 15.67% improvement over the initial Qwen2.5-32B-Instruct model.

Limitations

While this model demonstrates good performance on the AMSBench-TQA benchmark, it is specialized for this domain. Its applicability and performance in other, unrelated domains may be limited. Users should be aware that, like all language models, it may occasionally generate incorrect or nonsensical information, especially for highly novel or unrepresented concepts within its training data.

Sample Usage

You can use this model with the Hugging Face transformers library:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

model_id = "analogllm/analog_model"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Example chat interaction (Qwen2.5 Instruct format)
messages = [
    {"role": "user", "content": "What is the primary function of a common-emitter amplifier in analog circuits?"}
]

# Apply the chat template and prepare inputs
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors='pt').to(model.device)

# Configure generation parameters
generation_config = GenerationConfig(
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.8,
    repetition_penalty=1.05,
    eos_token_id=[tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|im_end|>")] # Ensure it stops correctly
)

# Generate response
outputs = model.generate(
    inputs=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    generation_config=generation_config
)

# Decode and print the response
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-06
train_batch_size: 1
eval_batch_size: 8
seed: 42
distributed_type: multi-GPU
num_devices: 8
gradient_accumulation_steps: 8
total_train_batch_size: 64
total_eval_batch_size: 64
optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.1
num_epochs: 1.0

Training results

json

{
  "epoch": 1.0,
  "num_input_tokens_seen": 113180672,
  "total_flos": 759612479373312.0,
  "train_loss": 1.1406613362056237,
  "train_runtime": 17617.7573,
  "train_samples_per_second": 0.784,
  "train_steps_per_second": 0.012
}

Framework versions

Transformers 4.52.4
Pytorch 2.5.1+cu124
Datasets 3.6.0
Tokenizers 0.21.1

analogllm/analogseeker

A 32B Fine-tuned Model for Analog Circuit Knowledge Learning

Model description