You can deploy this model and start earning money today!

ft-llm-team-mkj/
baseline-model-instruct-32k-yarn

Name: baseline-model-instruct-32k-yarn
Availability: InStock

ft-llm-team-mkj/baseline-model-instruct

finetune

other

Views

No views yet

baseline-model-instruct-32k-yarn

This model is ft-llm-team-mkj/baseline-model-instruct with YaRN (Yet another RoPE extensioN) applied to extend the context window.

Model Details

Base Model: ft-llm-team-mkj/baseline-model-instruct
Original Context Length: 4096 tokens
Extended Context Length: 32768 tokens
Extension Method: YaRN
Scaling Factor: 8.0

YaRN Configuration

json

{
  "rope_type": "yarn",
  "factor": 8.0,
  "original_max_position_embeddings": 4096,
  "beta_fast": 32,
  "beta_slow": 1
}

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "DeL-TaiseiOzaki/baseline-model-instruct-32k-yarn",
    torch_dtype="bfloat16",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("DeL-TaiseiOzaki/baseline-model-instruct-32k-yarn")

Note

This model has YaRN applied but has not been fine-tuned on long-context data yet. For optimal performance on long sequences, fine-tuning on long documents is recommended.

ft-llm-team-mkj/baseline-model-instruct-32k-yarn