This model is a decoder‑only transformer (Llama family) fine‑tuned with PEFT/LoRA on top of unsloth/Llama-3.3-70B-Instruct-bnb-4bit. It uses Meta’s Llama 3 chat format (<|begin_of_text|>…<|eot_id|>) and maintains the 128K long‑context setting (RoPE scaling with θ≈500k). Architecture hyperparameters are inherited from the Llama 3.x/70B family (80 layers, hidden size 8192, 64 attention heads, 8 KV heads; FFN dim ~28,672). (Hugging Face, Llama, ar5iv)

Tokenizer & special tokens (relevant to training/inference):

BOS: <|begin_of_text|>
End‑of‑turn: <|eot_id|> (recommended stop token for chat)
PAD: <|finetune_right_pad_id|> (Unsloth convention to avoid EOS-as-PAD pitfalls) (Llama, GitHub)

Quantization: bitsandbytes NF4 (with double‑quant), BF16 compute dtype; standard QLoRA‑style settings. (Hugging Face, arXiv)

Developed by: [More Information Needed]
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Model type: Autoregressive decoder‑only LM (Llama 3.3‑70B Instruct), PEFT/LoRA fine‑tune, 4‑bit NF4 quantized.
Language(s) (NLP): English, German, French, Italian, Portuguese, Hindi, Spanish, Thai. (Hugging Face)
License: Llama 3.3 Community License (note commercial terms incl. ≥700M MAU clause). (Hugging Face)
Finetuned from model [optional]: unsloth/Llama-3.3-70B-Instruct-bnb-4bit. (Hugging Face)

Model Sources [optional]

Repository (this fine‑tune): [More Information Needed]
Upstream model cards:
- Meta: meta-llama/Llama-3.3-70B-Instruct. (Hugging Face)
- Unsloth 4‑bit: unsloth/Llama-3.3-70B-Instruct-bnb-4bit. (Hugging Face)
License text: Llama 3.3 Community License. (Hugging Face)
Background papers/guides: Llama 3 (specs), QLoRA/NF4, PEFT/LoRA. (ar5iv, arXiv, Hugging Face)

Uses

Direct Use

Multilingual assistant/chat, summarization, content drafting, retrieval‑augmented generation (RAG) with long documents (up to ~128K tokens), and tool‑calling via the Transformers chat template. Always format prompts with the Llama 3 chat template and stop at <|eot_id|>. (Hugging Face)

Downstream Use [optional]

Domain‑adapted copilots, structured‑output agents, multi‑turn task assistants, and long‑context analytics (meeting minutes, codebases, legal/biomed corpora), leveraging LoRA for lightweight continual updates. (Hugging Face)

Out‑of‑Scope Use

Any use violating Meta’s Acceptable Use Policy or the Llama 3.3 license; unsafe, high‑risk decisions without human oversight; unsupported languages beyond those listed (unless you add targeted fine‑tuning and safety evaluations). (Hugging Face)

Bias, Risks, and Limitations

Base‑model limitations apply: potential for hallucinations, bias, and unsafe content; long‑context retrieval can still miss or over‑weight segments. (Hugging Face)
Quantization artifacts: 4‑bit loading reduces memory but can slightly affect quality; use NF4/double‑quant & BF16 compute to mitigate. (Hugging Face)
Chat formatting: Using EOS instead of <|eot_id|> or mis‑setting PAD to EOS can cause run‑on or truncated generations. Use the official template and a dedicated PAD (e.g., <|finetune_right_pad_id|>). (Llama, Hugging Face Forums, GitHub)

Recommendations

Run task‑specific evaluations before deployment; add system‑level safeguards (e.g., Llama Guard/Prompt Guard) and monitor outputs in production. (Hugging Face)

How to Get Started with the Model

Option A — Full merged model (if you pushed a full model repo)

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

model_id = "vhdrostami/sciloop-neuro-motor-70b"  # this fine-tune repo if weights are merged
bnb = BitsAndBytesConfig(load_in_4bit=True,
                         bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype=torch.bfloat16)

tok = AutoTokenizer.from_pretrained(model_id, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    quantization_config=bnb,
    attn_implementation="flash_attention_2",  # if available
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user",   "content": "Summarize the attached report in 5 bullets."},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True,
                                 return_tensors="pt").to(model.device)
eot_id = tok.convert_tokens_to_ids("<|eot_id|>")
out = model.generate(inputs, max_new_tokens=512, eos_token_id=eot_id)
print(tok.decode(out[0], skip_special_tokens=True))

Option B — PEFT/LoRA adapters (if you’re sharing adapters only)

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

base = "unsloth/Llama-3.3-70B-Instruct-bnb-4bit"
adapter = "YOUR-ORG/YOUR-ADAPTER"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype=torch.bfloat16)

tok = AutoTokenizer.from_pretrained(base, use_fast=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base, device_map="auto", torch_dtype=torch.bfloat16,
    quantization_config=bnb, attn_implementation="flash_attention_2",
)
model = PeftModel.from_pretrained(base_model, adapter)

(Official chat template & tool‑use are documented in the Llama 3.3 card; Flash‑Attention 2 is an optional speedup supported by Transformers.) (Hugging Face)

Preprocessing [optional]

Use the Llama 3 chat template and terminate each message with <|eot_id|>. Keep PAD as a dedicated token (e.g., <|finetune_right_pad_id|>) rather than EOS. (Llama, GitHub)

Summary

Evaluate with lm-eval or on HF Evaluation on the Hub; ensure the chat template is applied and stop at <|eot_id|> to avoid over‑generation bias. (GitHub)

Model Architecture and Objective

Decoder‑only transformer with GQA (8 KV heads), RoPE positional embeddings (θ≈500k), SwiGLU MLP, RMSNorm, and 128K long‑context scaling. Key 70B specs: 80 layers, 8192 hidden size, 64 attention heads, 28,672 FFN dim, vocab ~128k. (ar5iv)

Compute Infrastructure

Hardware

[More Information Needed]

Software

Transformers ≥ 4.45 for official Llama 3.3 support; your config shows 4.51.3.
bitsandbytes (NF4/double‑quant), Flash‑Attention 2 optional, PEFT for LoRA adapters. (Hugging Face)

Citation [optional]

BibTeX (Llama 3 family):

bibtex

@article{meta2024llama3,
  title={The Llama 3 Herd of Models},
  author={Meta AI},
  journal={arXiv preprint arXiv:2407.21783},
  year={2024}
}

(ar5iv)

BibTeX (QLoRA):

bibtex

@article{dettmers2023qlora,
  title={QLoRA: Efficient Finetuning of Quantized LLMs},
  author={Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
  journal={arXiv preprint arXiv:2305.14314},
  year={2023}
}

(arXiv)

Glossary [optional]

GQA (Grouped‑Query Attention): Reduces KV cache size and improves long‑context scalability. (Hugging Face)
RoPE: Rotary positional embeddings; Llama 3.x uses θ≈500k and scaling to reach 128K context. (ar5iv)
NF4: 4‑bit NormalFloat quantization introduced by QLoRA for high‑fidelity low‑memory finetuning. (arXiv)

More Information [optional]

Meta Llama 3.3 model card (usage, tool‑calling, chat template), and Unsloth’s 4‑bit release page for the same model. (Hugging Face)

Model Card Authors [optional]

Felix Schmitt, Vahid Rostami

Framework versions

PEFT 0.15.2
Transformers 4.51.3 (as recorded in config.json)
bitsandbytes (NF4/double‑quant)

Appendix: Configuration snapshot (from your `config.json`)

architectures=["LlamaForCausalLM"], model_type="llama"
num_hidden_layers=80, hidden_size=8192, num_attention_heads=64, num_key_value_heads=8, intermediate_size=28672, head_dim=128
max_position_embeddings=131072 (≈128K), rope_theta=500000.0, rope_scaling={"type":"llama3","factor":8,...}
bos_token_id=128000, eos_token_id=[128001,128008,128009], pad_token_id=128004
quantization_config: load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype="bfloat16"

For correctness and best behavior, format prompts using the Llama 3 chat template and stop generation on <|eot_id|>. (Llama)

vhdrostami/sciloop-neuro-motor-70b

Model Details

Model Description