You can deploy this model and start earning money today!

NorthernTribe-Research/
UMSR-Reasoner-7B-NF4

Name: UMSR-Reasoner-7B-NF4
Availability: InStock

NorthernTribe-Research/UMSR-Reasoner-7B

finetune

endpoints_compatible

Views

No views yet

UMSR-Reasoner-7B (NF4)

4-bit NF4 bitsandbytes runtime quantization profile.

Model Tree

Source model: https://huggingface.co/NorthernTribe-Research/UMSR-Reasoner-7B
Variant repo: https://huggingface.co/NorthernTribe-Research/UMSR-Reasoner-7B-NF4
Dataset: https://huggingface.co/datasets/NorthernTribe-Research/UMSR-v1

Quantization Profile

Variant: nf4
Scheme: bitsandbytes nf4 (4-bit)
Materialized weights in this repo: False

Usage

This variant repository is a profile card. Load weights from the source model id.

python

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

model_id = "NorthernTribe-Research/UMSR-Reasoner-7B"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    quantization_config=bnb_config,
)

Output Contract

<final_answer>...</final_answer>

NorthernTribe-Research/UMSR-Reasoner-7B-NF4