This is quantized version of
fusionbase/fusion-guide-12b-0.1 created using llama.cpp
fusion-guide is an advanced AI reasoning system built on the Mistral-Nemo 12bn architecture. It employs a two-model approach to enhance its problem-solving capabilities. This method involves a "Guide" model that generates a structured, step-by-step plan to solve a given task. This plan is then passed to the primary "Response" model, which uses this guidance to craft an accurate and comprehensive response.
fusion-guide is fine-tuned on a custom dataset consisting of task-based prompts in both English (90%) and German (10%). The tasks vary in complexity, including scenarios designed to be challenging or unsolvable, to enhance the model's ability to handle ambiguous situations. Each training sample follows the structure: prompt => guidance, teaching the model to break down complex tasks systematically.
Read a detailed description and evaluation of the model here:
https://blog.fusionbase.com/ai-research/beyond-cot-how-fusion-guide-elevates-ai-reasoning-with-a-two-model-systemThe prompt must be enclosed within <guidance_prompt>{PROMPT}</guidance_prompt> tags, following the format below:
<guidance_prompt>Count the number of 'r's in the word 'strawberry,' and then write a Python script that checks if an arbitrary word contains the same number of 'r's.</guidance_prompt>
fusion-guide can be used with vLLM and other Mistral-Nemo-compatible inference engines. Below is an example of how to use it with unsloth:
1from unsloth import FastLanguageModel
2
3max_seq_length = 8192 * 1 # Choose any! We auto support RoPE Scaling internally!
4dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
5load_in_4bit = False # Use 4bit quantization to reduce memory usage. Can be False.
6
7model, tokenizer = FastLanguageModel.from_pretrained(
8 model_name="fusionbase/fusion-guide-12b-0.1",
9 max_seq_length=max_seq_length,
10 dtype=dtype,
11 load_in_4bit=load_in_4bit
12)
13
14FastLanguageModel.for_inference(model) # Enable native 2x faster inference
15
16guidance_prompt = """<guidance_prompt>Count the number of 'r's in the word 'strawberry,' and then write a Python script that checks if an arbitrary word contains the same number of 'r's.</guidance_prompt>"""
17messages = [{"role": "user", "content": guidance_prompt}]
18inputs = tokenizer.apply_chat_template(
19 messages,
20 tokenize=True,
21 add_generation_prompt=True, # Must add for generation
22 return_tensors="pt",
23).to("cuda")
24
25outputs = model.generate(input_ids=inputs, max_new_tokens=2000, use_cache=True, early_stopping=True, temperature=0)
26result = tokenizer.batch_decode(outputs)
27
28print(result[0][len(guidance_prompt):].replace("</s>", ""))
The model may occasionally fail to generate complete guidance, especially when the prompt includes specific instructions on how the responses should be structured. This limitation arises from the way the model was trained.
