Llama-Thinker-3B-Preview-GGUF is a pretrained and instruction-tuned generative model designed for multilingual applications. These models are trained using synthetic datasets based on long chains of thought, enabling them to perform complex reasoning tasks effectively.
Model Architecture: [ Based on Llama 3.2 ] is an autoregressive language model that uses an optimized transformer architecture. The tuned versions undergo supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.
Use with transformers
Starting with transformers >= 4.43.0 onward, you can run conversational inference using the Transformers pipeline abstraction or by leveraging the Auto classes with the generate() function.
Make sure to update your transformers installation via pip install --upgrade transformers.
python
1import torch
2from transformers import pipeline
34model_id ="prithivMLmods/Llama-Thinker-3B-Preview"5pipe = pipeline(6"text-generation",7 model=model_id,8 torch_dtype=torch.bfloat16,9 device_map="auto",10)11messages =[12{"role":"system","content":"You are a pirate chatbot who always responds in pirate speak!"},13{"role":"user","content":"Who are you?"},14]15outputs = pipe(16 messages,17 max_new_tokens=256,18)19print(outputs[0]["generated_text"][-1])
Note: You can also find detailed recipes on how to use the model locally, with torch.compile(), assisted generations, quantised and more at huggingface-llama-recipes
Here’s a version tailored for the Llama-Thinker-3B-Preview-GGUF model:
How to Run Llama-Thinker-3B-Preview on Ollama Locally
This guide demonstrates how to run the Llama-Thinker-3B-Preview-GGUF model locally using Ollama. The model is instruction-tuned for multilingual tasks and complex reasoning, making it highly versatile for a wide range of use cases. By the end, you'll be equipped to run this and other open-source models with ease.
Example 1: How to Run the Llama-Thinker-3B-Preview Model
The Llama-Thinker-3B model is a pretrained and instruction-tuned LLM, designed for complex reasoning tasks across multiple languages. In this guide, we'll interact with it locally using Ollama, with support for quantized models.
Step 1: Download the Model
First, download the Llama-Thinker-3B-Preview-GGUF model using the following command:
ollama run llama-thinker-3b-preview.gguf
Step 2: Model Initialization and Download
Once the command is executed, Ollama will initialize and download the necessary model files. You should see output similar to this:
Once the model is fully loaded, you can interact with it by sending prompts. For example, let's ask:
>>> How can you assist me today?
A sample response might look like this [may / maynot be identical]:
plaintext
1I am Llama-Thinker-3B, an advanced AI language model designed to assist with complex reasoning, multilingual tasks, and general-purpose queries. Here are a few things I can help you with:
231. Answering complex questions in multiple languages.
42. Assisting with creative writing, content generation, and problem-solving.
53. Providing detailed summaries and explanations.
64. Translating text across different languages.
75. Generating ideas for personal or professional use.
86. Offering insights on technical topics.
910Feel free to ask me anything you'd like assistance with!
Step 4: Exit the Program
To exit the program, simply type:
/exit
Example 2: Using Multi-Modal Models (Future Use)
In the future, Ollama may support multi-modal models where you can input both text and images for advanced interactions. This section will be updated as new capabilities become available.
Notes on Using Quantized Models
Quantized models like llama-thinker-3b-preview.gguf are optimized for efficient performance on local systems with limited resources. Here are some key points to ensure smooth operation:
VRAM/CPU Requirements: Ensure your system has adequate VRAM or CPU resources to handle model inference.
Model Format: Use the .gguf model format for compatibility with Ollama.
Conclusion
Running the Llama-Thinker-3B-Preview model locally using Ollama provides a powerful way to leverage open-source LLMs for complex reasoning and multilingual tasks. By following this guide, you can explore other models and expand your use cases as new models become available.
