Quantized GGUF version of Orbi-1B, a fine-tuned TinyLlama-1.1B-Chat specialized for function calling and robotic assistant interactions. This model generates structured tool calls in response to natural language commands and is optimized for CPU inference with llama.cpp.

Model Description

Base Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0
Model Size: 1.1B parameters
Format: GGUF (llama.cpp compatible)
Quantization: Q4_K_M (4-bit quantization)
Optimized for: CPU inference, low memory usage
License: Apache 2.0

Why GGUF?

GGUF (GPT-Generated Unified Format) offers several advantages:

Faster CPU Inference: Optimized for running on CPU without GPU
Lower Memory Usage: 4-bit quantization reduces model size by ~75%
Cross-Platform: Works on Windows, Linux, macOS (including Apple Silicon)
No GPU Required: Perfect for edge devices and embedded systems
Efficient: Powered by llama.cpp's optimized C++ inference engine

File Information

File	Quant	Size	Use Case
orbi-1b-q4.gguf	Q4_K_M	~650MB	Recommended - Best balance of speed and quality

Installation

Requirements

pip install llama-cpp-python

For GPU acceleration (optional):

bash

# CUDA
CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python --force-reinstall --no-cache-dir

# Metal (macOS)
CMAKE_ARGS="-DLLAMA_METAL=on" pip install llama-cpp-python --force-reinstall --no-cache-dir

Usage

Basic Inference

python

import json
import re
from llama_cpp import Llama

# Load model
llm = Llama(
    model_path="orbi-1b-q4.gguf",
    n_ctx=4096,
    n_threads=8,
    use_mlock=True,
    verbose=False
)

# System prompt
system_prompt = """You are Orbi's brain.
Respond with one or more <tool_call> JSON blocks, in the exact order the user requests actions.
that calls the best tool for the user's request. Do not write stories yourself.
Do not summarize news yourself. Map synonyms to the tool argument enums.
If parameters are missing, pick sensible defaults. Keep outputs terse.

Available tools and enums:
- smile() -> {}
- cry() -> {}
- move_hands(direction ∈ {left,right,up,down,wave}, speed ∈ {slow,normal,fast})
- dance(style ∈ {hiphop,ballet,robot,random}, duration_sec ∈ [10..120])
- tell_news(topic: string)
- tell_story(topic: string, tone ∈ {wholesome,funny,dramatic,spooky,random}, length ∈ {short,medium,long})
"""

# Build prompt
user_input = "Wave your hands quickly and smile"
prompt = f"<|system|>\n{system_prompt}\n<|user|>\n{user_input}\n<|assistant|>\n"

# Generate
output = llm(
    prompt,
    max_tokens=256,
    temperature=0.0,
)

response = output["choices"][0]["text"]
print(response)

Interactive Controller

Save this as orbi_controller.py:

python

import json
import re
from llama_cpp import Llama

# Load model
llm = Llama(
    model_path="orbi-1b-q4.gguf",
    n_ctx=4096,
    n_threads=8,
)

def parse_tool_calls(text):
    pattern = r"<tool_call>\s*(\{.*?\})\s*</tool_call>"
    matches = re.findall(pattern, text, re.DOTALL)
    tools = []
    for m in matches:
        try:
            tools.append(json.loads(m))
        except:
            continue
    return tools

# Interactive loop
print("🤖 Orbi is ready! Type 'exit' to quit.\n")
while True:
    user_input = input("You: ").strip()
    if user_input.lower() in {"exit", "quit"}:
        break
    
    prompt = f"<|system|>\n{system_prompt}\n<|user|>\n{user_input}\n<|assistant|>\n"
    output = llm(prompt, max_tokens=256, temperature=0.0)
    response = output["choices"][0]["text"]
    
    tools = parse_tool_calls(response)
    print(f"Orbi: {json.dumps(tools, indent=2)}\n")

Expected Output Format

<tool_call>
{"name": "move_hands", "arguments": {"direction": "wave", "speed": "fast"}}
</tool_call>
<tool_call>
{"name": "smile", "arguments": {}}
</tool_call>

Performance Benchmarks

Approximate inference speeds on different hardware:

Hardware	Tokens/sec	Memory Usage
M1 MacBook Pro	~45 t/s	800MB
Intel i7-12700K	~35 t/s	750MB
Raspberry Pi 5	~8 t/s	700MB
AMD Ryzen 7 5800X	~40 t/s	750MB

Note: Actual performance may vary based on context length and system configuration.

Supported Tools

Physical Actions: smile(), cry(), move_hands(), dance()
Content Generation: tell_news(), tell_story()
Information: whats_your_name(), who_am_i()
Utilities: answer_arithmetic(), english_learning()

Configuration Options

llama-cpp-python Parameters

python

llm = Llama(
    model_path="orbi-1b-q4.gguf",
    n_ctx=4096,              # Context window size
    n_threads=8,             # CPU threads (adjust based on your CPU)
    n_gpu_layers=0,          # Set > 0 for GPU offloading
    use_mlock=True,          # Lock model in RAM (prevents swapping)
    verbose=False,           # Disable verbose logging
    seed=42,                 # Set seed for reproducibility
)

Generation Parameters

python

output = llm(
    prompt,
    max_tokens=256,          # Maximum tokens to generate
    temperature=0.0,         # 0.0 = greedy (recommended for tool calling)
    top_p=0.95,             # Nucleus sampling
    repeat_penalty=1.1,      # Penalize repetition
    stop=["</tool_call>"],   # Stop sequences
)

Use Cases

Robotics: Control physical robots with natural language
IoT Devices: Run on Raspberry Pi or similar edge devices
Embedded Systems: Low-memory environments
Offline Applications: No internet connection required
Desktop Assistants: CPU-only machines without GPU

Limitations

Quantization may result in slight quality degradation compared to full precision
Best performance with greedy decoding (temperature=0.0)
Limited to the predefined set of tools
Context window is 4096 tokens (inherited from base model)

Model Details

Training

Method: LoRA fine-tuning on TinyLlama-1.1B-Chat
Dataset: Custom conversational dataset with tool calling examples
Framework: Transformers + PEFT + TRL

Quantization

Method: Q4_K_M quantization via llama.cpp
Benefits: ~75% size reduction with minimal quality loss
Original Size: ~2.2GB → GGUF Size: ~650MB

Troubleshooting

Model loads slowly

Enable use_mlock=True to keep model in RAM
Increase n_threads based on your CPU cores

Out of memory

Reduce n_ctx (context window size)
Close other applications
Use a lower quantization (Q2_K or Q3_K_M)

Slow inference

Increase n_threads to match your CPU cores
Enable GPU offloading with n_gpu_layers
Reduce max_tokens if generating long responses

License

Apache 2.0 (inherited from TinyLlama base model)

Citation

bibtex

@misc{orbi-1b-gguf,
  title={Orbi-1B GGUF: Quantized Function Calling Model},
  author={Arojit Ghosh},
  year={2025},
  howpublished={\url{https://huggingface.co/Arojit/orbi-1b-gguf}}
}

Related Models

Full Precision: Arojit/orbi-1b
Base Model: TinyLlama/TinyLlama-1.1B-Chat-v1.0

Acknowledgments

Built on TinyLlama
Quantized using llama.cpp
Powered by llama-cpp-python