This repository contains GGUF quantized versions of the palmyra-mini-thinking-a model, based on the Qwen2 architecture. This model is specifically designed for reasoning tasks with explicit thinking capabilities through special <think> and </think> tokens. GGUF quantizations are optimized for efficient inference across various hardware platforms using llama.cpp and compatible frameworks.
Available Quantizations
BF16 (Brain Float 16)
File: palmyra-mini-thinking-a-BF16.gguf
Size: 3.3GB
Precision: 16-bit brain float
Use Case: Highest quality reasoning, requires more memory
Q8_0 (8-bit Quantization)
File: palmyra-mini-thinking-a-Q8_0.gguf
Size: 1.8GB
Precision: 8-bit integer
Use Case: Good balance of reasoning quality and efficiency
Quick Start
Installation
bash
1# Install llama.cpp2git clone https://github.com/ggerganov/llama.cpp
3cd llama.cpp
4make56# Or use a pre-built binary
Usage
bash
1# Run with thinking prompt2./main -m /path/to/palmyra-mini-thinking-a-BF16.gguf \3 -p "A rectangle has a length of 12 cm and width of 8 cm. What is its area and perimeter?<｜Assistant｜><think>"\4 -n 51256# Interactive mode7./main -m /path/to/palmyra-mini-thinking-a-Q8_0.gguf -i
LM Studio Use
Steps to download a model through the Discover tab can be found here
Q8_0: ~45% smaller size, faster inference with preserved reasoning capabilities
Training Details
Tokenizer
Type: LlamaTokenizerFast with 151,665 vocabulary size
Special Tokens:
BOS Token ID: 151646 ()
EOS Token ID: 151643 ()
Pad Token ID: 151643 ()
Think Start: 151648 (<think>)
Think End: 151649 (</think>)
Model Configuration
Hidden Activation: SiLU (Swish)
Normalization: RMSNorm (ε = 1e-06)
Initializer Range: 0.02
Attention Dropout: 0.0
Chat Template
The model uses a specialized chat template for reasoning:
User messages:
Assistant messages:
Thinking mode: Automatically initiated with <think> tokens
Tool calling support
Usage Examples
Reasoning Task
bash
1./main -m palmyra-mini-thinking-a-Q8_0.gguf \2 -p "A rectangle has a length of 12 cm and width of 8 cm. What is its area and perimeter?<｜Assistant｜><think>"\3 -n 300\4 --temp 0.7
Problem Solving
bash
1./main -m palmyra-mini-thinking-a-BF16.gguf \2 -p "Explain the water cycle step by step.<｜Assistant｜><think>"\3 -n 400\4 --temp 0.8\5 --top-p 0.9
Known Limitations
Context Length: Default context is 4,096 tokens, though the model supports up to 131,072
Thinking Overhead: Explicit thinking increases response length and generation time
Quantization Trade-offs: Lower bit quantizations may affect reasoning quality
Platform Optimization: Performance varies across different hardware configurations
Compatibility
llama.cpp: Compatible with recent versions
Frameworks: llama.cpp, Ollama, LM Studio, GPT4All, and other GGUF-compatible tools
Platforms: Windows, macOS, Linux (x86_64, ARM64)
Special Features: Requires framework support for thinking tokens
License
Apache 2.0
Original model card: palmyra-mini-thinking-a
Model Details
Model Name: palmyra-mini-thinking-a
Version: 1.0
Type: Generative AI Language Model
Model Description
The palmyra-mini-thinking-a model demonstrates exceptional performance in advanced mathematical reasoning and competitive programming. Its capabilities are highlighted by an outstanding score of 0.886 on the 'MATH500' benchmark, showcasing a robust ability to solve complex mathematical problems. The strength of the model in quantitative challenges is further confirmed by its score of 0.8287 on 'gsm8k (strict-match)', which demonstrates proficiency in multi-step arithmetic reasoning. Additionally, the model proves its aptitude for high-level problem-solving with a score of 0.8 on 'AMC23'. The model also shows strong potential in the coding domain, achieving a score of 0.5631 on 'Codeforces (pass_rate)' and 0.5481 on 'Olympiadbench (extractive_match)', indicating competence in generating correct solutions for programming challenges.
Benchmark Performance
This section provides a detailed breakdown of the palmyra-mini-thinking-a model's performance across a standardized set of industry benchmarks. The data is presented in its original order from the source evaluation.
This model is intended for research and development in the field of generative AI, particularly for tasks requiring mathematical and logical reasoning.
Limitations
The model's performance has been evaluated on a specific set of benchmarks. Its performance on other tasks or in real-world applications may vary.
Ethical Considerations
As with any language model, there is a potential for generating biased or inaccurate information. Users should be aware of these limitations and use the model responsibly.
