SandLogicTechnologies/

Phi-3-medium-128k-instruct-GGUF

Phi-3-Medium-128K-Instruct

Model Overview

• Model Name: Phi-3-Medium-128K-Instruct
• Base Model: microsoft/Phi-3-medium-128k-instruct
• Architecture: Decoder-only Transformer
• Parameter Count: 14 Billion
• Context Length: 128K tokens
• Modalities: Text
• Developer: Microsoft
• License: MIT License

Model Variants

Instruction-Tuned Version

• Optimized for conversational and task-oriented prompts
• Improved adherence to structured instructions
• Enhanced reasoning and step-by-step explanation capabilities
• Designed for safe and helpful outputs

Quantization Details

Q4_K_M

• Approx. ~71% size reduction
• Very low memory footprint (~7.98 GB)
• Optimized for CPU inference and low-VRAM GPUs
• Faster token generation speeds
• Minor degradation in complex analytical or long-chain reasoning tasks

Q5_K_M

• Approx. ~66% size reduction
• Better fidelity to the original FP16 model (9.38 GB)
• Improved coherence and reasoning consistency
• Recommended when slightly more memory is available

Training Background

Pretraining

• Trained on a curated mixture of high-quality text sources
• Focused on reasoning, coding, mathematics, and general knowledge
• Optimized using autoregressive next-token prediction

Instruction Tuning

• Fine-tuned on instruction-following datasets
• Enhanced performance on question answering, summarization, coding, and dialogue
• Improved alignment for clarity, relevance, and structured outputs

Key Capabilities

• Long-context understanding
  Supports up to 128K tokens, enabling large document processing and persistent conversational memory.
• Advanced reasoning
  Performs multi-step logical reasoning, analytical problem solving, and structured explanations.
• Code generation and analysis
  Capable of generating, debugging, and explaining code across multiple programming languages.
• Conversational AI
  Maintains coherence across extended multi-turn interactions.
• Document summarization and extraction
  Handles long reports, contracts, and research papers effectively.

Usage Example

llama.cpp

./llama-cli \
  -m SandlogicTechnologies\Phi-3-mini-4k-instruct_Q4_K_M.gguf \
  -p "Explain transformers in simple terms."

Recommended Applications

• Enterprise document processing :- Analyze long-form documents, compliance materials, and technical manuals.
• Research and experimentation :- Evaluate reasoning performance and long-context capabilities.
• Code assistance tools :- Integrate into development environments for coding support.
• Conversational AI systems :- Deploy in chatbots requiring extended context memory.
• Educational tools :- Generate structured explanations and step-by-step solutions.

Deployment Considerations

• Requires sufficient GPU memory for optimal performance
• Mixed precision (FP16/BF16) recommended for efficiency
• Suitable for distributed or high-memory inference setups
• Ensure adherence to the MIT license terms when deploying commercially

Acknowledgments

• The Microsoft team for developing and releasing the Phi-3-mini-4k-instruct model.
• Georgi Gerganov and the entire llama.cpp open-source community for enabling efficient model quantization and inference via the GGUF format.

Contact