JiRack 10B FP32, INT8, INT4
A fast and efficient coding assistant with a clean, modern built-in web UI.
Powered by Meta Llama 3.1 8B Instruct weights and a fully refactored architecture optimized for a 10B-scale model. The model was specifically designed for high-performance tuning with advanced quantization options.
The next version will be JiRack Ternary 10B — a highly optimized ternary model delivering exceptional speed and efficiency using Microsoft ONNX Runtime.
Setup flag ternary true allow to do IT to convert model to ternary format then use llama.cpp to run GGUF conversation with llama.cpp Ternary support kernel.
- JiRack is a cloud-ready model that helps save money on cloud usage. It can be used as an expert model in RAG deployments on cloud, with the ONNX JiRack Java server as an alternative.
- Subscription: $1 per month per user (updated license for non-company use).
- Corp Subscription: $3 per month per user (updated license for company use).
- It works without subscription but send message about subscription.
JiRack sevice options ONNX or bitnet.cpp engine or llama.cpp
- Current quantizations were done from the FP16 model, but the model allows for more compression thanks to its ternary architecture.
- If you need to do ternary compression, please write to me and I'll perform QAT from your dataset, tailored specifically to your task.
- Adapt train process to avoid catastrophic forgetting with NDA
- Adapt train process to avoid fast plato in training with NDA
- Convert model to bitnet.cpp kernel I2S_0 with support AVX2 and AVX-512 CPU instructions for high performance on CPU
CMS Manhattan RAG & Email Reply + Document and Email Analytics:
https://www.youtube.com/watch?v=KRu2nLEh_6g&t=78s
"So I don’t read my emails — I just ask JiRack to tell me the news!"
Welcome to the CMS Manhattan AI Front Office Solution.
Training the Model
It is easy to train on a Blackwell GPU with 96 GB VRAM.
You do not need a data center for fine-tuning or QLoRA — it works well on consumer-grade GPU cards.
Let me know if you need the training code!
Quick Start
Watch JiRack 10B in action and run it with Docker.
Run with Docker
Default CPU INT8
1docker run -d \
2 --name jirack_10b \
3 -p 7869:7869 \
4 --restart unless-stopped \
5 cmsmanhattan/jirack_10b_int8:latest
Default CPU INT4
1docker run -d \
2 --name jirack_10b \
3 -p 7869:7869 \
4 --restart unless-stopped \
5 cmsmanhattan/jirack_10b_int4:latest
Multi CPU
1docker run -d \
2 --name jirack_10b \
3 -p 7869:7869 \
4 --restart unless-stopped \
5 --memory=20g \
6 --cpus=12 \
7 cmsmanhattan/jirack_10b_int8:latest
GPU
1docker run -d \
2 --name jirack_10b \
3 -p 7869:7869 \
4 --gpus all \
5 --restart unless-stopped \
6 cmsmanhattan/jirack_10b_gpu_int8:latest
Docker Compose Example
1services:
2 jirack:
3 image: cmsmanhattan/jirack_10b_int8:1.0.2
4 container_name: jirack_onnx_service
5 ports:
6 - "7869:7869"
7 volumes:
8 - .:/app
9 - ./web:/app/web
10 environment:
11 - MAX_TOKENS=1024
12 - TEMPERATURE=0.7
13 - TOP_P=0.9
14 - DEFAULT_STREAM=False
15 - INTRA_THREADS=4
16 - USE_ENV_ALLOCATOR=1
17 deploy:
18 resources:
19 limits:
20 memory: 16g
Access the UI
Once the container is running, open your browser and go to:
http://localhost:7869
This opens the JiRack UI — a clean, modern web interface for chat.
Changing the Port
The listening port can be easily changed from the Settings panel inside the JiRack Chat UI.
Licensing
- The JiRack 10B model is provided under a commercial enterprise license.
- All JiRack UI clients are provided under a commercial license.
- UI clients can be used for free when running with the official JiRack Docker containers, as long as they are not redistributed separately.
Subscription Plans
- JiRack Enterprise: $36 per user per year
- JiRack Private: $12 per user per year
For commercial licensing, cluster deployment, performance tuning, or enterprise use, please contact us.
Hardware Recommendations for AMD Systems
Recommended Hardware for JiRack 10B INT8 (single Docker container)
| Use Case | CPU | GPU (ROCm) | VRAM / RAM | Expected Speed | Recommendation |
|---|
| Recommended | Ryzen 7 7700 / 9700X | RX 7900 XTX / 7900 XT | 24GB VRAM | 50-75 tokens/s | Best choice |
| High Performance | Ryzen 9 7950X / 9950X | RX 7900 XTX | 24GB+ VRAM | 65-90 tokens/s | Excellent |
| Enterprise | EPYC 7003/9004 series | MI300X or 2x RX 7900 XTX | 48GB+ VRAM | 90-140 tokens/s | For 32B model |
| Budget Option | Ryzen 5 7600 / 9600X | RX 7800 XT (16GB) | 16GB VRAM | 35-50 tokens/s | Acceptable |
Important Memory Notes
Even though the 10B INT8 model itself takes approximately 8–9 GB, we recommend at least 24GB VRAM for:
- KV-cache consumption during generation, especially with long context
- ONNX Runtime overhead and temporary buffers
- System stability and avoiding out-of-memory errors
- Support for larger context windows
Minimum recommended: 24GB VRAM (RX 7900 series)
Ideal: 24–32GB VRAM
For pure CPU inference, we recommend at least 64GB system RAM.
I added the default model in full FP32 precision (~62 GB). This serves as the base for quantization to find the best balance between size and performance.
📧 Contact & Licensing
For joint ventures, hardware integration, or licensing inquiries:
- Email: grabko@cmsmanhattan.com
- Phone: +1 (516) 777-0945
- Location: New York, USA
