Llama 3.2 3B Instruct (Q4_K_M GGUF) powers the CorraCAG iOS app (Clinical Operations Recruitment and Retention Assistant, CORRA v2.0.0). The deployment pairs this quantized model with Cache Augmented Generation (CAG), memory-aware inference, and clinical-trial safety filters. Weights are not fine-tuned in this project; CORRA behavior comes from prompting, retrieved knowledge, and post-generation safety layers.

Model Details

Model Description

This artifact is the on-device large language model used by CorraCAG for privacy-preserving, educational Q&A about clinical trials and related medical research topics.

Aspect	Detail
Base model	meta-llama/Llama-3.2-3B-Instruct (3B parameters, instruction-tuned)
Bundled GGUF	`Llama-3.2-3B-Instruct-HF_q4_k_m_20251105_145103.gguf`
Quantization	Q4_K_M (size/speed tradeoff for mobile)
Runtime	llama.cpp with Metal GPU acceleration on Apple devices
Application stack	`ModelManager` → `LlamaCppBridge`; prompts built by `CAGManager`; input/output filtered by `SafetyBoundaries`

Cache Augmented Generation (CAG): At inference time, CAGManager loads knowledge_cache.json from the app bundle, indexes knowledge into ~1000-character chunks (200-character overlap), retrieves relevant chunks via keyword matching, and injects them into the system context before generation.

Safety: SafetyBoundaries validates inputs for clinical-trial relevance, blocks common adversarial/jailbreak patterns, and filters responses that resemble personalized medical advice (English and several other languages).

Developed by: Corra / CorraCAG project (application and deployment); base weights by Meta
Funded by [optional]: [More Information Needed]
Shared by [optional]: [More Information Needed]
Model type: Causal language model (decoder-only), instruction-tuned; deployed as GGUF for text generation
Language(s) (NLP): English (primary); Spanish, French, German, Italian, Portuguese supported in app prompts and safety filters
License: Llama 3.2 Community License (Meta) for base weights; comply with Meta terms when redistributing GGUF
Finetuned from model [optional]: Not fine-tuned in CorraCAG — uses meta-llama/Llama-3.2-3B-Instruct with Q4_K_M quantization only

Model Sources [optional]

Repository: CorraCAG source tree (e.g. ModelManager.swift, CAGManager.swift, SafetyBoundaries.swift, LlamaCppBridge.mm)
Paper [optional]: Llama 3.2 model documentation
Demo [optional]: CorraCAG iOS application (on-device; no public web demo in repository)

Uses

Direct Use

Loading the GGUF with llama.cpp or compatible tools for text generation using the Llama 3.2 chat template. Raw inference without CorraCAG’s CAG layer and safety filters will not match in-app CORRA behavior.

Example (adjust binary path and prompt formatting to your build):

bash

./llama-cli -m Llama-3.2-3B-Instruct-HF_q4_k_m_20251105_145103.gguf \
  --temp 0.7 -n 512 \
  -p "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful assistant..."

Downstream Use [optional]

Primary intended use: Embedded in the CorraCAG iOS app as the generative core for CORRA, a clinical-trials educator persona:

User query → SafetyBoundaries.validateInput
CAGManager.buildCAGPrompt (retrieval + system instructions, including user language)
LlamaCppBridge generation (temperature 0.7, up to 2048 new tokens)
SafetyBoundaries.filterResponse → user

Bundle both the GGUF and knowledge_cache.json in the app target’s Copy Bundle Resources for full CAG behavior.

Out-of-Scope Use

Diagnosis, treatment plans, or personalized medical advice
Emergency or urgent care guidance
Replacing qualified clinicians, investigators, or regulatory professionals
High-stakes clinical or production decisions without organizational validation
Unsupervised use as a medical device or regulated clinical decision support
Off-topic general chat (app redirects to clinical-trial topics)

Bias, Risks, and Limitations

3B scale limits depth and factual reliability vs. larger server models; hallucinations are possible.
Q4_K_M quantization reduces precision vs. FP16/BF16 weights.
CAG is keyword-based, not dense-vector RAG; retrieval quality depends on query wording and chunking.
Safety filters are heuristic (keyword/pattern lists); they do not guarantee complete jailbreak or harmful-output prevention.
Multilingual behavior may be uneven across languages; CAG source text may be mixed-language.
Biases from Llama pretraining and from curated CAG documents may affect answers; audit knowledge_cache.json and document_sources for your audience.
On-device inference improves privacy vs. cloud APIs but requires multi-gigabyte storage and substantial RAM (~4 GB+ available for full configuration).

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases, and limitations above. Treat outputs as educational only. Display clear disclaimers that CORRA is not a substitute for professional medical advice. Validate accuracy, safety refusals, and multilingual behavior on target devices before research or field deployment. Review CAG knowledge sources and update knowledge_cache.json under your organization’s governance policies.

How to Get Started with the Model

In CorraCAG (recommended):

Open the CorraCAG Xcode project and add to Copy Bundle Resources:
- Llama-3.2-3B-Instruct-HF_q4_k_m_20251105_145103.gguf
- knowledge_cache.json
Build for a physical iOS device with sufficient RAM (prefer 4 GB+ available memory for full context/GPU offload).
Run the app; ModelManager loads the model and CAGManager initializes the knowledge cache at startup.

Default generation settings in app code:

Parameter	Value
Temperature	0.7
Max new tokens	2048
Context size (`n_ctx`)	4096 (normal) / 2048 (memory warning) / 1024 (critical)
GPU layers	All (-1) / 20 / 10 by memory pressure

Standalone llama.cpp: See Direct Use above; replicate CORRA system prompt and CAG context manually if matching app behavior.

Training Details

Training Data

This deployment does not include CorraCAG-specific fine-tuning data. The base model was trained by Meta per the Llama 3.2 3B Instruct model card.

Runtime knowledge (not in weights): knowledge_cache.json supplies CAG grounding (knowledge_prompt, knowledge_text, document_sources). Provenance and curation of that cache are organization-specific — document in your internal dataset card if published separately.

Training Procedure

Preprocessing [optional]

GGUF conversion from Hugging Face–format Llama 3.2 3B Instruct weights; Q4_K_M quantization for on-device inference. No additional CorraCAG weight training is performed in this repository.

Training Hyperparameters

Training regime: Not applicable for CorraCAG deployment (inference-only quantized weights). Base model training used Meta’s standard Llama 3.2 3B Instruct procedure.

Speeds, Sizes, Times [optional]

Checkpoint format: GGUF (multi-gigabyte on disk)
Inference: On-device via llama.cpp/Metal; latency depends on device (iPhone/iPad), memory pressure, and prompt length
Memory: MemoryManager reduces context and GPU layers under iOS memory warnings

Evaluation

Testing Data, Factors & Metrics

Testing Data

No formal public benchmark suite is included in the CorraCAG repository for this deployment.

Factors

Recommended internal evaluation factors: device type (iPhone vs. iPad), available RAM / memory pressure tier, query language, clinical-trial topic coverage, off-topic and adversarial prompts, and personal-medical-advice edge cases.

Metrics

Suggested metrics for organizational validation: factual accuracy vs. approved trial materials, refusal rate for out-of-scope queries, harmful-output rate, latency (tokens/sec), and user comprehension (task-specific).

Results

Formal benchmark results are not published in this repository.

Summary

Teams should run domain-specific validation before research or production use. See Recommendations under Bias, Risks, and Limitations.

Model Examination [optional]

No dedicated interpretability or probing study is bundled with CorraCAG. Standard Llama 3.2 documentation applies to the base architecture.

Environmental Impact

On-device inference avoids recurring cloud GPU use per query; one-time download and local inference energy depend on end-user hardware.

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: Apple iPhone / iPad (Metal-accelerated inference)
Hours used: [More Information Needed] (per-deployment)
Cloud Provider: N/A for inference (on-device); optional cloud use only for app distribution or knowledge-cache authoring
Compute Region: [More Information Needed]
Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

Architecture: Llama 3.2 decoder-only transformer, 3B parameters
Objective: Causal language modeling; instruction-tuned for dialogue
CorraCAG overlay: CAG retrieval + safety filtering (no weight updates)

Compute Infrastructure

Hardware

Target: iOS devices with Metal support
RAM: ~4 GB+ available memory recommended for full ModelConfiguration (context 4096, all GPU layers)
Storage: Multi-gigabyte app bundle for GGUF weights

Software

Inference engine: llama.cpp (embedded in CorraCAG)
Bridge: LlamaCppBridge (Objective-C++)
App: SwiftUI, CorraCAG v2.0.0

Citation [optional]

Glossary [optional]

CORRA: Clinical Operations Recruitment and Retention Assistant
CAG: Cache Augmented Generation — keyword-indexed retrieval from knowledge_cache.json injected into prompts at inference time
GGUF: File format for quantized llama.cpp models
Q4_K_M: 4-bit quantization scheme (K-quant, medium variant)

More Information [optional]

App disclaimer (from project README): educational assistant only; not a substitute for professional medical advice, diagnosis, or treatment.
System persona and rules: SafetyBoundaries.getSystemPrompt() and CAGManager.buildCAGPrompt(userQuestion:)
Resource verification scripts: check_bundle_resources.sh, verify_resources.sh

Model Card Authors [optional]

[Ken Wiley]

Model Card Contact

[ken.wiley@nih.gov] — CorraCAG project maintainers

xinj23/Llama-3.2-3B-Instruct-HF_q4_k_m_20251105_145103.gguf

Model Card for CorraCAG / CORRA On-Device LLM