A Q4_K_M GGUF of Gemma 4 E2B, fine-tuned on Apple Silicon (M3 Pro, 18 GB) using MLX LoRA to act as an autonomous AIOps Orchestrator Agent.

The model evaluates multi-domain infrastructure telemetry and outputs strict, execution-ready JSON commands across: Kubernetes · Nutanix · VMware ESXi · Active Directory · ADFS · PKI · Windows Server

Cross-platform inference: Trained on macOS (Apple Silicon / Metal), but the GGUF runs on macOS, Linux, and Windows via Ollama or llama.cpp (CUDA, ROCm, Vulkan, or CPU).

Model Details

Property	Value
Base model	`google/gemma-4-E2B-it`
Fine-tuning method	LoRA (MLX, 16 layers)
Training iterations	600
Trainable parameters	6.816M (0.147% of 4.63B)
Peak memory (training)	~11.3 GB on M3 Pro 18 GB
Val loss (start → end)	3.665 → 0.097
Quantization	Q4_K_M (llama.cpp)
FP16 size	9.3 GB
Q4_K_M size	3.2 GB (5.88 BPW)
Hardware	Apple M3 Pro, 18 GB Unified Memory
Training time	~5 minutes at ~2.4 it/sec

Quick Start

Ollama (recommended) — macOS · Linux · Windows

bash

# Pull and run directly from this repo
ollama run hf.co/htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

Custom Ollama Agent (with system prompt)

dockerfile

# Modelfile
FROM hf.co/htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

SYSTEM """You are an autonomous AIOps orchestrator agent. Evaluate
telemetry across Kubernetes, Nutanix, VMware, Active Directory, ADFS, and PKI
infrastructure. Output strict, execution-ready JSON commands."""

PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER top_k 40
PARAMETER repeat_penalty 1.1

bash

ollama create aiops-orchestrator -f Modelfile
ollama run aiops-orchestrator

llama.cpp — macOS · Linux · Windows

bash

llama-cli \
  -m gemma4-e2b-aiops-Q4_K_M.gguf \
  --temp 0.3 \
  -p "[AIOps-Agent] Node k8s-worker-03 status is NotReady. AD service account 'svc_k8s_cluster' authentication failed on ADFS."

vLLM (safetensors format)

A HuggingFace safetensors version is available at htunn/gemma-4-e2b-aiops-hf for use with vLLM or the Transformers library:

vllm serve htunn/gemma-4-e2b-aiops-hf --dtype bfloat16

python

from transformers import AutoTokenizer, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("htunn/gemma-4-e2b-aiops-hf")

Example Output

Prompt:

[AIOps-Agent] Node k8s-worker-03 status is NotReady.
Active Directory service account 'svc_k8s_cluster' authentication failed on ADFS.

Response:

json

{
  "action": "remediate_auth",
  "target_domain": "ADFS",
  "service_account": "svc_k8s_cluster",
  "steps": ["check_pki_cert_validity", "rotate_secret_k8s"],
  "api_call": "POST /api/v1/auth/refresh"
}

Prompt:

[AIOps-Agent] PKI certificate for win-srv-2019-app01 expires in 48 hours.
Certificate Authority 'DC-CA-ROOT' is unreachable via LDAP.

Response:

json

{
  "action": "renew_certificate",
  "target_host": "win-srv-2019-app01",
  "ca_authority": "DC-CA-ROOT",
  "steps": ["verify_ca_health", "request_cert_renewal", "deploy_via_gpo"],
  "priority": "high"
}

Prompt:

[AIOps-Agent] Nutanix VM 'win-dc-01' CPU usage exceeds 95%.
VMware host ESXi-02 reports datastore latency.

Response:

json

{
  "action": "diagnose_hypervisor",
  "target_vm": "win-dc-01",
  "infrastructure": ["nutanix", "vmware"],
  "recommendation": "Live-migrate VM via Prism API to secondary host ESXi-04."
}

Training Data Format

Each training example follows the Gemma 4 chat template:

json

{
  "text": "<bos><start_of_turn>user\n[AIOps-Agent] <telemetry_event><end_of_turn>\n<start_of_turn>model\n{\"action\": \"...\", \"steps\": [...]}<end_of_turn>"
}

Scenarios cover:

Kubernetes node failures with AD/ADFS auth issues
Nutanix VM resource exhaustion and live migration
VMware ESXi host maintenance and cluster recovery
PKI certificate expiry and CA unreachability
Windows Server domain controller incidents
Cross-domain cascading failures

Reproduce Fine-Tuning

Source repository: htunn/aiops-gemma4

bash

git clone https://github.com/Htunn/aiops-gemma4.git
cd aiops-gemma4

# Create Python 3.12 venv and install deps
make setup

# Fine-tune (requires HF access to google/gemma-4-E2B-it)
make train

# Fuse + convert to GGUF Q4_K_M
make quantize

# Upload GGUF to HF Hub (Ollama / llama.cpp)
make upload

# Upload safetensors to HF Hub (vLLM / Transformers)
make upload-hf

# Create Ollama model
make ollama

Known mlx-lm Compatibility Notes

Use --num-layers (not --lora-layers, removed in mlx-lm ≥ 0.28)
mlx_lm.fuse no longer needs --export-hf (HF format is now default)
Build llama.cpp with cmake: cmake -B build -DGGML_METAL=ON && cmake --build build --target llama-quantize
Pin numpy>=2.0, scipy>=1.13, transformers>=5.0 after installing llama.cpp requirements

Hardware Requirements

Task	Memory	Platform
Fine-tuning (MLX LoRA)	~11 GB	macOS Apple Silicon
Inference Q4_K_M (Ollama / llama.cpp)	~3.5 GB	macOS · Linux · Windows
Inference FP16 (vLLM / Transformers)	~9.5 GB	Linux (CUDA recommended)

License

Apache 2.0 — see LICENSE.

Base model (google/gemma-4-E2B-it) is subject to the Gemma Terms of Use.