You can deploy this model and start earning money today!

sdasdawq234/
H-LLM-LOCAL

Name: H-LLM-LOCAL
Availability: InStock

safetensors

gguf

Views

No views yet

LLM Stack — Gemma 4 E4B on CPU, Air-Gapped

Fully-containerized llama.cpp + Open WebUI deployment for an air-gapped, CPU-only ESXi VM, serving Persian-primary chat to multiple concurrent users.

Nothing on the host except Docker + systemd. All tuning is via a single .env file — same image, same compose, scales from 2 cores / 4 GB up to 64+ cores / 256+ GB by switching profiles.

Files at a glance

Path	Purpose
`docker-compose.yml`	Parametric stack — see § Services
`Makefile`	Lifecycle shortcuts; `make help` for the menu
`.env.example`	Every env knob, documented inline with rationale
`.env.profile.{small,medium,large,xlarge}`	Pre-tuned hardware profiles
`models/`	GGUF weights (LFS-tracked)
`models/embed/bge-m3/`	RAG dense embedder weights (LFS-tracked)
`models/rerank/bge-reranker-v2-m3/`	RAG reranker weights (LFS-tracked)
`models/SHA256SUMS`	Integrity manifest for all weights
`images/`	Saved Docker image tarballs (LFS-tracked)
`images/DIGESTS.lock`	Pinned sha256 digests (committed)
`scripts/`	Build, download, load, tune, smoke, bench, package
`scripts/fixtures/`	`smoke_image.png`, `smoke_audio.wav`, `smoke_doc_fa.txt`
`bridge/`	mtmd-bridge sidecar (Dockerfile + FastAPI shim)
`configs/arash_persona.md`	System prompt + OWUI paste-paths for Arash branding
`configs/ldap-ca/`	Drop AD CA bundle here as `ca.crt` (read-only mount)
`systemd/llm-stack.service`	Auto-start on boot

Services

Defined in docker-compose.yml. Always-on except minio-init (one-shot) and the RAG trio (qdrant, tei-embed, tei-rerank) which only run on RAG-enabled profiles.

Service	Image	Role
`llama-server`	`ghcr.io/ggml-org/llama.cpp:server-b8953`	LLM inference
`open-webui`	`ghcr.io/open-webui/open-webui:main`	Chat UI + orchestration
`postgres`	`postgres:16-alpine`	Auth, chats, configs
`minio`	`minio/minio:RELEASE.2025-09-07T16-13-09Z-cpuv1`	S3-compatible object store
`minio-init`	`minio/mc:RELEASE.2025-08-13T08-35-41Z-cpuv1`	One-shot bucket bootstrap
`mtmd-bridge`	`arash/mtmd-bridge:1` (built locally)	Audio STT shim for Gemma 4
`qdrant`	`qdrant/qdrant:v1.17.1`	Vector DB (RAG profiles only)
`tei-embed`	`ghcr.io/huggingface/text-embeddings-inference:cpu-1.9.3`	BGE-M3 dense embedder
`tei-rerank`	same image	BGE-reranker-v2-m3 cross-encoder

Build flow (on the staging host, has internet)

bash

# 1. Pull + save Docker images, pin digests
make pull-images          # or: bash scripts/pull_and_save_images.sh

# 2. Download GGUF weights + RAG models
#    Defaults: UD-Q2_K_XL + UD-Q4_K_XL + UD-Q6_K_XL + Q8_0 + mmproj + BGE-M3 + BGE-reranker
make download-models      # or: bash scripts/download_model.sh

# 3. Commit digests + checksums
git add images/DIGESTS.lock models/SHA256SUMS
git commit -m "chore: pin image digests and model checksums"

# 4. Smoke test the whole stack
cp .env.profile.small .env
make up && make smoke && make down

# 5. Package for handoff to Windows/VMware operator
make bundle               # or: bash scripts/package_bundle.sh
# → produces llm-stack-bundle-YYYYMMDD-HHMM.tar.zst

Deploy flow (on the air-gapped Ubuntu 24.04 VM)

bash

# 1. Pre-reqs (build host only, before disabling internet):
#    get.docker.com → Docker Engine + Compose v2.

# 2. Land the bundle
sudo tar --zstd -xf llm-stack-bundle-*.tar.zst -C /opt
cd /opt/llm-stack

# 3. Load all Docker images, verify digests
make load-images          # or: bash scripts/load_images.sh

# 4. Pick a profile for this hardware
bash scripts/tune.sh          # reports recommendation
cp .env.profile.<x> .env      # apply (review first)

# 5. Register with systemd so the stack boots on reboot
sudo INSTALL_DIR=/opt/llm-stack bash scripts/install_systemd.sh
sudo systemctl start llm-stack

# 6. Sanity check
make smoke                # or: bash scripts/smoke_test.sh

# 7. Open http://<vm-ip>:3000  → create first admin account in the UI

make help lists every wrapper target.

Tuning knobs (`.env`)

Var	What it controls
`LLAMA_MODEL_FILE`	Which GGUF in `models/` to load
`LLAMA_THREADS`	`--threads` (set to physical cores, not hyperthreads)
`LLAMA_THREADS_BATCH`	`--threads-batch` (usually = THREADS)
`LLAMA_CTX_SIZE`	Total context shared across all slots
`LLAMA_PARALLEL`	`--parallel` — concurrent generation slots (each costs KV RAM)
`LLAMA_CONT_BATCHING`	`--cont-batching` (1 = on, recommended)
`LLAMA_MLOCK`	`--mlock` (1 = keep model in RAM, avoids swap; prod only)
`LLAMA_CACHE_TYPE_K/V`	KV cache quantization (`f16` best, `q8_0` halves RAM)
`LLAMA_EXTRA_ARGS`	Passthrough for extra `llama-server` flags
`LLAMA_MMPROJ_FILE`	Multimodal projector (vision path on llama-server). Empty = text-only.
`LLAMA_AUDIO_ADAPTER_FILE`	Same projector, loaded by mtmd-bridge (audio path)
`MINIO_ROOT_USER` / `MINIO_ROOT_PASSWORD`	MinIO admin creds. Rotate per VM before first start.
`VECTOR_DB`	`qdrant` (RAG ON) or empty (Chroma fallback, RAG effectively OFF for Persian)
`RAG_EMBEDDING_` / `RAG_RERANKING_`	Wires Open WebUI into the TEI containers; see § RAG (Persian).
`OPENWEBUI_HOST_PORT`	Host port the chat UI binds to (default 3000)
`ENABLE_LDAP` / `LDAP_*`	Bind OWUI to Active Directory; see § LDAP / Active Directory.
`ENABLE_LOGIN_FORM` / `ENABLE_SIGNUP` / `WEBUI_AUTH`	Sign-in surface toggles
`ENABLE_PERSISTENT_CONFIG`	Keep `false` so `.env` always wins over Postgres on restart

Profile comparison

Knob	small	medium	large	xlarge
Target hardware	2c / 4 GB	8c / 16 GB	32c / 128 GB	64+c / 256+ GB
Model file	`…UD-Q2_K_XL.gguf`	`…UD-Q4_K_XL.gguf`	`…UD-Q6_K_XL.gguf`	`…Q8_0.gguf`
`LLAMA_THREADS` (= `_BATCH`)	2	8	32	64
`LLAMA_CTX_SIZE`	1024	8192	16384	32768
`LLAMA_PARALLEL`	1	2	8	16
`LLAMA_CACHE_TYPE_K/V`	`q8_0`	`f16`	`f16`	`f16`
`LLAMA_MLOCK`	0	1	1	1
Multimedia (vision + audio)	OFF	ON	ON	ON
RAG (qdrant + TEI)	OFF	OFF	ON	ON
Use case	Smoke test	≤10 concurrent	20–30 concurrent	30+ concurrent

All profiles share LLAMA_CONT_BATCHING=1, LLAMA_BATCH_SIZE=2048, LLAMA_UBATCH_SIZE=512, LLAMA_REASONING=off, LLAMA_ALIAS=Arash LLM, and the same auth defaults (local-only, ENABLE_SIGNUP=true so the first user becomes admin).

Multimedia (vision + audio-in)

Gemma 4 E4B is a native multimodal model. This stack ships vision input and audio input out of the box; image generation and TTS are out of scope for v1.

Reference: https://unsloth.ai/docs/models/gemma-4 GGUF source: https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF

What gets loaded. Unsloth ships a single combined projector, mmproj-BF16.gguf (~992 MB), that handles both modalities. Two env vars point at the same file:

LLAMA_MMPROJ_FILE — read by llama-server for image content parts (vision).
LLAMA_AUDIO_ADAPTER_FILE — read by the mtmd-bridge sidecar for audio.

The two-knob shape is preserved so a future split distribution (or a vision-only / audio-only opt-out) doesn't require a compose change.

Why the bridge sidecar exists. Upstream llama-server returns HTTP 500 on input_audio content blocks (ggml-org/llama.cpp#21868) — libmtmd supports Gemma 4 audio fully; only server.cpp's content-type dispatch is missing. The bridge is a ~140-LoC FastAPI service that exposes OpenAI-compatible /v1/audio/transcriptions and shells out to llama-mtmd-cli per request. OWUI is wired in via AUDIO_STT_ENGINE=openai

AUDIO_STT_OPENAI_API_BASE_URL=http://mtmd-bridge:8081/v1.

Quantization choice. BF16, not F16 or F32. F16 risks numerical drift in the audio Conformer encoder; F32 is 2× the size for no quality win; unsloth's documented example uses BF16.

Disable on RAM-starved hosts. The small (2c/4 GB) profile ships with both env vars empty — projector + UD-Q2_K_XL + OWUI + postgres + MinIO together saturate 4 GB. To toggle, set both to /models/mmproj-BF16.gguf (enable) or empty (disable) and docker compose up -d — no rebuild, missing files don't break the deploy.

Verify. bash scripts/smoke_test.sh exercises both paths against scripts/fixtures/{smoke_image.png,smoke_audio.wav}. Vision is fast (~one prompt-prefill); audio is slow (~100 s on 8c/16 GB for a 1-s clip — mtmd-cli reloads the full model per request by design, so we never hold GBs of weights resident between voice messages). Manual: drag an image into chat → Gemma describes it in Persian; click the mic → bridge transcribes.

Object storage (MinIO)

MinIO is always-on, mandatory. All Open WebUI uploads and (future) generated media land in S3 buckets instead of the open_webui_data volume. The console (port 9001) is intentionally not exposed; admin happens via mc exec.

Bucket	Purpose
`open-webui-uploads`	Chat attachments (images, audio, files)
`media-cache`	Transcoded / derived assets (room for v2 features)

Both are created idempotently by the minio-init sidecar on every boot.

Inspect from the host:

bash

# List buckets
docker compose run --rm --no-deps --entrypoint sh minio-init -c \
  'mc alias set local http://minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" && mc ls local'

# Stat a bucket
docker compose run --rm --no-deps --entrypoint sh minio-init -c \
  'mc alias set local http://minio:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD" && mc stat local/open-webui-uploads'

Rotate the admin password per VM (mandatory before users come online):

bash

echo "MINIO_ROOT_PASSWORD=$(openssl rand -hex 32)" >> .env
docker compose up -d minio
# Re-run minio-init so its mc alias re-binds to the new password:
docker compose up -d --force-recreate minio-init

MINIO_ROOT_USER / MINIO_ROOT_PASSWORD are also re-used by Open WebUI as S3_ACCESS_KEY_ID / S3_SECRET_ACCESS_KEY, so a single rotation propagates everywhere on the next compose-up.

RAG (Persian)

Open WebUI's stock RAG stack is built around sentence-transformers/all-MiniLM-L6-v2 (English-only, 22 MB) plus in-process Chroma — fine for English demos, near-random recall for Persian. This stack swaps that triplet for a Persian-strong, fully containerized, fully air-gapped path:

Layer	Service	Model	Endpoint
Vector store	`qdrant`	(HNSW + cosine, 1024-dim)	`http://qdrant:6333` — internal only
Embedder	`tei-embed`	`BAAI/bge-m3` dense, 1024-dim	`http://tei-embed:8080/v1/embeddings`
Reranker	`tei-rerank`	`BAAI/bge-reranker-v2-m3` cross-encoder	`http://tei-rerank:8080/rerank`

Both TEI containers run the same image (different --model-id paths, different container names). Weights mount read-only from models/embed/bge-m3/ and models/rerank/bge-reranker-v2-m3/, both downloaded by scripts/download_model.sh (skip with --no-rag). All three containers run with HF_HUB_OFFLINE=1 and Qdrant's telemetry is off (QDRANT__TELEMETRY_DISABLED=true) — zero outbound traffic at runtime.

Why these models. BGE-M3 (XLM-RoBERTa-XL backbone, 568M params) is the only multilingual dense embedder in its size class that scores well on Persian MIRACL; BGE-reranker-v2-m3 is the matched cross-encoder. Both were picked over Persian-specific options (HooshvareLab/ParsBERT) because the contrastive multilingual fine-tuning beats them on standard retrieval metrics even on Persian-only corpora.

Open WebUI integration is native (no shim sidecar). Source confirms RAG_RERANKING_ENGINE=external posts {query, documents, top_n} to RAG_EXTERNAL_RERANKER_URL and reads back [{index, relevance_score}, …] — identical to TEI's /rerank Cohere/Jina-shaped contract. Embeddings ride the OpenAI-compat /v1/embeddings route via RAG_EMBEDDING_ENGINE=openai + RAG_OPENAI_API_BASE_URL.

Sizing floor. Adding RAG lifts the realistic floor from ~6 GB (multimedia-on) to ~10 GB:

Component	RAM
v1 + multimedia floor	~6.0 GB
Qdrant (idle, small collection)	~0.3 GB
TEI BGE-M3 (CPU, dense)	~2.5 GB
TEI BGE-reranker-v2-m3 (CPU)	~1.5 GB
RAG-on floor	~10 GB

Disk: ~4.4 GB more in models/ (BGE-M3 pytorch_model.bin 2.27 GB + reranker model.safetensors 2.27 GB + tokenizers).

Profile defaults. small (4 GB) and medium (12 GB) ship with VECTOR_DB= empty (RAG OFF) — Qdrant + 2× TEI together don't fit 12 GB under sustained concurrency. large (≥24 GB) and xlarge enable RAG by default. To opt in on a medium host with extra headroom, append the RAG block from .env.example to your .env and docker compose up -d.

Verify. bash scripts/smoke_test.sh runs four RAG probes (qdrant /healthz, tei-embed dim=1024, tei-rerank top-1=Tehran, end-to-end doc → search → top-hit). For latency, bash scripts/bench.sh --rag [conc] [reps] indexes a synthetic Persian corpus and reports per-phase median + p95 — useful for sizing top_K under real load.

Bench reference (8c/16 GB host, 5×3 = 15 runs):

Phase	median	p95
embed	0.10 s	0.39 s
search	0.004 s	0.005 s
rerank	2.35 s	2.92 s
total	2.46 s	3.32 s

Rerank dominates because XLM-RoBERTa cross-encoders are quadratic in (query × top_K × doc) on CPU. With OWUI's default RAG_TOP_K_RERANKING=5 this is workable for short Persian docs; bump to 3 for longer docs.

Persistence

Postgres is always on. Four named Docker volumes hold all stateful data:

Volume	Mounted in	What lives there
`postgres_data`	postgres `/var/lib/postgresql/data`	SQL: chats, users, auth, model configs, prompts, tags
`open_webui_data`	open-webui `/app/backend/data`	Small OWUI cache, static assets (Chroma dir is inert when `VECTOR_DB=qdrant`)
`minio_data`	minio `/data`	All uploads/blobs (chat attachments, recorded audio, images)
`qdrant_data`	qdrant `/qdrant/storage`	Vector index (RAG dense embeddings, HNSW graph, payloads)

docker compose down keeps all four volumes. Only docker compose down -v destroys them — so the normal stop/start lifecycle is fully persistent.

DATABASE_URL is composed in docker-compose.yml from five .env knobs (POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB) so each piece is independently tunable without editing compose.

Rotate secrets per VM (mandatory before users come online)

Order matters: rotate POSTGRES_PASSWORD before the first docker compose up -d on the target VM. Once Postgres has initialized its data dir with a given password, editing .env won't change the on-disk credentials — Postgres keeps rejecting the new value from DATABASE_URL and Open WebUI fails to connect.

bash

cd /opt/llm-stack
{
  echo "POSTGRES_PASSWORD=$(openssl rand -hex 32)"
  echo "WEBUI_SECRET_KEY=$(openssl rand -hex 32)"
} >> .env
docker compose up -d

If the VM was already started with the staging defaults, rotate after the fact:

bash

NEW=$(openssl rand -hex 32)
docker compose exec postgres psql -U openwebui -d openwebui \
  -c "ALTER USER openwebui WITH PASSWORD '$NEW';"
sed -i "s/^POSTGRES_PASSWORD=.*/POSTGRES_PASSWORD=$NEW/" .env
docker compose up -d open-webui   # picks up new DATABASE_URL; chats preserved

WEBUI_SECRET_KEY rotation is independent: edit .env, then docker compose up -d open-webui. All currently-logged-in users get kicked out (their JWT cookies become invalid).

LDAP / Active Directory

Open WebUI binds to AD entirely through env vars — no Python patching, no rebuilt image. The full block lives in .env.example under # ─── LDAP / Active Directory ───. Profiles ship with ENABLE_LDAP=false, so a fresh deploy comes up on local auth, the operator creates the first admin, and only then flips LDAP on.

Reference: https://docs.openwebui.com/features/authentication-access/auth/ldap/

Why ENABLE_PERSISTENT_CONFIG=false is mandatory in this stack. OWUI normally snapshots every PersistentConfig env var (the LDAP block included) into Postgres on the first boot and ignores later .env edits. In an air-gapped datacenter where the admin UI requires a working admin login, that turns a typo in LDAP_SERVER_HOST into a DB-surgery incident. With the flag false (set in every profile), every restart re-reads .env — fixing a bad LDAP value is one edit + docker compose up -d.

CA certs. Drop the AD chain into ./configs/ldap-ca/ca.crt (PEM, chain order server → intermediate → root). The directory is bind-mounted read-only at /etc/openwebui/ldap-ca/, which is where LDAP_CA_CERT_FILE defaults to. Keep LDAP_VALIDATE_CERT=true in production; flip to false only as a temporary measure during bring-up.

First-boot bootstrap order.

Deploy with the profile defaults (LDAP off, signup on). First user to sign up becomes admin.
Edit .env: fill the LDAP block, optionally set ENABLE_SIGNUP=false. docker compose up -d (or restart open-webui) — values take effect.
AD users log in via LDAP, land as pending, admin promotes from the admin panel. AD-role → OWUI-role mapping is not supported by OWUI for LDAP today, only group sync.
(Optional) After stability, ENABLE_LOGIN_FORM=false to hide the local form. The seeded local admin (WEBUI_ADMIN_EMAIL / WEBUI_ADMIN_PASSWORD) still works — keep those creds in the customer's vault as the break-glass account.

AD vs OpenLDAP attribute map.

Attribute	Active Directory	OpenLDAP / FreeIPA
`LDAP_ATTRIBUTE_FOR_USERNAME`	`sAMAccountName`	`uid`
`LDAP_ATTRIBUTE_FOR_MAIL`	`userPrincipalName` (or `mail`)	`mail`
`LDAP_ATTRIBUTE_FOR_GROUPS`	`memberOf`	`memberOf` / `groupOfNames`

Recovery / worst-case ladder. Every row is recoverable from the host shell on the air-gapped VM, no DB surgery, no internet:

Failure	Recovery
LDAP host unreachable	`ENABLE_LDAP=false` → `docker compose up -d`. Local admin still logs in.
Bind DN / password rotated	Edit `LDAP_APP_DN` / `LDAP_APP_PASSWORD`, `up -d`. (`ENABLE_PERSISTENT_CONFIG=false` guarantees pickup.)
AD cert chain changed	Replace `./configs/ldap-ca/ca.crt`, `up -d`. Last resort: `LDAP_VALIDATE_CERT=false`.
Locked out (LDAP-only + AD down)	`ENABLE_LDAP=false`, `ENABLE_LOGIN_FORM=true`, `up -d`. Log in as the seeded local admin.
OWUI itself misbehaves on auth	`WEBUI_AUTH=false` for one restart, inspect, then re-enable. Never ship this way.
LDAP permanently broken, AD only via existing proxy	Populate `WEBUI_AUTH_TRUSTED_*_HEADER`, set `ENABLE_LOGIN_FORM=false`, point the datacenter proxy at port 3000. The proxy MUST strip these headers from inbound client requests or any client can forge an identity.

Group sync. Set ENABLE_LDAP_GROUP_MANAGEMENT=true (and optionally ENABLE_LDAP_GROUP_CREATION=true) and OWUI mirrors the user's memberOf list onto OWUI groups on each login. Use OWUI groups to gate model and tool access from the admin panel.

Verify. bash scripts/smoke_test.sh runs an LDAP probe when ENABLE_LDAP=true (skipped silently otherwise) — TCP-tests ${LDAP_SERVER_HOST}:${LDAP_SERVER_PORT} from inside the open-webui container so DNS resolves the same way OWUI sees it. Manual end-to-end: open the login page, confirm the "Sign in with ${LDAP_SERVER_LABEL}" button shows alongside the local form, log in with an AD user, confirm the user appears as pending in admin → users.

Switching model / quant without a rebuild

bash

# 1. Put a new GGUF in models/ (or run scripts/download_model.sh <QUANT>)
# 2. Edit .env → LLAMA_MODEL_FILE=<new-file>.gguf
# 3. Restart just the inference service:
docker compose up -d --force-recreate llama-server

Dual-socket NUMA tuning (if target is 2-socket)

Not built in. When needed, run two llama-server instances in a second compose file, pinning each to one NUMA node via cpuset/numactl, expose them on two distinct host ports, and let the datacenter's L7 load-balance between them (round-robin or least-conn — same upstream that already terminates TLS in front of the OVA). The current compose is intentionally single-instance for simplicity; copy-and-adapt when the datacenter gives you a dual-socket box.

Rollback

Every shippable state is tagged in git (v1.0-bundle-<date>). Images are pinned by sha256 digest in docker-compose.yml, so git revert also rolls back image versions deterministically. Model weights are content-addressed via models/SHA256SUMS.

TTFT tuning (read if first-token latency feels bad)

llama.cpp on CPU is memory-bandwidth bound. Visible time-to-first-token is roughly:

TTFT ≈ prompt_tokens × per_token_prefill_ms + reasoning_tokens × per_token_gen_ms

Knobs in order of impact:

Knob	What it changes	Typical win
`LLAMA_REASONING=off`	Kills Gemma 4's invisible thinking-mode generation	10–25 s
Physical cores (hardware)	Prefill scales ~linearly in cores w/ AVX2	2–8×
`LLAMA_CACHE_TYPE_K/V=f16`	Avoids q8_0 KV dequant overhead	5–15 %
Quant choice	UD-Q4_K_XL > Q4_K_M ≈ Q4_0 (Persian quality) ≫ Q2	varies
`LLAMA_CTX_SIZE`	Attention cost scales with ctx; cap to what users need	5–20 %

Rough TTFT for a 250-token conversation (Persian), Gemma 4 E4B, reasoning off:

Hardware	Quant	Estimated TTFT
2c / 16 GB (staging)	UD-Q2_K_XL	~25–30 s
2c / 16 GB (staging)	UD-Q4_K_XL	~25 s
8c / 16 GB	UD-Q4_K_XL	~8 s
16c / 64 GB	UD-Q6_K_XL	~4 s
32c / 128 GB	UD-Q6_K_XL	~2 s
64c / 256 GB	Q8_0	~1 s

Per-request override. Users who want thinking for a specific prompt can set reasoning_effort in the OpenAI request body (or toggle it from Open WebUI's advanced params) without restarting the server.

If LLAMA_REASONING=off doesn't take effect: check the llama-server boot log for srv init: init: chat template, thinking = 0. If it still says = 1, add LLAMA_EXTRA_ARGS=--chat-template-kwargs '{"enable_thinking": false}' to .env, restart. Last-resort fallback: dump the jinja template (docker exec llm-llama-server /app/llama-server --dump-chat-template), strip <|think|>, mount as a file, pass --chat-template-file.

Notes / caveats

Gemma 4 E4B is multimodal (text + image + audio). All three modalities are wired up in v1 — see § Multimedia. Image generation and TTS are out of scope for v1 and deferred to v2.
Persian is not explicitly listed in the model card's out-of-the-box language set, though the model is trained on 140+ languages. Quality benchmarking on Persian prompts is part of first-boot verification.

sdasdawq234/H-LLM-LOCAL

LLM Stack — Gemma 4 E4B on CPU, Air-Gapped

Files at a glance

Services

Build flow (on the staging host, has internet)

Deploy flow (on the air-gapped Ubuntu 24.04 VM)

Tuning knobs (.env)

Profile comparison

Multimedia (vision + audio-in)

Object storage (MinIO)

RAG (Persian)

Persistence

Rotate secrets per VM (mandatory before users come online)

LDAP / Active Directory

Switching model / quant without a rebuild

Dual-socket NUMA tuning (if target is 2-socket)

Rollback

TTFT tuning (read if first-token latency feels bad)

Notes / caveats

sdasdawq234/
H-LLM-LOCAL

Tuning knobs (`.env`)