A 4-bit (NF4, double-quantized) version of farbodtavakkoli/OTel-LLM-E4B-IT, re-packaged for low-VRAM inference on consumer and free-tier GPUs (e.g. NVIDIA T4, 15GB).

This repository does not modify the model's weights, training, or behavior in any substantive way — it is a quantized re-packaging intended to make the original OTel-LLM checkpoint runnable on hardware with limited GPU memory. All credit for the underlying model, training data, and evaluation belongs to the original author.

Model Details

Attribute	Value
Base model	`google/gemma-4-E4B-it`
Source model	`farbodtavakkoli/OTel-LLM-E4B-IT`
Parameters	4.5B (base)
Quantization	4-bit NF4, double quantization (bitsandbytes)
Compute dtype	float16
Approx. VRAM footprint	~3–4 GB (vs. ~9 GB at bf16/fp16)
Domain	Telecommunications (3GPP, O-RAN, GSMA, IETF RFCs)
Intended use	Context-grounded RAG question answering
Language	English
License	Apache 2.0 (inherits from base)

Why This Repo Exists

The original OTel-LLM-E4B-IT checkpoint requires roughly 9 GB of VRAM at full precision, which exceeds the memory available on free-tier GPUs such as the T4. This repository stores the model quantized to 4-bit precision, allowing it to be loaded and run entirely on GPUs with as little as ~6–8 GB of free VRAM, with no meaningful loss in usable output quality for RAG-style telecom Q&A.

Intended Use

This model is designed for context-grounded telecom answer generation inside Retrieval-Augmented Generation (RAG) pipelines. It expects retrieved context passages alongside the user's question and should not be used as an unrestricted, context-free general knowledge assistant.

In scope:

Answering telecom questions (3GPP, O-RAN, GSMA specifications, IETF RFCs) when relevant context is supplied
RAG pipelines built around telecom knowledge bases
Research and prototyping on telecom-domain LLM applications

Out of scope:

General-purpose chat or reasoning outside the telecom domain
Unrestricted, context-free QA (use the base OTel model or a dedicated -Safety variant instead)
Safety-critical, regulatory, or customer-facing decisions without human verification

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "YOUR-USERNAME/otel-llm-e4b-it-4bit"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    low_cpu_mem_usage=True,
)  # quantization_config is stored in this repo's config and applied automatically

prompt = """You are a precise telecom assistant in a RAG pipeline.
Use only the retrieved context to answer.

User Question
What is the purpose of the F1 interface in O-RAN?

Retrieved Contexts
CONTEXT 1
The F1 interface connects the O-RAN Distributed Unit (O-DU) to the O-RAN Central Unit (O-CU).

Answer:"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Requirements: transformers, accelerate, bitsandbytes, sentencepiece

Limitations

Inherits all limitations of the base OTel-LLM-E4B-IT model: English-only, telecom-domain-specific, and evaluated on held-out data from the same distribution as training (not an independent external benchmark).
4-bit quantization introduces a small, generally negligible accuracy trade-off versus the full-precision checkpoint; it has not been independently re-benchmarked in this repository.
Generated content should be verified by a domain expert before any operational, regulatory, safety-related, or customer-facing use.
Not intended for context-free, general-purpose question answering.

Attribution & License

Original model: farbodtavakkoli/OTel-LLM-E4B-IT by Farbod Tavakkoli, part of the OTel Family of Models
Base model: google/gemma-4-E4B-it
License: Apache 2.0, inherited from the upstream release. Users must comply with both the OTel release license and the Gemma base model's terms of use.

This is an unofficial, community-produced quantization and is not affiliated with or endorsed by the original OTel-LLM authors.

Citation

If you use this model, please cite the original work:

bibtex

@misc{otel_models_2026,
  title  = {OTel: Open Telco AI Datasets, Benchmarks, and Models},
  author = {Tavakkoli, Farbod and others},
  year   = {2026},
  note   = {Open Telco (OTel) model release},
  url    = {https://huggingface.co/farbodtavakkoli}
}

Alireza1913/otel-llm-e4b-it-4bit

OTel-LLM-E4B-IT — 4-bit Quantized