Metric	Original (FP16)	Quantized (GGUF Q4_K_M)	Change
VRAM Footprint	228.3 MB	69.6 MB	-69.5%
NLL Loss	6.2110	6.2483	-
Perplexity (PPL)	498.1917	517.1174	+3.80% (Drift)
Total Gen Time (50 seqs)	28.5s	36.2s	+27.0% (Slower)

Analysis

Fidelity: GGUF (Q4_K_M) exhibits superior fidelity compared to AWQ, with a significantly lower PPL drift (+3.80% vs +7.04%). It preserves the original model's predictive capabilities better.
De-novo Generation: This model shows an exceptional 100% pass rate in complexity filters. The quantization noise introduced by Q4_K_M helps break the repetitive "traps" often found in small 100M models during auto-regressive decoding.
Inference Latency: When used outside of llama.cpp (e.g., in Transformers or vLLM), GGUF currently suffers from higher latency due to the lack of optimized non-CPU kernels.

Quick Start

Using llama.cpp

./llama-cli -m models/GenomeOcean-100M-v1.2-GGUF/model.gguf -p "ATGCGATCGATCGATCGATCG" -n 100

Using Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "models/GenomeOcean-100M-v1.2-GGUF",
    gguf_file="model.gguf",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("models/GenomeOcean-100M-v1.2-GGUF")

inputs = tokenizer("ATGCGATCGATCGATCGATCG", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

ThomasYn/GenomeOcean-100M-v1.2-GGUF

GenomeOcean-100M-v1.2-GGUF

Model Details

Benchmark Results

Analysis

Quick Start

Using llama.cpp

Using Transformers

ThomasYn/
GenomeOcean-100M-v1.2-GGUF