I've been experimenting with a new quantization approach that selectively elevates the precision of key layers beyond what the default IMatrix configuration provides.
In my testing, standard IMatrix quantization underperforms at lower bit depths, especially with Mixture of Experts (MoE) models. To address this, I'm using the --tensor-type option in llama.cpp to manually "bump" important layers to higher precision. You can see the implementation here:
👉 Layer bumping with llama.cpp
While this does increase model file size, it significantly improves precision for a given quantization level.
I'd love your feedback—have you tried this? How does it perform for you?
Welcome to the Schematron series, Inference.net's long‑context extraction models specialized in converting noisy HTML into clean, typed JSON that conforms to your custom schema. The Schematron series was purpose‑trained for web scraping, data ingestion, and transforming arbitrary pages into structured records.
We're releasing these models in two different sizes:
Schematron‑8B — marginal quality lift on harder/longer pages
Schematron‑3B — recommended default; near‑parity quality at ~50% cost of Schematron-8B
[!NOTE]
This model card is dedicated to the smaller Schematron-3B model. Check out Schematron-8B for the larger model.
I/O at a glance
Input: Cleaned HTML + JSON Schema (can be extracted from typed model like Pydantic/Zod)
Output: Strictly valid JSON conforming to the provided schema (no narration)
[!NOTE]
The JSON Schema passed as input needs to conform to the schema.org schema.
Long context: Robust to lengthy, noisy HTML (up to 128K tokens)
Variants: 3B (default, most cost‑efficient) · 8B (marginal quality lift at ~2× cost)
Model Details
Family: Schematron (3B and 8B)
Context window: Up to 128K tokens
Input: Cleaned or raw HTML and a JSON Schema
Output: Strict JSON that conforms to the provided schema
Benchmarks
HTML-to-JSON Extraction Quality
We evaluated extraction quality using Gemini 2.5 Pro as a judge, scoring extractions from 1-5 where 5 represents perfect extraction.
Model
LLM-as-Judge Score
GPT-4.1
4.74
Schematron-8B
4.64
Schematron-3B
4.41
Gemini-3B-Base
2.24
Web-Augmented Factuality on SimpleQA
We evaluated Schematron's real-world impact on LLM factuality using SimpleQA.
Test Pipeline:
Query Generation: Primary LLM (GPT-5 Nano or GPT-4.1) generates search queries and defines extraction schema
Web Search: Search provider (SERP or Exa) retrieves relevant pages
Structured Extraction: Schematron extracts JSON data from retrieved pages using the schema
Answer Synthesis: Primary LLM produces final answer from structured data
image/png
Key findings:
Web search paired with JSON extraction improves factuality: Adding Schematron with web retrieval improves GPT-5 Nano's accuracy from 8.54% to 82.87%—nearly a 10x improvement
Search provider matters: Exa (82.9%) significantly outperforms SERP (64.2%) for factual retrieval, while also being more cost-effective
Structured extraction beats raw HTML: Processing raw HTML would require 100k+ tokens for 10 searches; Schematron's JSON extraction reduces this by orders of magnitude
Small specialized models win: Schematron-8B (82.87%) outperforms the much larger Gemini 2.5 Flash (80.61%) on this task, showing that fine-tuning for well-defined tasks beats general purpose models
Performance scales with model quality: When paired with GPT-4.1, Schematron achieves 85.58% accuracy, showing the approach benefits from stronger base models
Minimal Quickstart
Use these local snippets to prepare HTML and compose a schema‑guided prompt. The model returns strictly valid JSON; validate it against your schema downstream.
python
1from lxml.html.clean import Cleaner
2import lxml.html as LH
34HTML_CLEANER = Cleaner(5 scripts=True,6 javascript=True,7 style=True,8 inline_style=True,9 safe_attrs_only=False,10)111213defstrip_noise(html:str)->str:14"""Remove scripts, styles, and JavaScript from HTML using lxml.
15 """16ifnot html ornot html.strip():17return""18try:19 doc = LH.fromstring(html)20 cleaned = HTML_CLEANER.clean_html(doc)21return LH.tostring(cleaned, encoding="unicode")22except Exception:23return""
Compose messages with your schema and cleaned HTML:
python
1defconstruct_messages(schema:str, html:str):2"""Construct messages for a schema‑guided extraction request."""3 response_prompt ={4"prompt_part_one":(5"You are going to be given a JSON schema following the standardized JSON "6"Schema format. You are going to be given a HTML page and you are going "7"to apply the schema to the HTML page however you see it as applicable "8"and return the results in a JSON object. The schema is as follows:"9),10"prompt_part_two":"Here is the HTML page:",11"prompt_part_three":"MAKE SURE ITS VALID JSON.",12}1314 user_prompt =(15 response_prompt['prompt_part_one']16+"\n\n"+ schema +"\n\n"17+ response_prompt['prompt_part_two']18+"\n\n"+ html +"\n\n"19+ response_prompt['prompt_part_three']20)2122return[23{"role":"system","content":"You are a helpful assistant"},24{"role":"user","content": user_prompt},25]
[!NOTE]
In the serverless API there's no need to pass anything but the HTML. We handle the prompt formatting for you.
Recommendations
Temperature 0 and JSON mode for deterministic, parseable output
Validate responses against your schema (e.g., Pydantic or Zod)
Pre‑clean HTML (remove scripts/styles) when possible; avoid over‑aggressive removal
Using lxml to clean the HTML is not required, but is recommended as it matches the training data.
Limitations
Static HTML only; render client‑side content upstream
Very large pages may require truncation
Ambiguous fields depend on schema clarity; be explicit in field descriptions
Safety and Responsible Use
Extracted data may include personal or sensitive information present in the page—handle and store responsibly
Respect site terms, robots.txt, and applicable laws
Use downstream validation and guardrails for compliance
The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : Source Code Quantum Network Monitor. You will also find the code I use to quantize the models if you want to do it yourself GGUFModelBuilder
💬 How to test:
Choose an AI assistant type:
TurboLLM (GPT-4.1-mini)
HugLLM (Hugginface Open-source models)
TestLLM (Experimental CPU-only)
What I’m Testing
I’m pushing the limits of small open-source models for AI network monitoring, specifically:
Function calling against live network services
How small can a model go while still handling:
Automated Nmap security scans
Quantum-readiness checks
Network Monitoring tasks
🟡 TestLLM – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):
✅ Zero-configuration setup
⏳ 30s load time (slow inference but no API costs) . No token limited as the cost is low.
🔧 Help wanted! If you’re into edge-device AI, let’s collaborate!
Other Assistants
🟢 TurboLLM – Uses gpt-4.1-mini :
**It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited.
Create custom cmd processors to run .net code on Quantum Network Monitor Agents
Real-time network diagnostics and monitoring
Security Audits
Penetration testing (Nmap/Metasploit)
🔵 HugLLM – Latest Open-source models:
🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.
💡 Example commands you could test:
"Give me info on my websites SSL certificate"
"Check if my server is using quantum safe encyption for communication"
"Run a comprehensive security audit on my server"
'"Create a cmd processor to .. (what ever you want)" Note you need to install a Quantum Network Monitor Agent to run the .net code on. This is a very flexible and powerful feature. Use with caution!
Final Word
I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is open source. Feel free to use whatever you find helpful.
If you appreciate the work, please consider buying me a coffee ☕. Your support helps cover service costs and allows me to raise token limits for everyone.
I'm also open to job opportunities or sponsorship.
