retrico-lm-4b
retrico-lm-4b is a 4B-parameter language model built for universal structured information extraction. Give it any text and a JSON schema — it returns a valid, schema-conformant JSON object with no post-processing required.
The model handles the full spectrum of extraction tasks from a single interface: plain text, Markdown, HTML, and XML as input; flat facts, deeply nested objects, typed arrays, NER, and open relation extraction as output. There is no need to switch between specialized models — one template drives all extraction modes.
Built on Qwen3.5-4B, retrico-lm-4b is designed for production use and works best served via vLLM.
Key Features
- Universal input — plain text, Markdown documents, HTML pages, XML feeds
- Universal output — flat facts, nested objects, typed arrays, entity lists, relation triplets
- Template-driven — define any JSON schema and the model populates it from the input
- Typed fields — respects
string, integer, float, nested objects, arrays of objects
- Null-safe — missing values return as
null or [], never hallucinated
- Production-ready — optimized for vLLM with
language_model_only=True
Training
The model was trained in two stages:
- Supervised fine-tuning on synthetic data — training examples were generated using a large teacher LLM across a diverse set of domains and schema types
- Post-training on human-annotated data — further refined on a high-quality human-annotated dataset to improve precision, grounding, and schema adherence
Usage
The model uses a hybrid attention architecture and requires language_model_only=True and trust_remote_code=True.
1from vllm import LLM, SamplingParams
2from transformers import AutoTokenizer
3import json
4
5model_name = "knowledgator/retrico-lm-4b"
6tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
7
8llm = LLM(
9 model=model_name,
10 language_model_only=True,
11 gpu_memory_utilization=0.85,
12 max_model_len=65536,
13 trust_remote_code=True,
14 dtype="bfloat16",
15 enforce_eager=True,
16)
17
18sampling_params = SamplingParams(max_tokens=4096, temperature=0.0)
19
20def build_prompt(text, template):
21 if isinstance(template, (dict, list)):
22 template = json.dumps(template, indent=1, ensure_ascii=False)
23 content = (
24 "/no_think\n"
25 "Extract information from the following text according to the JSON template.\n\n"
26 "Important rules:\n"
27 "- If a field's value is not mentioned or cannot be found in the text, set it to null.\n"
28 "- Do not infer, guess, or hallucinate values that are not explicitly stated.\n"
29 "- If the template is completely unrelated to the text, return all fields as null.\n"
30 "- For list fields with no values found, return [] not [null].\n"
31 "- For dict/object fields with no values found, return {} not null.\n\n"
32 f"Template:\n{template}\n\nText:\n{text}\n\n"
33 "Return only the extracted JSON, nothing else."
34 )
35 return tokenizer.apply_chat_template(
36 [{"role": "user", "content": content}],
37 tokenize=False,
38 add_generation_prompt=True,
39 enable_thinking=False,
40 )
41
42def extract(text, template):
43 prompt = build_prompt(text, template)
44 output = llm.generate([prompt], sampling_params)[0]
45 raw = output.outputs[0].text.strip()
46 try:
47 return json.loads(raw)
48 except json.JSONDecodeError:
49 return {"__raw__": raw}
Examples
Plain Text Extraction
Structured fact extraction from prose. The model handles deeply nested schemas, typed numeric fields, arrays of objects, and null-safe output for fields absent in the source.
James Webb Space Telescope — deeply nested schema with typed fields, array of instrument objects, and multi-agency list
Input
1text = """
2The James Webb Space Telescope (JWST) is a space telescope designed to conduct infrared
3astronomy. It was launched on December 25, 2021, from the Guiana Space Centre in Kourou,
4French Guiana, aboard an Ariane 5 rocket. The telescope reached its final destination at
5the second Lagrange point (L2), approximately 1.5 million kilometers from Earth, on
6January 24, 2022, after a 30-day journey.
7
8JWST has a primary mirror diameter of 6.5 meters composed of 18 hexagonal gold-coated
9beryllium segments. Its total mass is 6,161 kg and it operates at a temperature of
10approximately minus 233 degrees Celsius. The telescope cost $10 billion to develop over
1120 years, led by NASA in partnership with ESA and the Canadian Space Agency.
12
13The telescope carries four scientific instruments: NIRCam (Near Infrared Camera) built by
14the University of Arizona, NIRSpec (Near Infrared Spectrograph) built by ESA, MIRI
15(Mid-Infrared Instrument) built jointly by NASA and ESA, and FGS/NIRISS (Fine Guidance
16Sensor and Near Infrared Imager and Slitless Spectrograph) built by CSA. Its science
17operations are managed by the Space Telescope Science Institute in Baltimore, Maryland.
18The expected mission lifetime is 10 years, with fuel reserves potentially extending it to 20.
19"""
20
21template = {
22 "name": "string",
23 "abbreviation": "string",
24 "type": "string",
25 "launch_date": "string",
26 "launch_site": "string",
27 "launch_vehicle": "string",
28 "destination": "string",
29 "distance_from_earth_km": "float",
30 "arrival_date": "string",
31 "journey_duration_days": "integer",
32 "specifications": {
33 "primary_mirror_diameter_m": "float",
34 "mirror_segments": "integer",
35 "mirror_material": "string",
36 "mirror_coating": "string",
37 "total_mass_kg": "integer",
38 "operating_temperature_celsius": "integer"
39 },
40 "cost_billion_usd": "float",
41 "development_years": "integer",
42 "mission_lifetime": {
43 "expected_years": "integer",
44 "maximum_years": "integer"
45 },
46 "agencies": ["string"],
47 "science_operations_center": "string",
48 "instruments": [
49 {
50 "name": "string",
51 "abbreviation": "string",
52 "built_by": "string"
53 }
54 ]
55}
Output
1{
2 "name": "James Webb Space Telescope",
3 "abbreviation": "JWST",
4 "type": "space telescope",
5 "launch_date": "2021-12-25",
6 "launch_site": "Guiana Space Centre",
7 "launch_vehicle": "Ariane 5",
8 "destination": "second Lagrange point (L2)",
9 "distance_from_earth_km": 1500000,
10 "arrival_date": "2022-01-24",
11 "journey_duration_days": 30,
12 "specifications": {
13 "primary_mirror_diameter_m": 6.5,
14 "mirror_segments": 18,
15 "mirror_material": "beryllium",
16 "mirror_coating": "gold",
17 "total_mass_kg": 6161,
18 "operating_temperature_celsius": -233
19 },
20 "cost_billion_usd": 10.0,
21 "development_years": 20,
22 "mission_lifetime": {
23 "expected_years": 10,
24 "maximum_years": 20
25 },
26 "agencies": ["NASA", "ESA", "Canadian Space Agency"],
27 "science_operations_center": "Space Telescope Science Institute",
28 "instruments": [
29 {"name": "NIRCam", "abbreviation": "Near Infrared Camera", "built_by": "University of Arizona"},
30 {"name": "NIRSpec", "abbreviation": "Near Infrared Spectrograph", "built_by": "ESA"},
31 {"name": "MIRI", "abbreviation": "Mid-Infrared Instrument", "built_by": "NASA and ESA"},
32 {"name": "FGS/NIRISS", "abbreviation": "Fine Guidance Sensor and Near Infrared Imager and Slitless Spectrograph", "built_by": "CSA"}
33 ]
34}
NVIDIA FY2024 Financials — financial report with null-safe segment data: growth percentages absent in the source correctly return as null
Input
1text = """
2NVIDIA Corporation, headquartered in Santa Clara, California, reported exceptional financial
3results for fiscal year 2024. Total revenue reached $60.9 billion, a 122% increase year-over-year,
4driven primarily by the Data Center segment which generated $47.5 billion, up 217% from the
5prior year. The Gaming segment contributed $10.4 billion, while Professional Visualization
6brought in $1.7 billion and Automotive $1.1 billion.
7
8Net income for the year was $29.8 billion compared to $4.4 billion in fiscal 2023, representing
9a 581% increase. Gross margin expanded to 72.7% from 56.9%. The company returned $9.5 billion
10to shareholders through buybacks and paid dividends of $395 million. As of January 2024,
11NVIDIA employed approximately 29,600 people worldwide. CEO Jensen Huang founded the company
12in 1993 alongside Chris Malachowsky and Curtis Priem.
13"""
14
15template = {
16 "company": "string",
17 "headquarters": "string",
18 "fiscal_year": "integer",
19 "ceo": "string",
20 "other_founders": ["string"],
21 "founded": "integer",
22 "employees": "integer",
23 "financials": {
24 "total_revenue_billion": "float",
25 "revenue_growth_yoy_pct": "float",
26 "net_income_billion": "float",
27 "net_income_growth_pct": "float",
28 "gross_margin_pct": "float",
29 "shareholder_returns_billion": "float",
30 "dividends_million": "float"
31 },
32 "segments": [
33 {
34 "name": "string",
35 "revenue_billion": "float",
36 "growth_pct": "float"
37 }
38 ]
39}
Output
1{
2 "company": "NVIDIA Corporation",
3 "headquarters": "Santa Clara, California",
4 "fiscal_year": 2024,
5 "ceo": "Jensen Huang",
6 "other_founders": ["Chris Malachowsky", "Curtis Priem"],
7 "founded": 1993,
8 "employees": 29600,
9 "financials": {
10 "total_revenue_billion": 60.9,
11 "revenue_growth_yoy_pct": 122.0,
12 "net_income_billion": 29.8,
13 "net_income_growth_pct": 581.0,
14 "gross_margin_pct": 72.7,
15 "shareholder_returns_billion": 9.5,
16 "dividends_million": 395.0
17 },
18 "segments": [
19 {"name": "Data Center", "revenue_billion": 47.5, "growth_pct": 217.0},
20 {"name": "Gaming", "revenue_billion": 10.4, "growth_pct": null},
21 {"name": "Professional Visualization", "revenue_billion": 1.7, "growth_pct": null},
22 {"name": "Automotive", "revenue_billion": 1.1, "growth_pct": null}
23 ]
24}
Moderna mRNA-1273 Clinical Trial — dense numerical extraction: efficacy stats, confidence intervals, demographic breakdowns, and adverse event arrays from a single paragraph
Input
1text = """
2The Moderna mRNA-1273 vaccine clinical trial enrolled 30,420 participants across 99 sites
3in the United States between July 27 and October 23, 2020. Participants were randomly assigned
4in a 1:1 ratio to receive two injections of either 100 micrograms of mRNA-1273 or placebo,
5administered 28 days apart. The median age of participants was 51.4 years; 47.3% were female,
624.8% were Hispanic or Latino, and 10.2% were Black or African American.
7
8The primary endpoint was prevention of COVID-19 illness with onset at least 14 days after
9the second injection. The trial reported 185 cases of COVID-19, with 11 in the mRNA-1273 group
10and 174 in the placebo group, yielding a vaccine efficacy of 94.1% with a 95% confidence
11interval of 89.3% to 96.8%. Severe COVID-19 occurred in 30 participants, all in the placebo
12group, suggesting 100% efficacy against severe disease. The most common adverse events were
13injection-site pain (92% of participants), fatigue (70%), headache (64.7%), and myalgia (61.5%).
14The trial was led by principal investigator Lindsey Baden at Brigham and Women's Hospital.
15"""
16
17template = {
18 "vaccine_name": "string",
19 "trial_sites": "integer",
20 "enrollment": "integer",
21 "dosage_mcg": "integer",
22 "efficacy": {
23 "overall_pct": "float",
24 "confidence_interval": {"lower": "float", "upper": "float"},
25 "against_severe_disease_pct": "float"
26 },
27 "covid_cases": {
28 "total": "integer",
29 "vaccine_group": "integer",
30 "placebo_group": "integer"
31 },
32 "adverse_events": [
33 {"name": "string", "frequency_pct": "float"}
34 ]
35}
Output
1{
2 "vaccine_name": "mRNA-1273",
3 "trial_sites": 99,
4 "enrollment": 30420,
5 "dosage_mcg": 100,
6 "efficacy": {
7 "overall_pct": 94.1,
8 "confidence_interval": {"lower": 89.3, "upper": 96.8},
9 "against_severe_disease_pct": 100.0
10 },
11 "covid_cases": {
12 "total": 185,
13 "vaccine_group": 11,
14 "placebo_group": 174
15 },
16 "adverse_events": [
17 {"name": "injection-site pain", "frequency_pct": 92.0},
18 {"name": "fatigue", "frequency_pct": 70.0},
19 {"name": "headache", "frequency_pct": 64.7},
20 {"name": "myalgia", "frequency_pct": 61.5}
21 ]
22}
Markdown Extraction
ML Reading List — repeated hierarchical entries: ### heading + bullet list blocks parsed into a uniform array of structured paper objects
Input
1text = """
2# Machine Learning Reading List
3
4## Foundational Papers
5
6### Attention Is All You Need
7- **Authors:** Vaswani, Shazeer, Parmar et al.
8- **Year:** 2017
9- **Venue:** NeurIPS
10- **Citations:** 90,000+
11- **Key contribution:** Introduced the Transformer architecture
12- **Tags:** transformers, attention, NLP
13
14### ImageNet Classification with Deep CNNs
15- **Authors:** Krizhevsky, Sutskever, Hinton
16- **Year:** 2012
17- **Venue:** NeurIPS
18- **Citations:** 120,000+
19- **Key contribution:** Demonstrated deep CNNs on large-scale image recognition
20- **Tags:** CNN, computer vision, deep learning
21
22### Playing Atari with Deep Reinforcement Learning
23- **Authors:** Mnih et al.
24- **Year:** 2013
25- **Venue:** NIPS Workshop
26- **Citations:** 18,000+
27- **Key contribution:** Combined deep learning with reinforcement learning
28- **Tags:** RL, DQN, Atari
29"""
30
31template = {
32 "list_name": "string",
33 "papers": [
34 {
35 "title": "string",
36 "authors": ["string"],
37 "year": "integer",
38 "venue": "string",
39 "citations": "string",
40 "key_contribution": "string",
41 "tags": ["string"]
42 }
43 ]
44}
Output
1{
2 "list_name": "Machine Learning Reading List",
3 "papers": [
4 {
5 "title": "Attention Is All You Need",
6 "authors": ["Vaswani", "Shazeer", "Parmar"],
7 "year": 2017,
8 "venue": "NeurIPS",
9 "citations": "90,000+",
10 "key_contribution": "Introduced the Transformer architecture",
11 "tags": ["transformers", "attention", "NLP"]
12 },
13 {
14 "title": "ImageNet Classification with Deep CNNs",
15 "authors": ["Krizhevsky", "Sutskever", "Hinton"],
16 "year": 2012,
17 "venue": "NeurIPS",
18 "citations": "120,000+",
19 "key_contribution": "Demonstrated deep CNNs on large-scale image recognition",
20 "tags": ["CNN", "computer vision", "deep learning"]
21 },
22 {
23 "title": "Playing Atari with Deep Reinforcement Learning",
24 "authors": ["Mnih"],
25 "year": 2013,
26 "venue": "NIPS Workshop",
27 "citations": "18,000+",
28 "key_contribution": "Combined deep learning with reinforcement learning",
29 "tags": ["RL", "DQN", "Atari"]
30 }
31 ]
32}
HTML Extraction
NeurIPS 2024 — structured data from HTML markup: speaker objects from div.speaker elements, numeric attendance from span tags, nested objects
Input
1text = """
2<div class="conference">
3 <h1>NeurIPS 2024</h1>
4 <p>Dates: <time>December 10-15, 2024</time> |
5 Venue: <span>Vancouver Convention Centre</span>, Canada</p>
6 <p>Organized by <strong>NeurIPS Foundation</strong>.
7 Expected attendance: <span>15,000</span> researchers from <span>90</span> countries.</p>
8 <div class="speakers">
9 <div class="speaker"><strong>Yann LeCun</strong> — Meta AI —
10 Talk: <em>The Future of Self-Supervised Learning</em></div>
11 <div class="speaker"><strong>Yoshua Bengio</strong> — Mila —
12 Talk: <em>AI Safety and Alignment</em></div>
13 <div class="speaker"><strong>Fei-Fei Li</strong> — Stanford —
14 Talk: <em>Spatial Intelligence</em></div>
15 </div>
16</div>
17"""
18
19template = {
20 "event_name": "string",
21 "dates": "string",
22 "venue": "string",
23 "country": "string",
24 "organizer": "string",
25 "attendance": {
26 "expected": "integer",
27 "countries": "integer"
28 },
29 "keynote_speakers": [
30 {
31 "name": "string",
32 "affiliation": "string",
33 "talk_title": "string"
34 }
35 ]
36}
Output
1{
2 "event_name": "NeurIPS 2024",
3 "dates": "December 10-15, 2024",
4 "venue": "Vancouver Convention Centre",
5 "country": "Canada",
6 "organizer": "NeurIPS Foundation",
7 "attendance": {
8 "expected": 15000,
9 "countries": 90
10 },
11 "keynote_speakers": [
12 {"name": "Yann LeCun", "affiliation": "Meta AI", "talk_title": "The Future of Self-Supervised Learning"},
13 {"name": "Yoshua Bengio", "affiliation": "Mila", "talk_title": "AI Safety and Alignment"},
14 {"name": "Fei-Fei Li", "affiliation": "Stanford", "talk_title": "Spatial Intelligence"}
15 ]
16}
XML Extraction
Apollo Program — XML attributes mapped to JSON fields: <period start="1961" end="1972"/> and <member role="commander"> resolved into typed schema fields
Input
1text = """
2<?xml version="1.0" encoding="UTF-8"?>
3<project>
4 <name>Apollo</name>
5 <organization>NASA</organization>
6 <period start="1961" end="1972"/>
7 <budget_billion_usd>25.4</budget_billion_usd>
8 <director>George Low</director>
9 <missions>
10 <mission>
11 <name>Apollo 11</name>
12 <date>1969-07-16</date>
13 <result>success</result>
14 <crew>
15 <member role="commander">Neil Armstrong</member>
16 <member role="lunar_module_pilot">Buzz Aldrin</member>
17 <member role="command_module_pilot">Michael Collins</member>
18 </crew>
19 </mission>
20 <mission>
21 <name>Apollo 13</name>
22 <date>1970-04-11</date>
23 <result>failure</result>
24 <crew>
25 <member role="commander">Jim Lovell</member>
26 <member role="lunar_module_pilot">Fred Haise</member>
27 <member role="command_module_pilot">Jack Swigert</member>
28 </crew>
29 </mission>
30 </missions>
31</project>
32"""
33
34template = {
35 "project_name": "string",
36 "organization": "string",
37 "director": "string",
38 "period": {
39 "start": "integer",
40 "end": "integer"
41 },
42 "budget_billion_usd": "float",
43 "missions": [
44 {
45 "name": "string",
46 "date": "string",
47 "result": "string",
48 "crew": [
49 {
50 "name": "string",
51 "role": "string"
52 }
53 ]
54 }
55 ]
56}
Output
1{
2 "project_name": "Apollo",
3 "organization": "NASA",
4 "director": "George Low",
5 "period": {"start": 1961, "end": 1972},
6 "budget_billion_usd": 25.4,
7 "missions": [
8 {
9 "name": "Apollo 11",
10 "date": "1969-07-16",
11 "result": "success",
12 "crew": [
13 {"name": "Neil Armstrong", "role": "commander"},
14 {"name": "Buzz Aldrin", "role": "lunar_module_pilot"},
15 {"name": "Michael Collins", "role": "command_module_pilot"}
16 ]
17 },
18 {
19 "name": "Apollo 13",
20 "date": "1970-04-11",
21 "result": "failure",
22 "crew": [
23 {"name": "Jim Lovell", "role": "commander"},
24 {"name": "Fred Haise", "role": "lunar_module_pilot"},
25 {"name": "Jack Swigert", "role": "command_module_pilot"}
26 ]
27 }
28 ]
29}
Relation Extraction
Open-domain NER and relation extraction. No predefined label sets — entity types and relation types are inferred directly from the text.
ARM Holdings — multi-hop corporate chain: acquisition, licensing, blocked deal, IPO, and CEO succession all extracted from one paragraph
Input
1text = """
2In September 2016, Softbank acquired ARM Holdings for $32 billion, the largest
3ever acquisition of a European technology company at the time. ARM's CEO Simon Segars
4continued to lead the company under SoftBank ownership. Masayoshi Son, founder and CEO
5of SoftBank, personally championed the deal. ARM Holdings, originally spun out of Acorn
6Computers in 1990, licenses its chip architecture to Apple, Qualcomm, Samsung, and NVIDIA
7among others. In 2020, NVIDIA announced an agreement to acquire ARM from SoftBank for
8$40 billion, a deal that was ultimately blocked by regulators in the UK, EU, and US in
9February 2022 following objections from Qualcomm, Google, and Microsoft. ARM subsequently
10went public on the NASDAQ in September 2023, with SoftBank retaining a 90% stake.
11Rene Haas replaced Simon Segars as CEO of ARM in February 2022 following the deal's collapse.
12"""
13
14template = {
15 "entities": [
16 {"entity": "string", "type": "string"}
17 ],
18 "relations": [
19 {"head": "string", "relation": "string", "tail": "string"}
20 ]
21}
Output
1{
2 "entities": [
3 {"entity": "Softbank", "type": "organization"},
4 {"entity": "ARM Holdings", "type": "organization"},
5 {"entity": "Simon Segars", "type": "person"},
6 {"entity": "Masayoshi Son", "type": "person"},
7 {"entity": "Acorn Computers", "type": "organization"},
8 {"entity": "Apple", "type": "organization"},
9 {"entity": "Qualcomm", "type": "organization"},
10 {"entity": "Samsung", "type": "organization"},
11 {"entity": "NVIDIA", "type": "organization"},
12 {"entity": "Rene Haas", "type": "person"},
13 {"entity": "NASDAQ", "type": "organization"}
14 ],
15 "relations": [
16 {"head": "Softbank", "relation": "acquired", "tail": "ARM Holdings"},
17 {"head": "ARM Holdings", "relation": "spun out of", "tail": "Acorn Computers"},
18 {"head": "ARM Holdings", "relation": "licenses to", "tail": "Apple"},
19 {"head": "ARM Holdings", "relation": "licenses to", "tail": "Qualcomm"},
20 {"head": "ARM Holdings", "relation": "licenses to", "tail": "Samsung"},
21 {"head": "ARM Holdings", "relation": "licenses to", "tail": "NVIDIA"},
22 {"head": "NVIDIA", "relation": "announced agreement to acquire", "tail": "ARM Holdings"},
23 {"head": "ARM Holdings", "relation": "went public on", "tail": "NASDAQ"},
24 {"head": "Rene Haas", "relation": "replaced", "tail": "Simon Segars"}
25 ]
26}
Higgs boson / CERN — diverse entity types (particle, facility, experiment) across institutions, experiments, and individuals
Input
1text = """
2The discovery of the Higgs boson was announced on July 4, 2012, by CERN, the European
3Organization for Nuclear Research, based in Geneva, Switzerland. The discovery was made
4using the Large Hadron Collider, the world's largest particle accelerator, built in a
527-kilometer tunnel beneath the French-Swiss border. Two independent experiments confirmed
6the discovery: ATLAS, led by Fabiola Gianotti, and CMS, led by Joe Incandela. Peter Higgs
7of the University of Edinburgh and François Englert of the Université Libre de Bruxelles
8were awarded the Nobel Prize in Physics in 2013 for their theoretical prediction of the
9particle in 1964. Robert Brout, who co-authored the original paper with Englert, had died
10in 2011 and was therefore ineligible for the prize. CERN's Director General at the time
11of the announcement was Rolf Heuer.
12"""
13
14template = {
15 "entities": [
16 {"entity": "string", "type": "string"}
17 ],
18 "relations": [
19 {"head": "string", "relation": "string", "tail": "string"}
20 ]
21}
Output
1{
2 "entities": [
3 {"entity": "Higgs boson", "type": "particle"},
4 {"entity": "CERN", "type": "organization"},
5 {"entity": "Geneva", "type": "location"},
6 {"entity": "Switzerland", "type": "location"},
7 {"entity": "Large Hadron Collider", "type": "facility"},
8 {"entity": "ATLAS", "type": "experiment"},
9 {"entity": "Fabiola Gianotti", "type": "person"},
10 {"entity": "CMS", "type": "experiment"},
11 {"entity": "Joe Incandela", "type": "person"},
12 {"entity": "Peter Higgs", "type": "person"},
13 {"entity": "University of Edinburgh", "type": "organization"},
14 {"entity": "François Englert", "type": "person"},
15 {"entity": "Université Libre de Bruxelles", "type": "organization"},
16 {"entity": "Robert Brout", "type": "person"},
17 {"entity": "Rolf Heuer", "type": "person"}
18 ],
19 "relations": [
20 {"head": "Higgs boson", "relation": "announced by", "tail": "CERN"},
21 {"head": "CERN", "relation": "based in", "tail": "Geneva"},
22 {"head": "Higgs boson", "relation": "discovered using", "tail": "Large Hadron Collider"},
23 {"head": "ATLAS", "relation": "led by", "tail": "Fabiola Gianotti"},
24 {"head": "CMS", "relation": "led by", "tail": "Joe Incandela"},
25 {"head": "Peter Higgs", "relation": "affiliated with", "tail": "University of Edinburgh"},
26 {"head": "François Englert", "relation": "affiliated with", "tail": "Université Libre de Bruxelles"},
27 {"head": "Peter Higgs", "relation": "awarded", "tail": "Nobel Prize in Physics"},
28 {"head": "François Englert", "relation": "awarded", "tail": "Nobel Prize in Physics"},
29 {"head": "Peter Higgs", "relation": "predicted", "tail": "Higgs boson"},
30 {"head": "François Englert", "relation": "predicted", "tail": "Higgs boson"},
31 {"head": "Robert Brout", "relation": "co-authored with", "tail": "François Englert"},
32 {"head": "CERN", "relation": "Director General", "tail": "Rolf Heuer"}
33 ]
34}
OpenAI — dense organizational history: founding, funding, role transitions, board conflict, and resignations into 20 entities and 19 relations from a multi-paragraph document
Input
1text = """
2OpenAI was founded in December 2015 as a nonprofit by Sam Altman, Greg Brockman, Ilya
3Sutskever, Wojciech Zaremba, John Schulman, and Elon Musk, with a $1 billion funding
4commitment from Musk, Peter Thiel, Reid Hoffman, and Amazon Web Services. Musk resigned
5from the board in 2018 citing potential conflicts of interest with Tesla's AI ambitions.
6
7In 2019, OpenAI transitioned to a capped-profit model and received a $1 billion investment
8from Microsoft. Ilya Sutskever, who had previously worked as a researcher at Google Brain
9under Geoffrey Hinton, became OpenAI's Chief Scientist. Greg Brockman, who had previously
10been CTO of Stripe, became OpenAI's President. Sam Altman, formerly president of Y Combinator,
11became CEO.
12
13GPT-4, released in March 2023, was developed under the leadership of Ilya Sutskever and
14deployed in Microsoft's Azure OpenAI Service. In November 2023, the OpenAI board briefly
15fired Sam Altman, citing concerns about his candor. Greg Brockman resigned in protest.
16Altman was reinstated five days later following pressure from Microsoft and OpenAI employees.
17Ilya Sutskever, who had voted to fire Altman, later expressed regret and subsequently
18resigned in May 2024 to found Safe Superintelligence Inc. together with Daniel Gross
19and Daniel Levy.
20"""
21
22template = {
23 "entities": [
24 {"entity": "string", "type": "string"}
25 ],
26 "relations": [
27 {"head": "string", "relation": "string", "tail": "string"}
28 ]
29}
Output
1{
2 "entities": [
3 {"entity": "OpenAI", "type": "organization"},
4 {"entity": "Sam Altman", "type": "person"},
5 {"entity": "Greg Brockman", "type": "person"},
6 {"entity": "Ilya Sutskever", "type": "person"},
7 {"entity": "Wojciech Zaremba", "type": "person"},
8 {"entity": "John Schulman", "type": "person"},
9 {"entity": "Elon Musk", "type": "person"},
10 {"entity": "Peter Thiel", "type": "person"},
11 {"entity": "Reid Hoffman", "type": "person"},
12 {"entity": "Amazon Web Services", "type": "organization"},
13 {"entity": "Tesla", "type": "organization"},
14 {"entity": "Microsoft", "type": "organization"},
15 {"entity": "Google Brain", "type": "organization"},
16 {"entity": "Geoffrey Hinton", "type": "person"},
17 {"entity": "Stripe", "type": "organization"},
18 {"entity": "Y Combinator", "type": "organization"},
19 {"entity": "GPT-4", "type": "technology"},
20 {"entity": "Safe Superintelligence Inc.", "type": "organization"},
21 {"entity": "Daniel Gross", "type": "person"},
22 {"entity": "Daniel Levy", "type": "person"}
23 ],
24 "relations": [
25 {"head": "OpenAI", "relation": "founded_by", "tail": "Sam Altman"},
26 {"head": "OpenAI", "relation": "founded_by", "tail": "Greg Brockman"},
27 {"head": "OpenAI", "relation": "founded_by", "tail": "Ilya Sutskever"},
28 {"head": "OpenAI", "relation": "founded_by", "tail": "Elon Musk"},
29 {"head": "OpenAI", "relation": "funding_from", "tail": "Peter Thiel"},
30 {"head": "OpenAI", "relation": "funding_from", "tail": "Amazon Web Services"},
31 {"head": "Elon Musk", "relation": "resigned_from", "tail": "OpenAI board"},
32 {"head": "OpenAI", "relation": "received_investment_from", "tail": "Microsoft"},
33 {"head": "Ilya Sutskever", "relation": "worked_at", "tail": "Google Brain"},
34 {"head": "Ilya Sutskever", "relation": "became_role_at", "tail": "OpenAI Chief Scientist"},
35 {"head": "Greg Brockman", "relation": "worked_as", "tail": "CTO of Stripe"},
36 {"head": "Greg Brockman", "relation": "became_role_at", "tail": "OpenAI President"},
37 {"head": "Sam Altman", "relation": "worked_as", "tail": "president of Y Combinator"},
38 {"head": "GPT-4", "relation": "developed_under_leadership_of", "tail": "Ilya Sutskever"},
39 {"head": "GPT-4", "relation": "deployed_in", "tail": "Microsoft's Azure OpenAI Service"},
40 {"head": "Greg Brockman", "relation": "resigned_in_protest_of", "tail": "firing of Sam Altman"},
41 {"head": "Ilya Sutskever", "relation": "voted_to_fire", "tail": "Sam Altman"},
42 {"head": "Ilya Sutskever", "relation": "resigned", "tail": "May 2024"},
43 {"entity": "Ilya Sutskever", "relation": "founded", "tail": "Safe Superintelligence Inc."}
44 ]
45}
Constrained Relation Extraction
The open-domain template shown above infers entity and relation types freely from the text. For benchmarking and production pipelines where label sets are fixed, you can constrain the model by injecting allowed types directly into the prompt:
1TEMPLATE = json.dumps({
2 "entities": [{"entity": "string", "type": "string"}],
3 "relations": [{"head": "string", "relation": "string", "tail": "string"}]
4}, indent=1)
5
6def build_prompt(text: str, entity_types: list[str], relation_types: list[str]) -> str:
7 et_str = ", ".join(entity_types)
8 rt_str = ", ".join(relation_types)
9 return (
10 "/no_think\n"
11 "Extract entities and relations from the following text according to the JSON template.\n\n"
12 "Important rules:\n"
13 "- If a field's value is not mentioned or cannot be found in the text, set it to null.\n"
14 "- Do not infer, guess, or hallucinate values that are not explicitly stated.\n"
15 "- For list fields with no values found, return [] not [null].\n"
16 "- Entity text must be exact substrings from the input text.\n"
17 f"- Entity types must be one of: {et_str}\n"
18 f"- Relation types must be one of: {rt_str}\n\n"
19 f"Template:\n{TEMPLATE}\n\n"
20 f"Text:\n{text}\n\n"
21 "Return only the extracted JSON, nothing else."
22 )
This is the setup used to produce the benchmark results below.
Benchmarks
All benchmarks are zero-shot — the model was not trained on any of these datasets.
Benchmark Charts
WL Graph F1 — overall
retrico-lm-4b0.761
gpt-oss-120b0.787
Llama-3.3-70B0.784
DeepSeek-V3.10.782
Qwen3-32B0.733
NuExtract30.730
ROUGE-L — overall
retrico-lm-4b0.532
Llama-3.3-70B0.550
DeepSeek-V3.10.525
gpt-oss-120b0.520
Qwen3-32B0.485
NuExtract30.375
Valid JSON rate — overall
retrico-lm-4b96.0%
Llama-3.3-70B98.9%
gpt-oss-120b98.6%
DeepSeek-V3.196.8%
Qwen3-32B93.9%
NuExtract392.5%
WL Graph F1 · 256–1023 tokens
retrico-lm-4b74.8%
NuExtract378.3%
DeepSeek-V3.176.2%
Llama-3.3-70B74.5%
Qwen3-32B61.2%
gpt-oss-120b34.3%
WL Graph F1 · 1024–3999 tokens
retrico-lm-4b82.5%
Llama-3.3-70B83.6%
DeepSeek-V3.183.5%
gpt-oss-120b82.2%
Qwen3-32B76.7%
NuExtract375.9%
WL Graph F1 · ≥4000 tokens
retrico-lm-4b34.1%
gpt-oss-120b76.4%
Qwen3-32B53.8%
Llama-3.3-70B25.1%
DeepSeek-V3.17.9%
NuExtract37.5%
Valid JSON · 1024–3999 tokens
retrico-lm-4b98.0%
gpt-oss-120b100%
Llama-3.3-70B98.7%
DeepSeek-V3.197.3%
Qwen3-32B97.1%
NuExtract392.1%
Valid JSON · ≥4000 tokens
retrico-lm-4b33.3%
gpt-oss-120b93.0%
Llama-3.3-70B66.7%
Qwen3-32B66.7%
NuExtract333.3%
DeepSeek-V3.120.0%
RE benchmarks — Micro-F1
CrossRE (test · ai/news/science)
retrico-lm-4b8.5
gliner2-large2.0
DocRED (validation)
retrico-lm-4b14.5
gliner2-large13.8
Relation Extraction
Evaluated on two standard RE benchmarks with constrained entity and relation type sets (see prompt format above).
CrossRE — cross-domain relation extraction. Evaluated on the
test split, domains:
ai,
news,
science.
DocRED — document-level relation extraction from Wikipedia and Wikidata. Evaluated on the
validation split.
| Dataset | Model | Micro-F1 | Macro-F1 | Precision | Recall |
|---|
| CrossRE | retrico-lm-4b | 8.5 | 7.3 | 7.7 | 9.6 |
| CrossRE | fastino/gliner2-large-v1 | 2.0 | 2.0 | 2.3 | 1.7 |
| DocRED | retrico-lm-4b | 14.5 | 6.7 | 13.3 | 15.9 |
| DocRED | fastino/gliner2-large-v1 | 13.8 | 6.9 | 13.0 | 14.6 |
Comparison with Large Language Models — Human-Annotated Eval Split
Evaluated on an internal held-out set with human-annotated ground truth. Metrics:
- WL Graph F1 — graph-based metric that converts predicted and reference JSON into trees, computes semantic node embeddings, and propagates via Weisfeiler-Leman message passing. Captures both structural correctness and semantic similarity of extracted values.
- ROUGE-L — longest common subsequence overlap between predicted and reference JSON strings.
- Valid JSON Rate — fraction of outputs that parse as valid JSON.
| Model | WL Graph F1 | ROUGE-L | Valid JSON Rate |
|---|
| retrico-lm-4b | 0.7606 | 0.5323 | 96.0% |
| openai/gpt-oss-120b | 0.7868 | 0.5204 | 98.6% |
| Meta-Llama-3.3-70B-Instruct | 0.7837 | 0.5503 | 98.9% |
| DeepSeek-V3.1 | 0.7821 | 0.5253 | 96.8% |
| Qwen3-32B | 0.7329 | 0.4852 | 93.9% |
| numind/NuExtract3 | 0.7302 | 0.3747 | 92.5% |
Valid JSON Rate by input length:
| Token bucket | gpt-oss-120b | Llama-3.3-70B | DeepSeek-V3.1 | retrico-lm-4b | Qwen3-32B | NuExtract3 |
|---|
| 256–1023 | 100% | 100% | 100% | 100% | 100% | 98.8% |
| 1024–3999 | 100% | 98.7% | 97.3% | 98.0% | 97.1% | 92.1% |
| ≥4000 | 93.0% | 66.7% | 20.0% | 33.3% | 66.7% | 33.3% |
WL Graph F1 by input length:
| Token bucket | gpt-oss-120b | Llama-3.3-70B | DeepSeek-V3.1 | retrico-lm-4b | Qwen3-32B | NuExtract3 |
|---|
| 256–1023 | 34.3% | 74.5% | 76.2% | 74.8% | 61.2% | 78.3% |
| 1024–3999 | 82.2% | 83.6% | 83.5% | 82.5% | 76.7% | 75.9% |
| ≥4000 | 76.4% | 25.1% | 7.9% | 34.1% | 53.8% | 7.5% |
Links
Citation
1@misc{knowledgator2025retrico,
2 title={retrico-lm: Schema-Guided Structured Information Extraction},
3 author={Knowledgator Engineering},
4 year={2025},
5 url={https://huggingface.co/knowledgator}
6}
