TechTokenBERT is a state-of-the-art patent embedding model based on BERT. It outperforms larger and task-specific models on IPC code classification, citation prediction, and title–abstract matching.

The core innovation is treating International Patent Classification (IPC) codes as dedicated tokens in the model's vocabulary (technological tokens). This allows the attention mechanism to operate directly between patent text and classification codes during fine-tuning, producing embeddings that are simultaneously aware of linguistic content and technological structure.

Introduced in:

Anticipating Innovation Using Large Language Models
Enrico Maria Fenoaltea, Filippo Santoro, Giordano De Marzo, Segun Taofeek Aroyehun, Andrea Tacchella
arXiv:2605.04875 · May 2026
https://arxiv.org/abs/2605.04875

Model Description

TechTokenBERT extends the vocabulary of BERT4Patent with one dedicated token per IPC code at group level (~8000 codes total). Fine-tuning uses masked-language-modelling on sequences of the form:

[CLS] patent title [SEP] patent abstract [SEP] [TT_1] [TT_2] ... [TT_N] [SEP]

During fine-tuning the attention mechanism learns to link each technological token both to the words of the patent text and to the other IPC codes in the same patent. The result is a model with two complementary uses:

Patent embeddings: use the [CLS] token as a single vector representation of the full patent (title + abstract + IPC codes). The [CLS] embedding encodes information from both the text and the IPC codes it is associated with, outperforming standard sentence-embedding models on patent similarity tasks.
IPC-code embeddings: read the hidden-state vector at each technological-token position to obtain a context-dependent embedding of that IPC code in the specific patent.

Training Data

Source: Full European Patent Bulletin AB (English-language patents, 1980–2024)
Fine-tuning split: EPO Patents published 1980–2023
IPC granularity: Group level, yielding 8000 unique codes

Evaluation Results

Patent-related downstream tasks

Model	IPC Macro-F1 ↑	Citation MAP ↑	Title–Abstract AUC-ROC ↑
BERT4Patents	0.354	59.46	0.920
BERT4Patents FT (Mirror-BERT)	0.262	52.78	0.832
PatentSBERTa	0.356	75.95	0.985
Paecter	0.420	68.11	0.944
LLaMA 3.1 8B FT (LLM2Vec)	0.343	56.78	0.973
TechTokenBERT	0.488	68.96	0.994

For details see the full paper.

Usage

Minimal example: build a batch where only the abstract is truncated (on the right), while the title, tech tokens, and all special tokens are preserved. Then run the model and extract the [CLS] embedding for each example.

python

import torch
from transformers import BertTokenizer, BertModel

# ---------------------------------------------------------------------------
# 1. Load model + tokenizer
# ---------------------------------------------------------------------------
MODEL_NAME = "AndreaTacchella/TechTokenBert"
tokenizer = BertTokenizer.from_pretrained(MODEL_NAME)
# The model was fine-tuned with an MLM head, kept in the checkpoint for more advanced
# uses (e.g. tech-code prediction). It is not needed for embeddings, so we load the
# encoder only via BertModel; the resulting "pooler.dense" newly-initialized warning
# is expected and harmless -- we only read last_hidden_state, not pooler_output.
model = BertModel.from_pretrained(MODEL_NAME)
model.eval()

# ---------------------------------------------------------------------------
# 2. Toy data (3 rows)
# ---------------------------------------------------------------------------
titles = [
    "Method for cooling electronic components",
    "Wireless charging apparatus",
    "Biodegradable packaging material",
]

abstracts = [
    "A heat sink assembly that dissipates thermal energy from a processor using "
    "a network of micro-channels through which a coolant is circulated, thereby "
    "maintaining the junction temperature below a predefined threshold under load.",

    "An inductive power transfer system comprising a transmitter coil and a "
    "receiver coil aligned via a magnetic guidance structure to maximize coupling "
    "efficiency across a variable air gap.",

    "A composite film derived from plant-based polymers that decomposes under "
    "industrial composting conditions while providing an oxygen barrier suitable "
    "for food preservation.",
]

# Already preprocessed tech tokens (list of lists of IPC group-level strings)
tech_tokens_list = [
    ["h05k7", "g06f1"],
    ["h02j50", "h01f27"],
    ["c08l101", "b65d65"],
]


# ---------------------------------------------------------------------------
# 3. Build the padded batch (abstract truncated on the right only)
# ---------------------------------------------------------------------------
def build_batch(titles, abstracts, tech_tokens_list, tokenizer, max_length=512):
    cls_id = tokenizer.cls_token_id
    sep_id = tokenizer.sep_token_id

    all_ids = []
    for title, abstract, tech_tokens in zip(titles, abstracts, tech_tokens_list):
        title_ids = tokenizer.encode(title, add_special_tokens=False)
        abstract_ids = tokenizer.encode(abstract, add_special_tokens=False)
        tech_ids = tokenizer.encode(" ".join(tech_tokens), add_special_tokens=False)

        # [CLS] title [SEP] abstract [SEP] tech [SEP]  -> 4 special tokens fixed
        fixed_len = 4 + len(title_ids) + len(tech_ids)
        abstract_budget = max(max_length - fixed_len, 0)
        abstract_ids = abstract_ids[:abstract_budget]  # right-side truncation

        ids = (
            [cls_id]
            + title_ids
            + [sep_id]
            + abstract_ids
            + [sep_id]
            + tech_ids
            + [sep_id]
        )
        all_ids.append(ids)

    return tokenizer.pad({"input_ids": all_ids}, padding=True, return_tensors="pt")


enc = build_batch(titles, abstracts, tech_tokens_list, tokenizer, max_length=512)

# ---------------------------------------------------------------------------
# 4. Forward pass + extract the [CLS] embedding
# ---------------------------------------------------------------------------
with torch.no_grad():
    outputs = model(**enc)

# last hidden state: (batch, seq_len, hidden_dim); position 0 is [CLS]
cls_embeddings = outputs.last_hidden_state[:, 0, :]
print(cls_embeddings.shape)  # (3, 1024)

Extracting IPC-code embeddings

To obtain the context-dependent embedding of a specific IPC code within a patent, read the hidden-state vector at the position of the corresponding technological token (the tokens after the second [SEP]):

python

with torch.no_grad():
    outputs = model(**enc)

last_hidden = outputs.last_hidden_state  # (batch, seq_len, 1024)

# By construction the TechTokens are always enclosed between the last two [SEP] tokens
# Find the positions of the last two [SEP] tokens and extract the embeddings for the tokens in between

tech_token_embeddings = []
input_ids = enc['input_ids']

sep_token_id = tokenizer.sep_token_id

for i, ids in enumerate(input_ids):
    ids = ids.tolist()
    sep_indices = [idx for idx, x in enumerate(ids) if x == sep_token_id]
    # tech tokens are between the last two [SEP]
    if len(sep_indices) < 2:
        tech_token_embeddings.append(torch.empty((0, last_hidden.size(-1))))
        continue
    start = sep_indices[-2] + 1
    end = sep_indices[-1]
    emb = last_hidden[i, start:end, :]  # shape: (# tech tokens, hidden_dim)
    tech_token_embeddings.append(emb)

# Now tech_token_embeddings is a list, one per sample in batch, containing
# a tensor of shape (num_tech_tokens, hidden_size)

Input Format

[CLS] <title tokens> [SEP] <abstract tokens> [SEP] <ipc_code_1> <ipc_code_2> ... [SEP]

IPC codes must be lower-cased and at group level (e.g., h05k7, g06f1).
The abstract is the only segment that should be truncated when the total length exceeds 512 tokens; title and IPC codes are always kept in full.
If IPC codes are not available at inference time, the model can still be used with only title and abstract (omit the third segment); performance on IPC-aware tasks will be reduced.

Citation

bibtex

@article{fenoaltea2026anticipating,
  title   = {Anticipating Innovation Using Large Language Models},
  author  = {Fenoaltea, Enrico Maria and Santoro, Filippo and De Marzo, Giordano
             and Aroyehun, Segun Taofeek and Tacchella, Andrea},
  journal = {arXiv preprint arXiv:2605.04875},
  year    = {2026}
}