Screenshot Intent Classifier
This repository contains a Longformer-based classifier fine-tuned to decide
whether a conversational agent should trigger a screenshot tool for the
latest user message.
Base Model
This model is fine-tuned from
allenai/longformer-base-4096,
and inherits the base encoder's maximum context length and tokenizer.
"longformer-base-4096" is a RoBERTa-style encoder with support for sequences up to 4,096 tokens using a combination of sliding-window local attention and user-configured global attention. We follow the common pattern for classification tasks with Longformer by assigning global attention to the first token in each sequence.
Classifier
0 / no_screenshot: do not call the screenshot tool.1 / take_screenshot: call the screenshot tool.
The input is a text block representing the recent conversation history,
formatted as one utterance per line, prefixed with a speaker tag, e.g.:
1USER: I'm wondering if blue goes well with yellow.
2USER: What's your take on this?
At inference time, the host application typically feeds the last few
conversation turns (most importantly the latest user message) in this format
and thresholds the classifier's take_screenshot probability to decide
whether to trigger the tool.
Training Data
The classifier was trained on a curated, hand-labelled private dataset. It
contains hundreds of single-turn and multi-turn examples specifying whether
each user message should or should not trigger a screenshot, including:
- Clear positive triggers ("look at this", "check this out", "rate this pic").
- Clear negatives (off-topic chit-chat, abstract statements, idioms like
"I'll look into it").
- Edge cases involving deictic pronouns, quantities ("take 2 screenshots"),
negation ("don't look"), multi-turn context, and more.
No external user logs or third-party datasets were used; the training data is
purely synthetic / curated for this intent task.
Training Setup
Approximate defaults:
- Epochs: 3
- Batch size: 16 (per device)
- Learning rate: 2e-5
- Weight decay: 0.01
- Max sequence length: 1536 tokens (truncation for old entries applied beyond this)
The script builds examples by concatenating conversation history up to and
including the current user message, one utterance per line prefixed with
"USER:". Multi-turn conversations therefore become multiple training examples
with growing context.
Usage
Basic usage with the Transformers library:
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
3
4MODEL_ID = "yapwithai/yap-longformer-screenshot-intent"
5
6tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
7model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)
8model.eval()
9
10text = "USER: look at this amazing sunset"
11inputs = tokenizer(
12 text,
13 return_tensors="pt",
14 truncation=True,
15 padding="max_length",
16 max_length=1536,
17)
18
19attention_mask = inputs["attention_mask"]
20global_attention_mask = torch.zeros_like(attention_mask)
21global_attention_mask[:, 0] = 1
22
23with torch.no_grad():
24 outputs = model(**inputs, global_attention_mask=global_attention_mask)
25 probs = outputs.logits.softmax(dim=-1)[0]
26
27p_no, p_yes = probs.tolist()
28print("P(no_screenshot)=", p_no)
29print("P(take_screenshot)=", p_yes)
In production, you would:
- Construct a conversation history string similar to the training format
(recent user turns, optionally assistant turns, each on its own line with a
speaker prefix).
- Run the classifier once per latest user message.
- Threshold
p_yes to decide whether to trigger the screenshot tool.
Longformer Citation
If you use Longformer in your work, please cite:
1@article{Beltagy2020Longformer,
2 title={Longformer: The Long-Document Transformer},
3 author={Iz Beltagy and Matthew E. Peters and Arman Cohan},
4 journal={arXiv:2004.05150},
5 year={2020},
6}
Longformer is an open-source project developed by the Allen Institute for Artificial Intelligence (AI2).
