It was specifically further fine-tuned on Cluster 2 - Hands on User data from the CowCorpus dataset to adapt to the specific intervention preferences and behavioral patterns of this user group.

This model is designed for the task of Human Intervention Prediction in collaborative web navigation. Unlike standard autonomous agents, this model predicts when Hands-on user (Cluster 2) needs to take control from an AI agent. It utilizes multimodal inputs (screenshots, DOM trees, and action history) to distinguish between safe autonomous execution and moments requiring human error correction, preference alignment, or assistance.

Model Details

Model Description

Developed by: CowCorpus Team (Huq et al.)
Model type: Multimodal Causal Language Model
Parent Model: CowCorpus/CowCorpus-llama3-llava-next-8b
Base model: lmms-lab/llama3-llava-next-8b
Language: English
License: Llama 3 Community License Agreement
Paper: Modeling Distinct Human Interaction in Web Agents
Repository: GitHub: oaishi/CowCorpus

Input Data

The model is trained on a rich, multimodal state representation:

Visual Screenshot: The pixel-level view of the current webpage.
UI Structure (AX Tree): The accessibility tree (textual representation of DOM).
Past Trajectory: The history of actions taken by the agent/human so far.
Proposed Next Action: The action that the autonomous agent intends to take. The model evaluates if this intent is erroneous.

How to Get Started

For inference code, prompt templates, and setup instructions, please refer to our GitHub Repository.

Training Data

The model underwent a two-stage training process:

Stage 1 (General Adaptation): Fine-tuned on the complete CowCorpus dataset.
Stage 2 (User Personalization): Further fine-tuned on the User Cluster 2 subset of CowCorpus, consists of 37 trajectories and 296 steps.

User Cluster 2 Characteristics:

Data Source: A subset of the collaborative trajectories specific to User Group 2.
Behavioral Profile: Hands-on user, Frequent and substantial interventions, often later, balancing between taking over and giving control back.

Training Configuration

Hyperparameters:
- Learning Rate: Linear decay from 1e-5 to ~2e-9
- Epochs: 6
- Global Steps: 120
- Batch Size: 1
- Precision: bfloat16

Evaluation: Cross-Cluster Personalization

We evaluate the model using the Perfect Timing Score (PTS), a metric designed to measure the temporal accuracy of intervention predictions.

Because this is a personalized model, we report Cross-Cluster PTS. This measures how well the model (trained on Cluster 2) performs on its own test data versus test data from other user clusters. High performance on the diagonal (matching train/test groups) indicates successful personalization.

Cross-Cluster PTS Heatmap

The table below displays the PTS values. Rows represent the User Cluster the model was trained on, and Columns represent the User Cluster data it was tested on.

Trained On (Model)	Tested On: Collaborative (User 0)	Tested On: Hands-on (User 2)	Tested On: Takeover (User 3)
Collaborative	0.187	0.130	0.058
Hands-on	0.417	0.583	0.468
Takeover	0.000	0.027	0.009

Note: All models are evaluated in a zero-shot setting without reasoning.

Citation [optional]

If you use this model or dataset, please cite our work: Paper incoming