Lenormand HumOmni Track 2 System
This repository contains the reproducibility materials for Team Lenormand in HumOmni Track 2 ProactivEval.
Base Repository
Our system is built on top of the public MMDuet2 repository:
We use MMDuet2 as the streaming video-language backbone and modify the inference and response-control pipeline for HumOmni Track 2.
Overview
The final system is a multi-stage agentic inference harness rather than a single-pass VLM generation pipeline. We found that the streaming VLM should not simultaneously act as the visual reader, memory controller, response gate, and final answer writer. Therefore, we decouple the system into several functional modules.
The core idea is to treat the VLM as a mostly stateless visual reader, while external modules handle timing, answer selection, memory/state tracking, and post-processing.
System Components
1. V36 Reader / Timing Agent
We use the MMDuet2-style streaming pipeline to propose candidate response times. The raw V36-style output is not directly used as the final submission because it may contain long responses, repeated content, and late-stage narrative contamination.
2. Causal Visual Answer Regeneration
For each candidate response time t, the system regenerates a concise answer using only visual frames up to t. No future frames are used when generating a response at time t.
3. QEA Selection Module
We include a lightweight Question–Evidence–Answer selection module to choose concise and question-relevant answers from candidate generations. This module compares the question, candidate answer, and causal visual evidence. It does not use hidden labels, ground-truth reply spans, or leaderboard feedback.
4. External Memory Agent
The external memory module stores only question-relevant textual states rather than raw visual history. It checks whether a candidate turn is new, relevant, non-redundant, and not contaminated by previous narrative outputs.
This module is designed to reduce answer self-contamination in streaming generation. The VLM is used mainly for visual reading, while memory and response control are handled outside the autoregressive dialogue context.
5. Clean-Light-Plus Post-processing
The final global post-processing step removes only obvious generation artifacts, such as:
NO REPLYNS REPLYNSWYRREPLYING:no image displayedNo active movement occursNo actions take placeNo description availableNo response availableNo additional contextNo additional elements
This filtering is global and rule-based. It does not use ground-truth labels, hidden reply spans, leaderboard feedback, or video-specific manual edits.
Final Submission Variant
The final prediction file submitted to the official Google Drive is generated by:
memory_agent_top6_clean_light_plus
Final sanity check:
1N_records = 500
2unique_question_ids = 500
3duplicate_question_ids = 0
4empty_outputs = 0
5avg_turns_per_video = 3.03
6max_turns_per_video = 6
7time_min = 4.0
8time_max = 30.0
9chronological_per_video = True
10all turns have time and content = True
Base Models and Checkpoints
We do not duplicate large public checkpoints in this repository. The system uses public base resources, including MMDuet2-style streaming components and Qwen2.5-VL-based vision-language inference components.
This repository provides the modified inference harness, post-processing code, configuration notes, and reproducibility notebook needed to regenerate the submitted system output.
Modified Components
Compared with the public MMDuet2-style baseline, our submitted system modifies the response-control pipeline in the following ways:
-
Deterministic decoding and concise generation
- We use deterministic decoding for stable outputs.
- We favor concise visual answers instead of long dialogue-style responses.
-
KV-cache / writeback control
- We avoid feeding long generated answers back into the streaming context.
- We use short summary/marker-style writeback to reduce narrative self-contamination.
-
Timestamp correction and causal frame filtering
- Each response at time
t is generated only from frames up to t.
- The pipeline corrects timestamp alignment for frame sampling.
-
Per-turn answer regeneration
- V36-style outputs are used as timing proposals.
- Final answers are regenerated at candidate times using causal visual context.
-
QEA selection and targeted fallback
- Candidate answers are selected using a lightweight QEA-style selection module.
- Targeted fallback generation is used to rescue low-confidence or generic answers.
-
External Memory Agent
- The memory/state module filters redundant, irrelevant, or contaminated turns.
- It keeps only question-relevant textual states instead of storing the full visual history.
-
Clean-Light-Plus filtering
- Obvious generation artifacts are removed using a global rule-based filter.
- No test labels, hidden reply spans, leaderboard feedback, or sample-specific manual edits are used.
Repository Files
The repository includes:
1README.md
2requirements.txt
3HumOmni_V4_FINAL_K1_MEMORY_AGENT_TOP6_CLEAN_PLUS_FULL500.ipynb
4patch_mmd duet2_inference.py
5clean_light_plus_cell.py
6submission_generation_notes.md
7export_modified_inference_notes.md
If a fully patched local inference.py is exported from the Colab runtime, it can also be uploaded as modified_inference.py for convenience.
Environment
The system was run on Google Colab Pro with an NVIDIA A100 GPU.
Main dependencies are listed in requirements.txt. The environment used the public MMDuet2 codebase with the provided inference patch and a Qwen2.5-VL-based visual-language inference setup.
Reproducibility
To reproduce the final submission:
- Prepare the official HumOmni Track 2 Phase 1 test data.
- Clone or prepare the public MMDuet2 repository.
- Apply the provided inference patch / modified inference script.
- Run
HumOmni_V4_FINAL_K1_MEMORY_AGENT_TOP6_CLEAN_PLUS_FULL500.ipynb.
- Generate the
memory_agent_top6 output.
- Run the Clean-Light-Plus post-processing cell.
- Verify the final JSONL format using the final audit cell.
The official prediction JSONL is submitted separately through the official Google Drive prediction folder.
Notes on Fairness and Test Usage
The submitted system does not use hidden test labels, ground-truth reply spans, or manual video-specific edits. The final filtering rules are global, deterministic, and applied uniformly across all test samples.
