Official code release for “Look Before You Leap: Context-Sensitive GUI Grounding for Boosting Automated Extended Reality (XR) Testing.”

Paper · Project page · Pipeline guide · Environment · Required assets · Model manifest · Release checklist

Orienter detects context-sensitive, user-interactable GUI elements in XR screenshots. The public pipeline has three explicit stages:

The repository contains source code, selected evaluation ground truth, and the APE-L(_D) checkpoint needed by the main detector. Dataset image payloads, private data, API credentials, experiment logs, generated predictions, and previous experimental results are deliberately excluded.

The offline control path and utilities are unit-tested. Maintainer validation has exercised one-image live paths for the default OpenRouter VLM, APE-L(_D), and the PII.5–PII.7 advisor loop without retaining generated artifacts in this public copy. Historical experiment outputs and validation result tables are not bundled.

A fresh-clone paper-scale inference reproduction still requires the external assets in docs/ASSETS.md, especially the dataset image payload, provider access, and a validated CUDA/APE environment.

The paper is published in PACMSE, Volume 3, Issue FSE, Article FSE127 (DOI 10.1145/3808134). Do not publish this directory until the repository license, third-party revision audit, and final public model/data links are complete. Machine-readable citation metadata is available in CITATION.cff.

The supported public environment targets Python 3.10 or later so it can use a currently supported Pillow release with the latest security fixes:

requirements.txt covers the release utilities and provider adapters. APE-L(_D) has a separate CUDA/PyTorch stack under approach/ovod/APE; the checkpoint is bundled at approach/ovod/APE/ape_d_model_final.pth and recorded in docs/MODEL_MANIFEST.md. The public tree intentionally excludes compiled APE extensions, build directories, and egg-info metadata, so build/install the APE extension inside the target CUDA environment before running Stage 2. See docs/ENVIRONMENT.md and environment.reference.yml for the tested historical environment, a public compatibility starting point, build commands, and verification boundary. The reference YAML is not a cross-platform lockfile or a security attestation. The historical combined dependency snapshot remains at approach/requirements.txt, but it is not a clean lockfile.

The default provider is OpenRouter. In this project workspace, use openrouter-run so the key is injected by the local credential wrapper and never written to source files, logs, or shell history:

For a public fresh clone outside this workspace, inject OPENROUTER_API_KEY through the runner's secret manager or process environment. Do not commit .env, shell history, logs, or generated manifests containing credential values.

The default profile uses openai/gpt-5.6-sol, the current flagship model configured for this release. Override it without editing code:

Paper-oriented profiles remain available through OpenRouter:

Profiles are defined in approach/config.py. Library callers can override the catalog with a YAML or JSON profile file; CLI users select a named profile and may override the default model through OPENROUTER_MODEL. The paper profiles preserve the evaluated identifiers for GPT-4o-2024-08-06, Claude 3.5 Sonnet, and Gemini 1.5 Pro. Provider catalogs may retire historical routes; if one is unavailable, use a custom profile or direct vendor access and record the substitution instead of silently changing the paper profile. The client records only the credential variable name in run manifests; it never writes the credential value.

The commands below show the complete public interface. They require the external files listed in docs/ASSETS.md. Keep screenshots, metadata, prompts, predictions, visualizations, and evaluation files outside the repository:

Build a question manifest from mounted screenshots:

Screenshot filenames must match <numeric_app_id>_<numeric_frame>.<supported_ext>, for example 123_4.jpg; additional underscores are not supported. The frame must be in [0, 999] and is zero-padded to three digits when deriving image_id. Fill every generated app_name and app_description field in $ORIENTER_RUN_DIR/app_metadata.json before Stage 1; blank or missing records fail closed.

Validate the mounted inputs and the metadata coverage:

Generate candidate descriptions:

--app-metadata-cache accepts JSON or JSONL records keyed by app_id and avoids live Steam page drift. When supplied, missing app IDs fail closed. Omit it only for exploratory runs where live Steam lookup is acceptable.

Stage 1 sends each source screenshot to the selected OpenRouter/provider model. When reflection is enabled, PII.5 additionally sends the bounding-box crop and PII.6 sends the annotated full scene. Do not run private, personally identifying, confidential, or otherwise unauthorized screenshots through a remote provider. Review the selected provider's retention and data-processing terms, obtain the necessary permission, and use an approved local/provider deployment when screenshots cannot leave your environment. Generated candidates, predictions, traces, and error files may also reveal image labels or local filenames; keep them outside the release repository.

Ground candidates with APE-L(_D):

The released checkpoint is SHA-256 verified before loading. A different checkpoint is rejected unless --trust-custom-checkpoint is supplied explicitly after independently verifying its source; PyTorch checkpoints are executable pickle-based artifacts and must not be loaded from an untrusted source.

Enable the paper reflection loop explicitly because it incurs additional LMM calls:

PII.5 sends the original scene and the bbox crop as separate images. The crop is neither enlarged nor flipped. PII.6 sends the full scene with current boxes, and PII.7 returns structured verification/refinement feedback. Rejected false positives are removed before the next detector call; verified detections are retained.

Use either a range or deterministic modulo shards; the two selection modes are intentionally mutually exclusive. Output filenames receive a selection suffix automatically, so parallel jobs cannot overwrite each other.

Add --resume to continue from an existing output. Progress sidecars record completed images even when a valid inference produces zero detections.

Merge shards only after validating overlap and coverage:

The merge command fails on overlapping shard progress, duplicate detections, missing sidecars, or incomplete expected image coverage.

Convert prediction fields explicitly for each task:

--task accepts semantics, interactable, interaction, or all; all writes separate outputs and never overwrites one task with another. The remaining metric scripts and expected ground-truth layouts are documented in PIPELINE_README.md.

Run the release-critical offline suite from the repository root:

The deterministic control-path smoke uses a generated image plus injected provider/detector functions; it does not call a paid API or load APE weights. The unit tests are also offline. The final release gate additionally requires a real one-image inference, reflection smoke test, and evaluation smoke test after the external materials are supplied.

approach/method.py, method_fastuse.py, method_claude.py, and method_gemini.py preserve historical branches and commented stage variants so experiment provenance is not lost. Their active APE paths now produce valid accumulated JSON and accept range/shard controls; method.py and method_fastuse.py also accept explicit data/checkpoint/output paths and reflection flags.

For new runs, prefer run_vlm.py and run_ape.py. They make stage selection, paths, model profiles, app metadata caches, ranges, and shards configurable instead of requiring source edits.

Dataset images, annotations, bundled weights, and vendored third-party code may use different licenses. Publish every external asset with its stable URL, version, SHA-256, license, and expected local path. See docs/THIRD_PARTY.md and docs/MODEL_MANIFEST.md.

The paper is published in Proceedings of the ACM on Software Engineering, Volume 3, Issue FSE, Article FSE127, pages 2858–2881, in 2026:

CITATION.cff contains the same preferred paper citation. The top-level code license and final Hugging Face repository links remain publication decisions; licenses for the paper, data, model, and vendored source are separate and must not be inferred from one another.