MementoGUI: Learning Agentic Multimodal Memory Control
for Long-Horizon GUI Agents

TL;DR

Long-horizon GUI control is bottlenecked not by single-step perception, but by active management of long-term multimodal state. MementoGUI reframes this as an online memory-control problem: a plug-in controller, MementoCore, decides when to update memory, what to preserve, how to compress, and when to retrieve — without finetuning the underlying GUI backbone.

Abstract

Recent GUI agents have made substantial progress in visual grounding and action prediction, yet they remain brittle in long-horizon tasks that require maintaining task state across many interface transitions. Existing agents typically rely on raw history replay or text-only memory, which either overwhelms the model with redundant screenshots or discards localized visual evidence needed for future decisions. To address these limitations, we introduce MementoGUI, a plug-in agentic memory framework that equips MLLM-based GUI agents with MementoCore, a learned controller for online memory selection, compression, and retrieval. Rather than treating interaction history as a fixed context, MementoGUI formulates long-horizon GUI control as an online memory-control problem: working memory selectively preserves task-relevant interface events with textual summaries and ROI-level visual evidence, while episodic memory retrieves reusable past trajectories through learned relevance selection. MementoCore modularizes memory control into specialized operators for step processing, memory compression, episodic writing, and episodic selection, enabling plug-in memory augmentation without finetuning the GUI agent backbone. We further develop a scalable data curation pipeline that converts computer-use trajectories into memory-controller training data, introduce MementoGUI-Bench for evaluating long-horizon decision-making in GUI agents, and design MLLM-based metrics for semantic action matching, task progress, and memory consistency. Experiments on GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench show that MementoGUI consistently improves GUI agents over no-history, history-replay, and text-only memory baselines, with larger MementoCore backbones further strengthening memory-augmented GUI control.

MementoCore: Four Memory-Control Operators

MementoCore attaches four task-specific LoRA adapters to a shared frozen Qwen3-VL backbone. Each adapter plays one well-defined role in the online memory loop; together they turn raw interaction history into decision-oriented multimodal context.

Data Curation Pipeline

We automatically convert raw computer-use videos into structured supervision for all four operators. The pipeline parses each trajectory into frame-level and subgoal-level annotations, assembles SFT data for step processing, memory compression, episodic writing, and episodic selection, and finally constructs DPO preference pairs for the two online operators via rule-based corruption and VLM-judged filtering. On 200 human-validated trajectories, 197 are judged fully correct.

Figure: (A) Raw computer-use videos are parsed into hierarchical frame- and subgoal-level annotations. (B) These annotations are converted into SFT data for four MementoCore operators. (C) Step-processing and memory-compression samples are further corrupted and VLM-filtered to form DPO training pairs.

Main Results

We evaluate four frozen open-source GUI backbones and two proprietary generalist MLLM agents on GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench. MementoGUI consistently improves over no-history, raw-history, and text-only memory baselines across all backbones.

Metrics: AMS and trajectory success (%) on GUI-Odyssey, Step SR (%) on MM-Mind2Web, and VAM / TPS / MCS on MementoGUI-Bench (Gemini-3.1-Pro as judge).

Figure: GUI-Odyssey performance on UI-Venus-1.5-8B, broken out by trajectory-length bins. Gains from working + episodic memory grow with trajectory length, where raw history and text-only memory struggle the most.

Figure: Effect of episodic-memory bank size on GUI-Odyssey across frozen backbones. Larger banks generally improve trajectory success, indicating that reusable experience mainly benefits long-horizon completion.

Ablations

Visual grounding matters in working memory

Removing ROI reference images from the memory context (keeping the same learned memory controller) consistently hurts AMS, VAM, and MCS on both UI-Venus-1.5-8B and GUI-Owl-1.5-8B. Localized visual evidence is not redundant with textual summaries.

Learned episodic selection beats raw retrieval

Random episodic context and single-stage embedding retrieval both trail the full two-stage (retrieve + learned-select) pipeline. MementoGUI benefits from both ROI-level grounding and filtered episodic experience, rather than just adding more context.

Scaling MementoCore

With the memory architecture and frozen GUI backbone fixed, we scale the memory controller from 2B → 4B → 8B. Larger controllers generally improve long-horizon decision support, with the 8B variant reaching the strongest AMS for both UI-Venus-1.5-8B and GUI-Owl-1.5-8B on GUI-Odyssey, and the strongest VAM for GUI-Owl-1.5-8B on MementoGUI-Bench. MementoGUI therefore scales as a plug-in memory layer — one can replace the controller with a stronger variant without touching the action model.

MementoGUI-Bench

MementoGUI-Bench is an offline benchmark for memory-dependent long-horizon GUI decision making, derived from PSAI computer-use videos. It contains 200 trajectories and 6,953 steps (average 34.8 steps per trajectory). 80 trajectories are used for testing and 120 for accumulating episodic memory. The benchmark focuses on cases where the next action depends on accumulated state: delayed constraints, completed subgoals, or prior experience.

Alongside standard reference-based scores, we report three VLM-judged, memory-aware metrics:

Benchmark composition. 200 trajectories stratified by category (Browser / Computer) and difficulty (Easy / Medium / Hard).

Trajectory length. Distribution of action frames per trajectory across the 200-trajectory pool.

Training coverage — head. Top-15 applications in the memory-controller training pool.

Training coverage — long tail. Word cloud over the remaining 429 applications in the training pool.

MementoCore Working Visualization

To give a concrete sense of how the four adapters interact at inference time, our supplementary visualization traces one pass through the pipeline on a real trajectory: a frame is scored and ROI-localised by the Step Processor, tail-step entries are collapsed by the WM Compressor, a completed trajectory is committed by the Episodic Writer, and retrieved candidates are re-ranked by the Episodic Selector before the merged context is handed to the action backbone. The full per-stage prompts, multi-image inputs, and raw JSON outputs — for every stage and the backbone — are provided there.

BibTeX

@article{zeng2026mementogui,
  title   = {MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents},
  author  = {Zeng, Ziyun and Hua, Hang and Zou, Bocheng and Cai, Mu and Feris, Rogerio and Luo, Jiebo},
  journal = {arXiv preprint},
  year    = {2026}
}