Optimizing Streaming Memory Design via Open-World Auto-Research

An LLM editor diagnoses the recurring failures of a streaming video harness, reads papers, model cards and code, and rewrites its memory.

Ce Zhang1*, Jianshu Zhang2*, Jinxi He1, Xiyuan Yang3,
Pan Wang4, Han Liu2, Katia Sycara1, Yaqi Xie3

1Carnegie Mellon University 2Northwestern University 3University of Illinois Urbana-Champaign 4University of Pittsburgh *Equal contribution
75.1%OVO-Bench +3.1
78.9%StreamingBench +2.7
71.6%StreamBench +6.3
0.30sper answer 6.0× faster

Numbers above: one frozen model, Qwen3-VL-8B; gains over the next-best evaluated harness.

01 / The idea

Stuck on a failure? Read the literature.

Closed-world editors repair a harness from what they already know. MORSE's can search, read, and ask for tools.

LIVE video stream PipelineHarness memory + retrieval Frozen VLM answer ResearchersLLM Editor redesign it by handrewrites the harnessresearches, then rewrites redesignresults editstraces Hand-crafted rules compressretrievesummarize write Models & tools fixed CLIPOCR no new tools Tools grow reviewed CLIPOCR + action+ flow requests Open worldsearch, read, acquire PapersarXiv ModelsHugging Face CodeGitHub closed search & readfindings
02 / The loop

Diagnose. Research. Propose. Validate.

One turn of the loop, replayed from the editor's log: a recurring failure leads to a new tool.

1

Diagnose

Group the failures by cause.

2

Research

Search papers, model cards, code.

3

Propose

Write candidates. Request tools.

4

Validate

Keep what improves held-out streams.

The MORSE optimization loop. Left: the harness writes observations to memory and retrieves evidence for the frozen VLM. Middle: the editor diagnoses, researches, proposes and validates. Right: papers, model cards and code inform program edits and tool requests, and approved tools become callable after host integration. Below: accuracy rounds followed by efficiency rounds.
03 / The run

25 rounds. First more accurate, then faster.

Every tested candidate of the Qwen3-VL-8B run. Below, drag through all 25 rounds.

Drag to any round. Left alone, it moves on by itself. keptpassed the gatedid not

Open the paper's figure of the run.

04 / The final harness

The final memory mechanism, evolved over 25 rounds.

What the final harness holds at question time for three OVO-Bench questions, and what it sends to the frozen model. Click any item.

stored frameevictedsent to the modelstep-log entry

Open the paper's figure of these three examples.

05 / Quantitative results

Accuracy across models and benchmarks.

Each group shares one frozen model. Pick a model and a benchmark.

MODEL
BENCHMARK

Per-task numbers

† our reimplementation. On ODV-Bench (driving videos, unseen in development): 67.1%, against 62.6% for MemEvo and 58.6% for the model alone.

It builds over the run

HarnessRTBTOverall
Seed78.452.867.4
After round 380.061.372.0+4.6
After accuracy rounds82.764.875.0+7.6
Final82.465.375.1+7.7

Research and tools both matter

VariantRTBTOverall
Seed78.452.867.4
No tool requests78.959.770.6+3.2
No online research80.862.472.9+5.5
MORSE82.465.375.1+7.7

It transfers to other models

Model / methodRTBTOverall
LLaVA-OneVision-2-8B61.546.054.8
+ MemEvo†74.852.865.3+10.5
+ MORSE81.058.071.1+16.3
Gemma-4-26B-A4B71.757.165.4
+ MemEvo†77.562.070.8+5.4
+ MORSE82.665.175.1+9.7

OVO-Bench. RT: real-time perception. BT: backward tracing.

06 / Efficiency

0.30 s per answer, 6.0× faster than MemEvo.

Ask a question: each bar grows for as long as that system takes.

0.00 s
00.51.01.52.0 s
VLM callsother models
07 / Cite

BibTeX

@article{zhang2026morse,
  title  = {MORSE: Optimizing Streaming Memory Design via Open-World Auto-Research},
  author = {Zhang, Ce and Zhang, Jianshu and He, Jinxi and Yang, Xiyuan and Wang, Pan and Liu, Han and Sycara, Katia and Xie, Yaqi},
  year   = {2026}
}