Recursive Video In-Context Learning for Agentic Robot
The agent gets one demonstration video. It reads an outline before planning and opens one clip whenever a step needs detail.
- No training
- Frozen VLA
- One demonstration per task
- 96.5%
- LIBERO-PRO
+3.9 over HarnessVLA - 95.8%
- LIBERO-Plus
up from 86.7% - 97%
- Re-entry
share of episodes
A demonstration the agent can navigate
Each demonstration is indexed offline as keyframes plus an event index. The agent reads both once, then returns for one clip at a time.
Read once before planning
Keyframes and the event index. No clip yet.
view_task_videoview_task_mapRe-entry during execution
One clip of the current sub-goal.
view_demo_clipNo fixed way of showing a video fits the agent’s context
Text memory can record that a grasp failed. It cannot show how a grasp is done. A video can.
Length
Hundreds of frames, re-read at every turn.
Resolution
A few keyframes skip the frames around each contact.
Timing
Structure before acting, contact frames only at that step.
One demonstration: keyframes and an event index
Pick a phase or a moment to see the clip the agent would get.
phase 1 [0,62) open: approach phase 2 [62,109) closed: opens flat_stove_1 phase 3 [109,203) open: retreat phase 4 [203,262) closed: carries moka_pot_1 into flat_stove_1_burner phase 5 [262,272) open: hold moments: grasp 1 (62), open flat_stove_1 (100), release 1 (109), grasp 2 (203), release 2 (262), end (271) goal: (And (Turnon flat_stove_1) (On moka_pot_1 flat_stove_1_cook_region))
One recorded episode with the agent’s tool calls
Playback pauses where the agent opened the demonstration. The clip it asked for plays beside the robot.
phase 2 [62,109) gripper closed: opens flat_stove_1 phase 4 [203,262) gripper closed: carries moka_pot_1 into flat_stove_1_burner moments: grasp 1 (62), open flat_stove_1 (100), release 1 (109), grasp 2 (203), release 2 (262)
The agent’s message appears here.
Seven clips in this episode. A clip that comes back was requested again.
3.9 points on top of the strongest agent on LIBERO-PRO
Eight cells, 50 episodes each. Two-proportion z-test: z = 2.97, p = 0.003.
LIBERO-PRO by cell
The gains sit in the three cells where HarnessVLA is weakest.
Same video, different access
Recursive access: up to +12. Other forms: at most +2.
LIBERO-Plus
120 variants, 86.7% to 95.8%.
Clip views per episode
97% of the 400 episodes re-enter the demonstration.
Why, in 100 sampled re-entries:
What the agent asks for
Share of all clip views, %.
Four more episodes, at twice the speed
BibTeX
@article{bao2026rvicl,
title = {Recursive Video In-Context Learning
for Agentic Robot},
author = {Bao, Wenrui and Liu, Xinxin and
Xu, Bingxin and Shang, Yuzhang},
journal = {arXiv preprint arXiv:2610.06843},
year = {2026}
}