arXiv 2611.25415

Stuck in PlaceStall Diagnosis and Remedies for JEPA Planners

JEPA world-model planners fail on pushing tasks mostly by stalling: the object stays in place while the model predicts progress, and the warm-started planner returns nearly the same plan. Unstick breaks this fixed point without test-time weight updates.

Junhyeok LeeDongheon LeeKyu Sung Choi

Seoul National University

Latent costs and frames of one PushObj-T episode for the base GD planner and Unstick-Mem
Latent costs and frames of one PushObj-T episode. Top: base GD planner; bottom: Unstick-Mem (weights fixed at test time). Solid: actual latent cost; dashed: predicted after each chunk; arrow: predicted drop per chunk; gray band: stall. Frames: last step; blue: agent; red: agent path during the stall; dashed outline: goal.
≥ 96%
of the base planner's failures are stalls in 12 of 15 pushing settings, with and without distribution shift.
0 of 101
analyzed stalls are search failures. The model overrates staying in 98% of stalls on T.
53.3 → 78.7%
PushObj-T success, base → Unstick-Mem (GD). AdaJEPA, with test-time weight updates: 84.7%.
64.0 → 80.7%
PushT success on clean frames, base → Unstick-Mem. AdaJEPA: 84.0%.
Abstract

Joint-embedding predictive architecture (JEPA) world models plan from pixels with model predictive control, yet often fail on tasks their model should handle. Test-time adaptation attributes such failures to model error under distribution shift and updates the weights on the current episode, without checking this attribution against the planning loop itself. We follow the executed planning loop on pushing tasks. In most settings, with and without distribution shift, nearly all failures (at least 96%) are stalls, where the object stays in place while the model predicts progress. The warm-started gradient-descent (GD) planner then returns nearly the same plan, a fixed point where the agent mostly idles. We propose Unstick, which pairs on-policy fine-tuning before deployment with a fixed-weight loop breaker: a residual memory of the episode's prediction errors or random re-initialization at every replan. With GD, Unstick recovers most of the adaptation gain on pushing tasks with the memory, and with re-initialization most of it on shape shifts but much less on visual shifts. A large part of what limits these planners under GD is thus a fixed point of the planning loop, present with and without distribution shift, which a planner with fixed weights can break.

Diagnosis

Failures are stalls, and stalls are a fixed point

We follow the executed planning loop of the base gradient-descent (GD) planner, replaying executed actions in the simulator to recover true states.

Failure analysis of the base GD planner: stalls, calibration and plan ranking
Failure analysis of the base GD planner. (a) Idle: agent moves <2 px per MPC step. (b, c) The 101 stalls at MPC step 25 (T, clean); (c) labels give counts. Objective trap: no simple plan lowers the simulated cost of staying by ≥5%; misranked: a simple plan lowers the simulated cost, but the model prefers staying.
a

The object stays, the model sees progress

On T, I, square and clean, at least 97% of the failures are stalls. During them the model predicts a cost reduction of 0.030–0.095 per executed chunk, while the actual progress is zero.

b

Search is not the bottleneck

No analyzed stall is a search failure. On T, the model overrates staying in 98% of stalls and underrates the best simple plan.

c

Warm start makes it a fixed point

Warm-started GD returns nearly the same plan, and the agent moves back and forth: net displacement below 0.3 of the path length for 89–99% of GD chunks, against 3% of expert chunks on T.

Method

Unstick: break the loop, keep the weights

Fine-tuning before deployment corrects the model on the planner's own executions. At test time, a fixed-weight loop breaker removes the stall.

FT

On-policy fine-tuning

Before deployment, fine-tune the model on the base planner's own executions of separate episodes, because the planner visits states that offline data miss.

M

Residual memory

Store the model's one-step errors on executed chunks and add them back to predictions near them, so a re-proposed stay-in-place chunk is predicted to stay in place.

R

Random re-initialization

Start every replan from a random plan instead of the warm start, so the same state no longer yields the same plan.

Unstick-Mem = fine-tuning + residual memory  ·  Unstick-ReInit = fine-tuning + random re-initialization. Neither updates the weights at test time.
Results

Most of the adaptation gain, with fixed weights

With GD, Unstick recovers most of the gain of test-time adaptation on pushing tasks with the memory, and with re-initialization most of it on shape shifts but much less on visual shifts.

Success (%) ↑Seen shapesUnseen shapes
MethodTLZcrossIsmall Tsquare
Base53.3±6.850.7±6.644.7±5.052.0±4.326.0±5.742.7±5.223.3±3.4
+ Unstick-Mem78.7±5.782.0±4.378.7±3.877.3±3.840.7±1.968.7±3.836.0±3.3
+ Unstick-ReInit76.0±4.371.3±2.576.7±1.971.3±5.750.0±11.868.0±5.942.7±0.9
+ straight chunks55.3±0.949.3±1.945.3±2.548.0±4.922.0±4.342.0±14.222.7±3.8
+ uniform weights60.0±3.357.3±4.156.7±2.564.7±6.633.3±8.158.0±5.920.7±0.9
AdaJEPA84.7±5.786.7±3.879.3±5.078.7±1.948.0±8.665.3±3.436.7±2.5

Success on seen and unseen shapes (PushObj). GD planner, success (%). Mean ± SD over three evaluation seeds of 50 episodes. Bold, underline: best, second best without test-time weight updates; ties share marks. Straight chunks: each chunk's actions replaced by their mean; uniform weights: uniform weights on all predicted frames of the objective. AdaJEPA: test-time weight updates.

Success over MPC steps, averaged over seven shapes and seven visual conditions
Success over MPC steps. GD; averages over the seven shapes (left) and the seven visual conditions (right). Lines: mean over 3 seeds; shading: SD.

What neither remedy fixes

Failure cases that neither remedy fixes trace to an encoder that discards a task variable, as under the red block and darkening at level 0.5, or to local minima of the planning objective, as in PointMaze-Medium, rather than to the planner.

What it means for evaluation

JEPA planners should be evaluated in closed loop, by checking whether the state moves and whether the plan repeats, before their failures are attributed to distribution shift. Test-time adaptation should be compared with remedies that keep the weights fixed.

Qualitative

First episode per outcome

First episode per outcome on PushObj, PushT and PointMaze for the base planner and the two remedies
First episode per outcome (GD, seed 1). Each triplet: base planner, Unstick-Mem, Unstick-ReInit (PointMaze: base planner, residual memory, random re-initialization). Last frame; agent path, start (○) to last step (●); green: block goal; star: maze goal; ✓ k: success at MPC step k; ✗ idle k: idle from MPC step k on; ✗: fails.
Citation

BibTeX

lee2026stuck
@article{lee2026stuck,
  title   = {Stuck in Place: Stall Diagnosis and Remedies for {JEPA} Planners},
  author  = {Lee, Junhyeok and Lee, Dongheon and Choi, Kyu Sung},
  journal = {arXiv preprint arXiv:2611.25415},
  year    = {2026}
}