NeurIPS 2026Main Track Poster

Space-Aware World Models

Spatial Persistence Through Factored Scene Representations

01 Introduction

When an imagined trajectory returns to a previously visited state, a world model should render the same place. We call this requirement spatial persistence. World models predict observations under actions, allowing agents to anticipate the consequences of their behavior without further interaction with the environment [1, 2]. In many recurrent state-space models [2, 3], transformers [4] and diffusion-based models [5], scene layout, appearance and agent motion are carried together in an evolving predictive representation. Over a long rollout, accumulated errors can change the scene that this representation describes. Individual frames may remain plausible while a familiar corridor looks different when the agent returns.

SWM architecture: persistent scene memory and agent state feed the observation renderer, while actions drive state transitions.
Figure 1. SWM combines a fixed scene code z with the changing agent state to render observations. The transition model updates the state using scene context and actions.

Space-Aware World Models (SWM) are generative models that separate persistent scene memory from agent dynamics (Figure 1). A scene code z is encoded from a top-down map or sampled from the prior to generate a new environment, then held fixed throughout the rollout. An action-conditioned transition model evolves the explicit agent state st, and a renderer produces observations from (z,st). Training uses map-like observations and low-dimensional state supervision, with scene layouts assumed static within each episode. We study this setting with compact models trained on a single workstation.

02 Space-Aware World Model

Let m denote a top-down map, st the agent state and at the action. With state and action encoders πs and πa, SWM defines three components:

Scene encoderz∼qϕ(z∣m),(1)
Observation modelo^t=Gθ(z,πs(st)),(2)
Transition modelst+1=st+Tψ(πs(st),z,πa(at)).(3)

A rollout starts with a scene code, obtained by encoding a map once or sampling from the prior to generate a new environment. The model then alternates between rendering an observation and updating the agent state (Figure 2). The scene code remains fixed throughout. Autoregressive prediction acts only on the explicit agent state, such as position and heading.

SWM rollout with a fixed scene code, action-conditioned state transitions and observation rendering at successive steps.
Figure 2. An SWM rollout with a shared scene code. Each transition updates the agent state; each observation is decoded from the state and scene codes.

2.1 Scene Encoder

The scene encoder is a variational autoencoder [6] that maps a top-down image to a compact scene code. The shared encoder maps held-out layouts into the same latent space without per-scene optimization, unlike the scene-specific fitting used by methods such as NeRF [7]. Its prior also allows new scene codes to be sampled. Figure 3 illustrates the scene-code manifold through a projection onto the first two principal components.

Scene-code projection, latent coordinates and decoded map along a path through encoded maps and prior samples.
Figure 3. Encoded maps and prior samples in scene-code space: projection onto two principal components (left), selected latent coordinates (centre) and decoded map (right).

The scene prior lets SWM generate new environments without an input map: a sampled code can be decoded into a map and used directly by the observation and transition models (Figure 4).

SWMTop-down trajectory
Figure 4. Rollout in a scene sampled from the prior, z∼𝒩(0,I). Left: generated observations. Right: trajectory on the decoded map.

2.2 Observation Model

The observation model Gθ is a deterministic renderer conditioned on the scene code and current state. A Fourier feature transform [8] and a linear projection map the state to πs(st)∈ℝd, the same dimensionality as the scene code. The two pathways meet through bounded additive fusion,

ft=αz+(1−α)πs(st),α∈[ε,1−ε],
(4)

which the decoder 𝒟θ maps to the observation o^t=𝒟θ(ft). The mixing weight α is a learned scalar shared across scenes within each domain. The bound ε keeps both fusion weights nonzero, discouraging reliance on state alone. Training combines a pixel reconstruction loss with a perceptual loss [9].

For a fixed scene code, the same state always produces the same rendered observation. The learned dynamics can still accumulate state drift. Before decoding, error separates into a constant scene term α(z^−z⋆) and a time-varying state term (1−α)(πs(s^t)−πs(st⋆)). This is a decomposition in fusion space, not in image pixels. Rendering at the ground-truth state removes the second term and defines the SWM-oracle variant in Section 5.4. Figure 5 shows the renderer along a recorded trajectory, using only the scene code and the agent’s pose, represented here by (x,y,sinθ,cosθ).

Simulator observation and SWM render at the same pose, alongside the decoded map and state inputs.
Figure 5. Simulator observation (centre) and SWM rendering (right) at the same pose. Left: decoded map, agent pose and state inputs.

03 Transition Model and Wall Contacts

The transition model uses the scene code as context to predict action-conditioned state increments:

Δst=Tψ(πs(st),z,πa(at)),st+1=st+Δst.
(5)

The explicit state describes the agent, while the scene code z provides the geometric context for Tψ. The same state–action pair can therefore produce different increments in an open region and near an obstacle. Predicting increments is a natural fit for these domains, where consecutive motions are small relative to the environment. It also makes rollout error interpretable as gradual mislocalization in physical state space.

Figure 6 probes whether the transition model uses this scene information. Starting from the first recorded state of an episode held out from transition-model training, we run Tψ under the recorded actions without correcting its predictions. With the episode’s own scene code, the predicted trajectory follows the recording, including stops at wall contacts. Replacing only the scene code with that of a different map causes the prediction to diverge at the first wall contact.

Recorded trajectory and predicted rollouts with correct and mismatched scene codes, alongside speed, heading and position error.
Figure 6. Free-running trajectories with the correct scene code (blue) and another map’s code (violet), compared with the recording (grey). Crosses and shaded columns mark wall contacts. Right: speed, heading and position error.

The transition model learns motion and wall contacts from recorded transitions, without explicit collision checks or map lookups (Section 04). The longer held-out trajectories in Figure 7 follow local map structure and obstacle-induced turns while gradually accumulating state drift.

Top-down maps of three held-out episodes with ground-truth and SWM autoregressive trajectories.
Figure 7. Ground-truth and autoregressive SWM trajectories on held-out layouts.

04 Data Collection with Fast Sweeping

For the DeepMind Lab (DMLab) [10] navigation experiments, we collect training rollouts with a map-guided procedure that requires no pretrained exploration policy. A random policy can remain in a small part of a maze and provide limited coverage. To collect longer traversals, we guide the agent with the episode’s top-down map.

We first segment each map into a binary occupancy grid, separating free space from walls. After sampling a waypoint g in free space, we compute the geodesic travel time T from each free cell to g by solving the eikonal equation

|∇T(x)|=1in free space,T(g)=0,
(6)

with walls excluded from the domain. The fast sweeping method [11, 12] solves this equation using Gauss–Seidel upwind updates in four alternating sweep orders. Each cell is updated from the smallest neighboring values along the two grid axes. Figure 8 illustrates how successive sweeps propagate travel times around obstacles.

The negative gradient −∇T points along a shortest path through free space to the waypoint, bending around walls. We convert this direction into environment actions using the agent’s current pose. Small stochastic perturbations encourage varied trajectories and broader state-space coverage. When the agent reaches the waypoint, we sample a new one and recompute the field.

Fast-sweeping data collection with the agent observation, travel-time field and recorded trajectory.
Figure 8. Observation (left), travel-time field to the waypoint (centre) and map with the descent route and recorded trajectory (right).

These trajectories cover long traversals and wall contacts, providing examples of how scene geometry constrains motion (Figure 6).

We train the three modules sequentially: the scene encoder on maps, the observation model on tuples (z,st,ot) with precomputed scene codes, and the transition model on action-conditioned state sequences with z as context.

05 Experiments

5.1 Experimental Setup

We evaluate on two procedurally generated domains with held-out layouts. DMLab tests revisitation and return-to-origin consistency in first-person 3D navigation. CarRacing [13] tests whether generated observations preserve the track information used by a controller. Training uses 2 × 104 episodes, roughly 107 frames, with one top-down image per episode. We compare SWM with compact implementations of four latent-dynamics families: an MDN-RNN world model [1], an IRIS-style transformer [4], DreamerV3 [3] and LeWM [14]. All systems use the same rollout data and action sequences, with workstation-scale training budgets. SWM additionally receives a top-down map and state supervision. These cross-family comparisons therefore evaluate the complete structured approach; they do not isolate the effect of factorization under equal input information.

5.2 DMLab: First-Person Navigation

We evaluate long imagined rollouts on held-out layouts. State-match LPIPS [9] measures perceptual agreement between generated and real observations at matched physical states, including revisits separated by long temporal gaps. The return-to-origin benchmark tests place identity with full panoramas captured at the start and after an out-and-back excursion. Origin-match LPIPS compares each returned view with the real start view at the same yaw. Recall@1 tests whether the returned panorama retrieves its own episode from a gallery of real start panoramas. Yaw error measures the angular offset of the best circular match.

Table 1. DMLab results on 100 held-out rollouts of 500 steps. Panorama metrics use 36-view panoramas separated by a 250-step out-and-back excursion. Rel. Imp. compares SWM with the best baseline per metric. ± denotes standard deviation across episodes.
MetricMDN-RNNTransformerDreamerV3LeWMSWMRel. Imp. (%)
State-match LPIPS ↓0.208± 0.1150.168± 0.1400.269± 0.1850.407± 0.1420.033± 0.061+80%
Origin-match pano LPIPS ↓0.545± 0.0980.523± 0.0980.593± 0.0700.613± 0.0940.298± 0.117+43%
Origin pano Recall@1 ↑0.070± 0.2550.100± 0.3000.000± 0.0000.010± 0.0990.770± 0.421+670%
Pano yaw error (deg) ↓74.6± 61.579.3± 63.678.4± 68.489.5± 57.731.4± 25.4+57%
PRDC Density ↑0.06590.14620.00570.02530.3710+154%
PRDC Coverage ↑0.10240.20780.00860.04020.5288+154%
ECS Inlier Ratio ↑0.21760.23960.05210.28780.3507+22%

SWM better preserves local appearance and place identity on revisits, with closer perceptual matches, more reliable panorama retrieval and lower yaw error than the latent-dynamics baselines (Table 1). Higher PRDC density and coverage [15] also suggest that this consistency is not achieved by collapsing to a small set of plausible frames. Geometric agreement improves as well: a larger fraction of SIFT correspondences [16] agrees with the known camera motion, as measured by the ECS inlier ratio using epipolar agreement [17].

Figure 9 compares the out-and-back excursion with stationary control, where the model receives no-op actions while the ground-truth pose remains fixed. Both regimes use the same panorama protocol. SWM remains stable under stationary control and degrades moderately after the excursion, consistent with a retained scene code and accumulated transition drift. Several baselines perform poorly even under stationary control, suggesting that path integration alone does not explain their errors: scene or pose information can drift within an evolving predictive state. Figure 10 shows the return panorama of one held-out episode; Figure 11 compares imagined rollouts under the same actions.

Origin-match panorama LPIPS, Recall@1 and yaw error for five models in the out-and-back and stationary-control regimes.
Figure 9. Return-to-origin panorama consistency on held-out layouts after an out-and-back excursion (solid) and under stationary control (hatched).
Real and generated return panoramas with per-pixel error maps.
Figure 10. Real and generated return panoramas after an out-and-back excursion, with per-pixel error maps.
RealSWMMDN-RNNTransformerDreamerV3
Figure 11. Real observations and imagined rollouts under the same actions.

5.3 CarRacing: Continuous Control

In CarRacing, we evaluate generated observations with a frozen pixel-based controller that was never trained on generated frames. SWM’s generated views lead to action distributions closer to those from real rollouts and better preserve the controller’s steering decisions on paired frames than the latent-dynamics baselines (Table 2). These results show that the observations retain information used by a fixed controller; they do not evaluate policy learning inside SWM.

Table 2. Frozen controller probe in CarRacing over 5 × 104 evaluations. Dream action statistics compare action distributions on imagined and real rollouts; matched-frame agreement compares decisions on paired observations.
Dream action statisticsMatched-frame agreement
ModelAction W1 ↓Action JS ↓Coupling err. ↓Action err. ↓Steer sign ↑Steer corr. ↑
SWM0.023± 0.0340.002± 0.0010.0070.655± 0.4690.9580.891
LeWM0.077± 0.0960.019± 0.0200.2091.367± 0.9080.6400.127
Transformer0.085± 0.0770.018± 0.0140.0931.423± 0.7040.7550.091
DreamerV30.063± 0.0540.014± 0.0120.0971.446± 0.9980.6270.147
MDN-RNN0.104± 0.0980.023± 0.0180.2051.467± 0.7890.624-0.025
Top-down CarRacing frames over time for the real rollout and five models.
Figure 12. Real and generated observations along a held-out CarRacing rollout.

5.4 Horizon Diagnostics

Figure 13 shows how frame fidelity changes with rollout length in both domains. The SWM-oracle variant uses the same scene code and renderer as SWM, but renders at the ground-truth state instead of the predicted state. This removes accumulated transition drift while retaining scene encoding and rendering errors. SWM degrades more slowly than the baselines, and the flatter oracle curves identify state-transition drift as a major source of its remaining long-horizon error.

LPIPS, DISTS, MSE and SSIM against rollout horizon in DMLab and CarRacing for five models and the SWM oracle.
Figure 13. Frame fidelity over rollout length in DMLab (top) and CarRacing (bottom). Lower is better for LPIPS, DISTS and MSE; higher is better for SSIM. SWM-oracle renders at ground-truth states.

5.5 Matched Scene and State Information

Cross-family comparisons cannot isolate the effect of factorization from the additional scene and state information available to SWM. To test the representation directly, we compare it with an entangled-latent control given the same pretrained scene code, initial state, actions, state supervision, data, renderer, sequence length and training budget, with comparable recurrent capacity. The control folds scene and state into one evolving code instead of keeping scene memory fixed. Both methods have similar cumulative LPIPS at shorter horizons, with the control slightly ahead at an intermediate horizon. SWM retains lower error over longer rollouts and higher panorama retrieval accuracy under repeated no-op updates (Table 3). The control’s loss of place identity without physical motion supports the role of a persistent, separately accessible scene representation.

Table 3. Matched-information comparison in DMLab. The entangled control reports mean ± standard deviation over three training seeds. Stationary metrics follow 250 no-op updates and a 36-view panorama sweep.
MetricSWMEntangled-latent control
H = 10 cumulative LPIPS ↓0.08610.0882± 0.0003
H = 200 cumulative LPIPS ↓0.16310.1581± 0.0014
H = 500 cumulative LPIPS ↓0.22560.3341± 0.0026
Out-and-back Recall@1 ↑0.7700.260± 0.050
Stationary origin-match LPIPS ↓0.26150.5408± 0.0105
Stationary Recall@1 ↑0.8900.053± 0.015

06 Inference Speed

Each rollout step requires one transition of the low-dimensional state and one rendered frame. The scene code is initialized once, from a map or the prior, so the per-step cost does not grow with the rollout history. Figure 14 shows how throughput changes with the number of parallel environments. GPU batching supports many simultaneous rollouts while maintaining real-time performance per environment. The CPU also supports real-time rollouts at smaller parallel batch sizes.

SWM frames per second per environment and total throughput as the number of parallel environments increases, on GPU and CPU.
Figure 14. DMLab inference throughput with a state update and 96 × 96 render per step. Left: frames per second per environment. Right: total throughput. GPU: RTX 3090 Ti, bfloat16, outputs kept on the GPU. CPU: i7-12700K, 12 threads, float32.

07 Limitations

The quantitative evaluations use top-down maps and explicit state supervision; prior-sampled rollouts illustrate scene generation qualitatively. The cross-family baselines receive observations and actions, so those comparisons combine the effects of structured information and model design. They do not establish an advantage for generic video prediction, unknown-state settings, changing layouts or learning without map-like supervision.

A fixed scene code does not eliminate autoregressive error. The oracle diagnostics identify state-transition drift as a source of the remaining long-horizon degradation. The current formulation also assumes a compact scene code, static geometry within each episode and a single agent with a low-dimensional state. Open directions include inferring scene codes from partial observations, extending scene memory to larger or unbounded environments, modeling dynamic scenes and multiple agents, and learning policies directly within SWM rollouts.

References

  1. Ha, David; Schmidhuber, Jürgen. Recurrent World Models Facilitate Policy Evolution. Advances in Neural Information Processing Systems (NeurIPS), 2018.
  2. Hafner, Danijar; Lillicrap, Timothy P.; Ba, Jimmy; Norouzi, Mohammad. Dream to Control: Learning Behaviors by Latent Imagination. International Conference on Learning Representations (ICLR), 2020.
  3. Hafner, Danijar; Pasukonis, Jurgis; Ba, Jimmy; Lillicrap, Timothy. Mastering Diverse Control Tasks through World Models. Nature, 2025.
  4. Micheli, Vincent; Alonso, Eloi; Fleuret, François. Transformers are Sample-Efficient World Models. International Conference on Learning Representations (ICLR), 2023.
  5. Alonso, Eloi; Jelley, Adam; Micheli, Vincent; Kanervisto, Anssi; Storkey, Amos; Pearce, Tim; Fleuret, François. Diffusion for World Modeling: Visual Details Matter in Atari. Advances in Neural Information Processing Systems (NeurIPS), 2024.
  6. Kingma, Diederik P.; Welling, Max. Auto-Encoding Variational Bayes. International Conference on Learning Representations (ICLR), 2014.
  7. Mildenhall, Ben; Srinivasan, Pratul P.; Tancik, Matthew; Barron, Jonathan T.; Ramamoorthi, Ravi; Ng, Ren. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Computer Vision – ECCV 2020, 2020.
  8. Tancik, Matthew; Srinivasan, Pratul; Mildenhall, Ben; Fridovich-Keil, Sara; Raghavan, Nithin; Singhal, Utkarsh; Ramamoorthi, Ravi; Barron, Jonathan; Ng, Ren. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems (NeurIPS), 2020.
  9. Zhang, Richard; Isola, Phillip; Efros, Alexei A.; Shechtman, Eli; Wang, Oliver. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  10. Beattie, Charles; Leibo, Joel Z; Teplyashin, Denis; Ward, Tom; Wainwright, Marcus; Küttler, Heinrich; Lefrancq, Andrew; Green, Simon; Valdés, Víctor; Sadik, Amir; et al. DeepMind Lab. arXiv preprint arXiv:1612.03801, 2016.
  11. Zhao, Hongkai. A fast sweeping method for eikonal equations. Mathematics of Computation, 2005.
  12. Sethian, James A. Level set methods and fast marching methods. 1999.
  13. Towers, Mark; Kwiatkowski, Ariel; Terry, Jordan; Balis, John U; De Cola, Gianluca; Deleu, Tristan; Goulão, Manuel; Kallinteris, Andreas; Krimmel, Markus; KG, Arjun; et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024.
  14. Maes, Lucas; Le Lidec, Quentin; Scieur, Damien; LeCun, Yann; Balestriero, Randall. Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026.
  15. Naeem, Muhammad Ferjad; Oh, Seong Joon; Uh, Youngjung; Choi, Yunjey; Yoo, Jaejun. Reliable fidelity and diversity metrics for generative models. International Conference on Machine Learning (ICML), 2020.
  16. Lowe, David G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004.
  17. Hartley, Richard; Zisserman, Andrew. Multiple view geometry in computer vision. 2003.