RatatouGym

A simulation environment for neuroscience and embodied AI that directly models sensory encodings for ultra-fast simulation.

Demo Read the docs

Loading rooms

Idealized navigation simulation

2D trajectory

3D flight

Trajectory simulation

Simulate navigation trajectories in idealized environments to build neuroscience models and test navigation algorithms. [5] [6]

Directly simulating sensory responses

Sample response fields at each position and heading to directly simulate encodings of vision, position, heading, and boundaries. [4] This supports fast, highly parallel simulation of sensory responses without rendering images.

Response fields

Population response

Encoding sequence

Sensors

Room

Response fields

Encoding sequence

Sensor response space

Loading the demo…

Visually realistic navigation simulation

Cinema den

Timber bedroom

Billiards room

Rendering and encoding visual observations

Room geometry, camera pose, and navigation trajectory
Simulation
The front camera image in a sequence of captured views It
Camera images
Eθ
DINO / MAE / SAM3D
Vision encoder
Encoding
World modelTransformer
PolicyExternal model
Biological modelRecurrent network
Downstream models
Decoder
Slightly blurred camera image illustrating reconstruction, not a model output
Ît
Reconstruction

Scaling visual simulation for world models

World models need visual experience at scale. We target highly parallel simulation by predicting visual encodings directly from room geometry and learned features [1] [2] [3]. This skips per-step RGB rendering and vision-encoder inference, reducing the work needed to generate each training observation.

Mesh visibility

Camera pose pt

Visible surfaces

x, y, z + hit mask

Spatial features

XY / XZ / YZ · multiscale

Predicted encoding

Image latent

Loading the illustration…

Documentation

Read the docs Source code

References

  1. [1]
    Deferred Neural Rendering: Image Synthesis using Neural Textures

    Thies et al. ACM TOG / SIGGRAPH 2019.

  2. [2]
    Efficient Geometry-aware 3D Generative Adversarial Networks

    Chan et al. CVPR 2022.

  3. [3]
    Modular Primitives for High-Performance Differentiable Rendering

    Laine et al. ACM TOG / SIGGRAPH Asia 2020.

  4. [4]
    Time Makes Space: Emergence of Place Fields in Networks Encoding Temporally Continuous Sensory Experiences

    Wang et al. NeurIPS 2024.

  5. [5]
    REMI: Reconstructing Episodic Memory During Internally Driven Path Planning

    Wang et al. NeurIPS 2025.

  6. [6]
    A Simple Model of Co-Emergence of Grid and Place Fields

    Wang et al. NeurIPS 2026.