Real2sim2real terrain

Reconstruction produces meshes that look right; a physics simulator needs terrain that behaves right. The hard part is getting from one to the other. Small processing choices can quietly change what the robot sees—for example, smoothing out a sharp edge can teach the policy that the edge isn’t there.

Those errors carry through everything downstream. A terrain that is 5 cm rougher than it should be can topple a robot that would have walked it, while a terrain that is too smooth can train a policy that falls over when exposed to the real environment.

1. 01

   Make the model export succesfully

   LingBot-Map turns ordinary RGB video into camera poses and dense 3D point clouds, frame by frame, without LiDAR or offline reconstruction. Its demo doesn’t save any of the results, so I built a wrapper that exports the point cloud, camera trajectory, and run data. That makes each reconstruction reproducible and gives us something we can actually measure and compare.

   I memory-mapped the checkpoint and cast the aggregator to bf16 before GPU transfer, reducing 4.6 GB of weights to 2.81 GB of GPU memory. This made the model runnable on an 8 GB GPU.

2. 02

   Find the inference path the README doesn’t show

   Upstream ships two pipelines, but the documented one is not the one used for the best results. The published demos came from a streaming mode with windowed visual odometry and adaptive optical-flow keyframing, forced to run at a fixed interval. That option only exists in one of the two implementations and has no command-line flag. I exposed it so the published results could be reproduced.

   The wrapper adds controls for the flow threshold and maximum keyframe gap, then records how often keyframes actually occur using the model’s own keyframe mask.

3. 03

   Map the operating envelope instead of guessing it

   On an 8 GB GPU, the limits are clear: 24 keyframes per window uses 6.21 GB at peak, while 32 keyframes runs out of memory. The system can process about 660 frames before CPU memory becomes the bottleneck. But hardware wasn’t the main limit. After about 25 seconds of video, the reconstruction starts to drift, and both streaming and windowed modes hit the same limit. The bigger difference came from the environment. Indoor scenes and simple ground planes reconstructed well enough to use, while snow-covered alpine terrain failed completely. With mostly white surfaces, there isn’t enough visual detail for the model to match features between frames, so reconstruction breaks down before hardware becomes the problem.

   Drift is measured by comparing the distance the camera travels to the size of the area it reconstructs: 2.76 at 25 seconds, 9.7–11.8 at 49 seconds, and 31–46 at 132 seconds. Meanwhile, streaming stays at a steady 5.63 GB of VRAM across a 660-frame clip. That stable memory usage is what makes long, live reconstruction possible.

4. 04

   Score it against the ground truth

   Using assumed camera height to set scale is just a guess, and that guess affects every distance in the final terrain. Instead, I benchmarked the reconstruction against survey-grade GNSS/INS data collected by a legged robot on real terrain, so the error is measured in meters against a real reference. The benchmark uses alpine rock and gravel, where the model has enough visual detail to work. The same terrain under snow is where reconstruction completely fails, which is why the survey data becomes the fallback there.

   The benchmark converts the released camera model to a standard pinhole model and accounts for the camera’s 0.417 m offset from the robot. On alpine rock and gravel, the reconstruction reached a 1.27 length-to-area ratio, with 1.168 m of trajectory error over a 23.0 m path.

5. 05

   Cleaning the point cloud in Open3D

   I used Open3D to filter, downsample, calibrate scale, and align the ground, taking 4.07M points down to 964k usable points. I then converted them into a 5 cm heightfield that a posed humanoid could stand on, with just 0.3 mm of foot penetration. This phase allowed me to come across 3 unexpected findings, with each changing the terrain the policy sees.

   We found three things that changed how the terrain had to be processed. First, using the highest point in each area works well outdoors, but indoors it turns walls and furniture into unrealistic spikes. Second, when the camera never sees part of the terrain, we fill that area with a flat surface instead of guessing what is there, and save a mask showing which areas were actually observed. Finally, smoothing the terrain is important for the robot: just 5 cm of depth noise on flat ground was enough to make it fall. Because of that, the amount of smoothing is saved with each terrain model.

6. 06

   Build the terrain the policy actually trains on

   Because reconstruction stays reliable for only about 25 m, I used a national elevation survey to build the training terrain instead. The survey has a 0.5 m grid covering 5.4 km of alpine trail with 713 m of descent. I turned it into a continuous terrain corridor, built the mesh, and exported it to USD with exact triangle-mesh collision. The terrain comes from the same rock-and-gravel environment used to benchmark the reconstruction, so the training and evaluation terrain represent the same type of ground. For snowy terrain, where the camera cannot get enough visual detail, the next step is to start with survey geometry and add visual reconstruction on top of it rather than relying on images alone. The reconstruction side is still being evaluated. Niantic Spatial's large geospatial model is the next model on the bench, using the same drift, operating-limit, and ground-truth tests used here.

   Before training starts, a validation check makes sure the robot starts on the trail and that all observations are valid. The terrain metadata also records that no extra small-scale roughness was added.

• Built the reconstruction wrapper to export results, load the model with memory-mapped bf16 weights, and expose the keyframing controls needed to reproduce the quality of the published results.
• I mapped out where the model works and where it fails. On an 8 GB GPU, memory becomes a limit at larger windows, but for longer videos, drift becomes the real problem. The model works well on structured indoor scenes and simple ground, but struggles on white-on-white alpine snow (our use case), where there isn’t enough visual detail for reliable reconstruction.
• Built the ground-truth benchmark harness, scoring reconstruction by absolute trajectory error against survey-grade GNSS/INS (GPS-based positioning and motion data).
• Built the Open3D cleanup pipeline to turn raw reconstructions into simulator-ready heightfields, including three findings that changed the terrain the policy actually trains on.
• Built the 5.4 km alpine trail terrain from 0.5 m survey elevation data, along with the validation gate that checks it before training.

Trajectory errorover 23.0 m, against GNSS/INS reference

1.168 m

Drift, alpinelength over extent; 24.9 on a hard urban scene

1.27

Cleaned cloudfrom 4.07M

964k pts

Heightfield572 × 357

5 cm cells

Foot penetration

0.3 mm

Streaming VRAMflat across 660 frames

5.63 GB

Consistent scenedrift-limited, not memory-limited

25 s

Trail terrain713 m descent at 0.5 m grid

5.4 km

I built the pipeline that turns real trail footage into usable simulation terrain, including reconstruction, cleanup, benchmarking, and Isaac Sim integration. This work lays the groundwork for training robots on realistic terrain.

LingBot-Map · Open3D · Isaac Sim · Isaac Lab · USD · PyTorch · swissALTI3D

• Repository
• LingBot-Map (Robbyant, Apache-2.0)
• Niantic Spatial LGM — next under evaluation