Real2sim2real terrain
2026
Work
Video in, a simulator a policy can be trained in out. Streaming monocular reconstruction, Open3D cleanup, and a USD collider — with the failure modes measured rather than assumed, down to the environment classes where monocular reconstruction stops working at all.
- 1.17 m
- 964k
- 0.5 m
The problem
Reconstruction produces meshes optimised to look right; a physics solver needs colliders optimised to behave right. The interesting work is the translation, and the failures there are quiet — a resampling filter that smooths away a sharp edge does not throw an error, it just teaches a policy the wrong thing.
Everything downstream inherits those errors silently. A terrain that is 5 cm too rough topples a robot that would have walked; a terrain that is too smooth trains a gait that falls over the first real rock.
Approach
Make the model export anything at all
LingBot-Map is a streaming reconstruction model that turns ordinary RGB video into camera poses and dense point clouds frame by frame, with no LiDAR and no offline structure-from-motion. Its demo script writes no artifacts, so the wrapper around it emits the cloud, the trajectory and a run record itself — without which nothing downstream can be reproduced or compared.
Loading the checkpoint memory-mapped and casting the aggregator to bf16 before GPU transfer lands 4.6 GB of weights at 2.81 GB resident. Upstream's loader cannot start on an 8 GB card at all.
Find the inference path the README does not show
Upstream ships two pipelines, and the one documented is the weaker of the two. The published demo videos came from a windowed visual-odometry mode with adaptive optical-flow keyframing, forced at a fixed interval — a mechanism that exists in only one of the two streaming implementations and is exposed by no command-line flag. Re-exposing it made the published quality reproducible.
The wrapper adds flow-threshold and maximum-gap controls and reports the realised keyframe fraction, read from the model's own keyframe mask, as a run diagnostic.
Map the operating envelope instead of guessing it
On a single 8 GB card the limits are specific: 24 keyframes of window at 6.21 GB peak, with 32 out-of-memory; roughly 660 frames before CPU-side accumulation dominates. The binding constraint turned out not to be memory at all — past about 25 seconds of footage the reconstruction drifts, and both streaming and windowed modes hit that wall identically. Hardware is only one axis of the envelope. Run across environment classes, the model splits cleanly: structured indoor scenes and simple ground planes reconstruct well enough to build on, while snow-covered alpine surfaces collapse outright — white-on-white imagery starves the photometric matcher, and spatial estimation fails before drift or memory ever becomes the limit.
Drift is tracked as trajectory length over spatial extent: 2.76 at 25 s, 9.7–11.8 at 49 s, and 31–46 at 132 s. Streaming VRAM stays flat at 5.63 GB across a 660-frame clip, which is what makes a live pipeline possible at all.
Score it against real ground truth
Scale calibration by assumed camera height is a guess that propagates into every downstream metre. Benchmarking instead runs against a dataset that ships survey-grade GNSS/INS reference from a legged robot walked across real terrain, so reconstruction error is reported as absolute trajectory error in metres against a real reference rather than against an assumption. The benchmark terrain — alpine rock and gravel — is deliberately from the textured side of the envelope; the same slopes under snow are where reconstruction stops being an option at all, which is what forces the survey-data fallback below.
The harness rectifies the released camera model to an explicit pinhole and composes ground truth through the camera's 0.417 m lever arm. On alpine rock and gravel it reconstructs at 1.27 length-over-extent with 1.168 m of trajectory error over 23.0 m.
Clean the cloud, which is where the policy is decided
Filtering, downsampling, scale calibration and ground alignment in Open3D take 4.07M points to 964k metric ones, then to a heightfield at 5 cm cells that a posed humanoid settles on at 0.3 mm of foot penetration. Three findings came out of this stage that were not in the plan, and each changes the terrain a policy sees.
Robust max-z is the correct surface outdoors and turns every indoor partition into a spire. Unobserved cells are filled flat rather than interpolated, so the observation mask ships beside the asset. And smoothing is not cosmetic — roughly 5 cm of monocular depth noise on flat ground topples a posed robot, so the smoothing sigma is recorded in the asset metadata.
Build the terrain the policy actually trains on
Because reconstruction caps out near 25 m of consistent scene, the training terrain came from a national elevation survey at 0.5 m grid instead: 5.4 km of alpine trail with 713 m of descent, straightened into an arc-length by cross-track corridor, meshed, and exported to USD with exact triangle-mesh collision. It is deliberately the same rock-and-gravel terrain family as the reconstruction benchmark, so both halves describe the same ground. For the white-on-white domains, the working direction is geometry-first rather than photometric — depth-fusion or splatting-based reconstruction primed with survey elevation, and texture-augmented capture where imagery alone cannot anchor scale. The evaluation is live rather than settled: Niantic Spatial's large geospatial model line is the next backbone on the bench, judged by the same drift, envelope and ground-truth harness this one was scored with.
A gate asserts that spawn origins land on the trail and that observations are finite before any training run starts. The metadata records that no micro-roughness was added, because a terrain that claims detail it does not have is the same class of error as an unmarked placeholder.
Plates
Figure 4 from LingBot-Map (Robbyant), arXiv:2604.14141 — the upstream architecture, not this project's output (opens in new tab) 
Figure 2 from LingBot-Map (Robbyant), arXiv:2604.14141 — the environment classes the paper demonstrates; the wrapper's own probing is what mapped where they hold (opens in new tab) 
Figure 5 from LingBot-Map (Robbyant), arXiv:2604.14141 — trajectory accuracy as the paper reports it; the benchmark here re-scores it against GNSS/INS (opens in new tab)
What I built
- Wrote the reconstruction wrapper: artifact export, memory-mapped bf16 loading, and the re-exposed keyframing controls that made upstream's published quality reproducible.
- Mapped the model's full operating envelope — the 8 GB hardware ceiling, drift rather than memory as the binding limit on scene length, and the environment classes where it holds (structured indoor, simple ground planes) versus where white-on-white alpine snow defeats spatial estimation entirely.
- Built the ground-truth benchmark harness, scoring reconstruction by absolute trajectory error against survey-grade GNSS/INS instead of an assumed scale.
- Built the Open3D cleanup chain to simulator-ready heightfields, including the three findings that change what a policy is trained on.
- Built the 5.4 km alpine trail terrain from 0.5 m survey elevation data, and the gate that validates it before training.
Results
- Trajectory error
- 1.168 m
- Drift, alpine
- 1.27
- Cleaned cloud
- 964k pts
- Heightfield
- 5 cm cells
- Foot penetration
- 0.3 mm
- Streaming VRAM
- 5.63 GB
- Consistent scene
- 25 s
- Trail terrain
- 5.4 km
Stack
The reconstruction wrapper, the cleanup chain, the benchmark harness, and the terrain build. Deliberately a starting point: world generation for RL simulation is the area this work narrows toward, and the backbone evaluation continues past LingBot-Map.