NVIDIA FlashDreams Is the Open-Source Inference Engine Robotics World Models Needed
FlashDreams is NVIDIA's Apache 2.0 inference framework for real-time autoregressive world models. Built for robotics synthetic training data and sim-to-real pipelines, it delivers 68–103 FPS closed-loop generation with OmniDreams on GB300, and ships first-party integrations for OmniDreams, Cosmos-Predict2.5, and Self-Forcing.
On this page
The bottleneck in robotics training is not model architecture — it is data, and the speed at which you can generate it.
World models promise photorealistic training scenes on demand: a robot grasping a mug from any angle, a humanoid navigating a cluttered apartment, a warehouse arm sorting packages under varied lighting. NVIDIA has been building this stack for years — GR00T-Dreams for synthetic trajectory generation, Cosmos for world foundation models. But the autoregressive generation pattern these models use — where each frame depends on the previous one — creates a latency bottleneck that standard inference stacks were not designed to handle. A world model you cannot run in real time is a world model you cannot train with.
FlashDreams is NVIDIA’s answer. Released as an open-source Apache 2.0 project, it is a high-performance inference and serving framework built specifically for interactive autoregressive world models. It began as the runtime behind the closed-loop OmniDreams driving demo at GTC 2026 and has since become a general platform for real-time world-model applications across robotics, autonomous driving, and simulation.
Why Robotics Specifically Needs This
Synthetic data generation for robotics has a hard real-time constraint. When a policy network sends a motor command to a simulated humanoid, it needs visual feedback fast enough that the training loop stays stable. Excessive generation latency corrupts temporal consistency — the simulator must keep up with the policy, or the policy learns noise.
Existing inference frameworks optimize for batch throughput on independent requests. They load a model, process a prompt, generate an output, and unload. World models do the opposite: they maintain persistent state across thousands of sequential steps, update KV caches continuously, and accept per-frame control inputs that alter the generated scene. FlashDreams was designed for exactly this streaming pattern.
For the open-source robotics community, this matters because it removes the infrastructure barrier that previously separated published world models from practical deployment. A researcher can now take NVIDIA’s own Cosmos-Predict2.5 or OmniDreams weights, run them through FlashDreams, and generate interactive training environments at real-time rates — OmniDreams reaches 68–103 FPS in closed-loop mode on GB300 hardware (range reflects single- vs. multi-GPU configurations) — without writing custom CUDA kernels or managing KV cache growth.
How FlashDreams Solves the Streaming Problem
FlashDreams rethinks inference around four architectural decisions that directly address robotics use cases.
Bounded Memory for Infinite Rollouts
Standard autoregressive generation accumulates KV cache state indefinitely. Over a 10,000-step training rollout, that exhausts GPU memory and crashes the session. FlashDreams maintains a per-rollout cache with bounded VRAM usage, meaning you can run arbitrarily long simulations — whether training a humanoid to walk or a warehouse robot to sort packages — without memory growth. The framework handles cache eviction and state management internally.
Plugin Architecture for Model Portability
FlashDreams uses entry-point-based model discovery. Third-party packages register runner configurations that appear automatically in the CLI. This means a robotics lab can integrate its own world model — perhaps a custom diffusion model trained on proprietary manipulation data — without forking the framework or patching core code. The repository already ships first-party integrations for eight model families, and the developer guide for adding new methods is documented and public.
Multi-GPU Context Parallelism
Larger world models do not fit on consumer GPUs. FlashDreams supports automatic sharding across ranks via torchrun, scaling from one to eight GPUs with a single CLI flag. On OmniDreams, moving from one GPU to eight cuts diffusion model latency nearly in half while preserving frame-to-frame coherence. For robotics labs with multi-GPU workstations, this means world models previously reserved for data centers can now run on local hardware.
Hot-Path Generation, Background State Updates
Each autoregressive step splits into generate() — the latency-critical diffusion denoising and decoding — and finalize() — KV cache updates and world state maintenance. The separation allows finalize() to run on a background thread, so cache bookkeeping never blocks the next frame. In a closed-loop robotics simulator, this means the policy receives visual feedback on a predictable schedule regardless of how complex the world state has become.
The Open-Source Model Stack You Can Run Today
FlashDreams ships with integrations for multiple model families. The five most relevant to robotics and embodied AI are:
- NVIDIA OmniDreams — The HDMap-conditioned driving world model from GTC 2026. It takes one real camera frame, a text prompt, and per-frame trajectory data to generate photorealistic multi-camera driving video in closed loop. Steering input alters the generated scene in real time — this is the model powering NVIDIA’s interactive driving demo. Requires ~48 GB VRAM.
- Cosmos-Predict2.5 — NVIDIA’s latest world foundation model, unifying text-to-world, image-to-world, and video-to-world in a single flow-based network. Available in 2B and 14B sizes, post-trained on a 200-million-clip corpus for robotics and autonomous vehicle tasks. Requires ~56 GB VRAM for the 2B variant and ~80 GB for the 14B.
- Self-Forcing — An autoregressive video diffusion model built on Wan2.1 that closes the train-test gap by simulating inference rollouts during training with KV caching for efficient streaming. At ~24 GB VRAM, it is the most accessible integration for researchers without data-center GPUs.
- LingBot-World — A camera-controllable image-to-video model for environments where viewpoint matters.
- Causal-Forcing and Causal Wan2.2 — Streaming variants optimized for causal video generation, including FastVideo’s 14B mixture-of-experts architecture.
Every integration includes documented runner slugs, multi-GPU commands, VRAM requirements, and profiling benchmarks.
The Robotics Pipeline FlashDreams Enables
A researcher training a humanoid policy for household manipulation would typically spend months collecting real-world demonstrations. With FlashDreams, the flow looks different:
- Synthetic scene generation — Cosmos-Predict2.5 or a custom world model registered through FlashDreams produces photorealistic kitchen environments with varied lighting, object arrangements, and camera angles.
- Closed-loop policy training — Motor commands from the policy feed back into the world model in real time, with FlashDreams maintaining frame rates high enough for stable reinforcement learning.
- Sim-to-real transfer — The resulting policy transfers using established pipelines like NVIDIA Isaac Lab and GR00T-Dreams.
NVIDIA’s Physical AI Data Factory Blueprint, unveiled at GTC 2026, uses exactly this world-model-as-data-layer pattern for humanoid skills and autonomous driving. FlashDreams is the open-source inference engine that makes it accessible outside NVIDIA’s internal infrastructure.
Getting Started
FlashDreams is available on GitHub under the Apache 2.0 license. The repository assumes an NVIDIA GPU with at least 24 GB of VRAM for smaller integrations like Self-Forcing, and 48–80 GB for OmniDreams and Cosmos-Predict2.5. You will need CUDA 13.x, PyTorch 2.11 or newer, and the uv package manager.
The fastest path to a running world model:
git clone https://github.com/NVIDIA/flashdreams.git
cd flashdreams
uv sync --extra runners
export HF_TOKEN=<your-hf-token>
uv run flashdreams-run self-forcing-wan2.1-t2v-1.3b --total-blocks 7For the interactive driving demo — which streams a live-generated world to your browser and accepts steering wheel or game controller input — FlashDreams includes interactive-drive. It requires no display server or Vulkan support, only a CUDA-capable GPU.
What This Means for Open-Source Robotics
Before FlashDreams, the open-source robotics community had access to world model weights but not the infrastructure to run them interactively. Turning Cosmos or OmniDreams checkpoints into a real-time simulator required custom streaming pipelines, manual KV cache management, and debugging CUDA memory growth across long rollouts.
FlashDreams solves that by releasing the same runtime NVIDIA uses internally under Apache 2.0, with documented APIs and a plugin system designed for third-party extensions. For researchers building sim-to-real pipelines, it closes the gap between a model that exists in a paper and a model you can actually train with.
FlashDreams is available at github.com/NVIDIA/flashdreams. Full documentation and model-specific integration guides are at nvidia.github.io/flashdreams.
