SYNWorld V0.5

01Why we start with scaling

Choosing scaling as a research and development approach

We have chosen to build learning systems that can keep using more data and compute. In The Bitter Lesson, Sutton emphasized the long-term value of general learning methods that scale with computational resources. For embodied intelligence, experiments must determine whether the next stage should expand model size, add data, or introduce more modalities and tasks.

When designing the system, we considered how to bring in new data and modalities, and how to support training with larger models and longer sequences. This allows later expansion to use the existing data workflows and training systems.

Scaling is anisotropic

Expanding models, data, modalities, and task coverage can all bring benefits, but the same investment does not produce the same capability gains in every direction. We need experiments to find the direction that offers the greatest return at the current stage, or the steepest scaling slope.

We are pursuing four directions:

• Model capacity: make the model larger so it can learn more complex patterns of manipulation.
• Data scale: add high-quality training data so the model can learn different manipulation processes more thoroughly.
• Modality coverage: expand from vision and action to signals such as touch, giving the model information about contact states and related conditions.
• Tasks and scenarios: cover more combinations of objects, environments, and tasks so the model can handle situations it has not encountered during training.

We consider these four directions together, comparing improvements in actual task performance while assessing data quality and the costs of training and inference. As model capabilities and data distributions change, the direction most worth investing in will also change.

Data to support scaling

First-person human activity data (Ego) capture a range of everyday actions and scenes, making them an important source for expanding our data. These videos record multi-finger coordination, contact adjustments, and tool use, from which the model can learn how human hands complete tasks. To lower the barriers to collection, we developed an EMG-based multimodal data collection system that reconstructs hand poses and estimates tactile information from wristband signals. We plan to use Ego data that include EMG-derived hand poses in foundation model training.

Different data sources provide different forms of supervision. Videos of natural activities help the model learn about objects, scenes, and motion, while Ego data provide close-up records of human hand manipulation. However, human hand poses and trajectories cannot be used directly as robot control commands, and video alone cannot fully recover contact information. Real-robot data provide direct action supervision, but collection is more expensive and the data are usually tied to a particular robot embodiment. We combine these data for pretraining, then use real-robot data for post-training on specific tasks.

Integrated hand-and-brain design

Many tools and workspaces are designed for human hands, and many complex tasks require in-hand adjustments, multi-point contact, and multi-finger coordination. We have therefore committed to dexterous hands as the primary means of performing a broad range of tasks in everyday work and life, pursuing an integrated hand-and-brain approach that optimizes data, models, and hardware together.

In development, we use task requirements and hardware capabilities to determine which hand states to collect and which actions the model should output. We then test model predictions through actual execution and use the results to improve the data, model, and hardware. The infrastructure supporting this integrated hand-and-brain design has been used in developing SYNWorld V0.5, and we will publish a fuller account of this joint development work later.

02Data systems: infrastructure for continued scaling

Scaling Engine connects data preparation, training inputs, and distributed training to support the continued expansion of models and data. The data system organizes heterogeneous compute into composable pipelines, turning growing volumes of raw data into traceable, reusable training inputs.

Data recipe and coverage

SYNWorld's current data pyramid moves from broad coverage to task depth: general images and videos provide world knowledge, visual structure, and temporal dynamics; first-person Ego data cover human behavior, interactions, and long-horizon tasks; UMI data connect states, actions, and outcomes; and real-robot task data focus on precise manipulation, difficult examples, and capability limits. The training data total 300,000+ hours, currently growing by 10,000+ hours per day, and cover 1K+ scenarios and 10+ robot embodiments.

Composable distributed pipelines

Our Ray-based distributed pipeline organizes decoding, filtering, annotation, and encoding into composable stages, each with independently configured resources and concurrency. Sample states and outputs are persisted independently, allowing processing models, execution tasks, and compute resources to evolve separately.

Optimizing compute, networking, and storage together

Heterogeneous hardware and operator optimization selects operators, precision, and batch configurations for each GPU architecture; sensitive operators fall back to higher precision, with errors checked against reference outputs. Network optimization combines topology awareness with processing near the data to reduce transfers, serialization, and memory copies; storage and I/O optimization combines sharding and batched reads and writes, local caching, and asynchronous prefetching to overlap data movement with compute.

Elastic execution and continuous autotuning

Adaptive flow control and elastic compute let data processing and training share a resource pool, adjusting concurrency and scaling data workers according to queue demand and available resources, while backpressure controls the volume of intermediate data. Autotuning runs trials of batch size, concurrency, prefetching, and hardware configurations, then applies more efficient settings after precision and quality requirements are met.

Agent-assisted monitoring and operations correlate logs, resource metrics, and sample progress to diagnose bottlenecks and orchestrate experiments; experiment histories and validated configurations form persistent context, allowing long-running tasks to continue across sessions.

03Training systems: scaling multimodal models

The training layer of Scaling Engine organizes diverse tasks into efficient distributed computation, allowing model size, sequence length, and training scale to grow together. System-level optimizations to the training infrastructure deliver a 10× improvement in overall training efficiency.

A composable, unified training framework

The component-based framework decouples models, modalities, conditions, losses, and optimization, combining multiple tasks and training paradigms through configuration. Frozen encoding branches reuse offline features and caches whose input and encoding configurations match; branches that need updates and augmentation of raw inputs use online processing and augmentation. The two paths are combined for each task and share the same pretraining and post-training loops.

Joint sampling, variable-length training, and load balancing

Joint sampling dynamically controls the mix of data sources, tasks, and modalities; variable-length training reduces padding through length grouping and sequence packing while preserving sample boundaries, masks, and positional coordinates. Shard Balance assigns workloads across ranks using sequence length, masks, and compute costs, and dynamically adjusts micro-batches to a token budget; synchronization steps are aligned across ranks, and losses are normalized by global target weights.

Custom attention masks × CP × Varlen

Optimized attention operators jointly support custom structured masks, context parallelism (CP), and variable-length sequences (Varlen). Sample isolation, cross-modal visibility, and each sample's positional coordinates remain consistent after partitioning; mask-aware scheduling, fused computation, and overlap of KV communication with compute support long-sequence training together.

Multidimensional parallelism and execution efficiency

TP, PP, EP, CP, and HSDP are combined according to model structure, cluster topology, and implementation compatibility. BF16 / FP32 precision policies and selective activation recomputation, together with state sharding, control memory use; fused operators, compilation optimizations, parameter prefetching, and overlap of compute and communication improve training efficiency.

Asynchronous saving and reliable operation

Async Save first stages a consistent snapshot before allowing training to modify the corresponding state; the checkpoint is committed after all shards and metadata have been written in the background. Recovery loads the model, optimizer, scheduler, random state, and sampling progress, and supports resharding when loading at a different scale. Anomaly detection and performance diagnostics correlate losses, gradients, slow ranks, and communication state, continuously tracking MFU, effective tokens/s, and step time.

04SYNWorld V0.5

Handling language, vision, and action in one architecture

The architecture of SYNWorld V0.5 accounts for four kinds of differences: language, vision, and action use different representations; video and action have different sampling rates; data sources contain different action fields; and tasks have different conditions and prediction targets. To address these differences, the model uses a Mixture-of-Transformers (MoT) architecture. Action and vision retain their own modality-specific parameters and exchange information through cross-modal attention; the language model encodes descriptions and task instructions as per-layer K/V to condition action and visual generation.

Actions are transformed into a fixed-length, 59-dimensional representation after coordinate transformation and normalization, as detailed below:

Aligning modalities in physical time

Video frames, VAE-compressed latent frames, and action sequences usually have different sampling rates. The model uses three-axis mRoPE to retain time, height, and width coordinates for visual tokens and assign time coordinates to action tokens; both are mapped to a shared timeline according to their sampling rates. This allows attention to relate action and visual changes within the same interval of physical time.

Unifying learning tasks through conditions and targets

The model uses flow matching to learn continuous visual and action generation. For targets that need to be predicted, data and Gaussian noise are interpolated at noise level $\tau$:

The model predicts the velocity along the path from data to noise, then integrates from the noise end toward the data end during generation. Observations or actions used as conditions stay unchanged, and the loss is computed only for valid target elements that need to be generated. Vision and action have separate output heads and losses.

This interface allows different tasks to share the backbone: visual generation targets images or videos; action-conditioned prediction takes future actions as input to predict future vision; joint generation produces future action and vision together. The choice of conditions and targets determines where noise is added, validity masks determine where losses apply, and attention rules separately control visibility.

From visual generation to joint action and visual learning

Pretraining has two stages. The first, GV (Generation of Video), learns visual generation and temporal changes from general images and videos; the second, GVA (Generation of Video and Action), keeps the visual interface and generation objective, introduces Ego, UMI, and robot data, and adds action prediction. The GVA checkpoint is the shared starting point for post-training on different downstream tasks.

Let $z^v$ denote visual latents and $z^a$ action sequences, with $M^v$ and $M^a$ their binary masks for valid targets. $X$ denotes all action and visual inputs for the current task, while $c$ denotes observation, language, or action conditions. When constructing $X_{\tau }$, conditions stay clean and each generation target is noised according to $z_{\tau }^m=(1-\tau )z^m+\tau {ϵ}^m$. The velocity head for modality $m$ reads the shared input $X_{\tau }$, and its loss is computed only over that modality's valid targets; $\delta >0$ prevents a zero denominator:

The two stages differ in input organization, generation targets, and conditions. $X_{\mathrm{GV}}$ contains only visual inputs; $X_{\mathrm{GVA}}$ contains both vision and action, with the two targets interacting within the same forward pass and their losses computed separately.

Starting from the GVA checkpoint, separate post-training runs produce WAM and ACWM. WAM generates future actions and visual predictions from observation history and task conditions; ACWM predicts environmental changes from past observations and future actions. The two paths share their pretrained foundation, then update their parameters and save checkpoints independently.

WAM post-training retains the joint objective. $X_{\mathrm{WAM}}$ contains observation history and the future actions and vision to be predicted, with the history kept clean and the future targets jointly denoised:

ACWM training uses real future actions as conditions to predict the corresponding future vision. If the sequence is divided into K temporal chunks, $X_k$ targets vision in chunk k; $c_k$ contains the initial observation, visible real visual history $z_{<k}^v$, given actions, and language. During inference, the visual history is replaced with previously generated results:

Inference efficiency: reducing repeated computation and supporting chunked generation

WAM inference includes multiple denoising steps through the action-vision Transformer and decoding of video latents. In the baseline configuration, one Transformer step takes 263 ms, and 30 denoising steps take 7,934.61 ms in total. Compilation, FP8 computation, and CP4 parallelism reduce the per-step time to 64.63 ms, a 4.07× speedup; two-step DMD2 then reduces full denoising to 131.28 ms, a 60.44× speedup over the baseline 30-step process. Compilation, width sharding, CUDA Graph, and Winograd optimization reduce video decoding time from 1,418.14 ms to 149.64 ms, a 9.48× speedup.

05Evaluation: does pretraining improve downstream task capability?

We use three types of evaluation to assess how pretrained capabilities transfer to downstream tasks:

• Initialize networks with pretrained models and perform WAM post-training with different numbers of samples on real-robot tasks, comparing task performance and data efficiency;
• Perform multitask WAM post-training in the LIBERO-Plus simulation environment, comparing success rates under different perturbations;
• Compare the base models' video generation and world prediction performance through double-blind human preference tests.

Real-robot tasks: post-training with different data budgets

The real-robot evaluation covers four tasks: placing utensils, pouring beans, stacking plates, and folding clothes. Each task uses 50, 100, 200, or 400 demonstration trajectories for post-training, comparing three pretrained models: SYNWorld V0.5, π0.5, and Cosmos3 Nano. The main metric is the reliable task utility score (0-100), calculated as a weighted combination of substep scores and the overall task completion rate.

Across all comparable data budgets for all four tasks, SYNWorld V0.5 scores higher than π0.5 in reliable task utility; at the budget of 400 trajectories, it also scores higher than Cosmos3 Nano on every task.

LIBERO-Plus: standardized simulation evaluation

LIBERO-Plus contains 10,030 task cases with external perturbations in seven categories: Background, Camera, Language, Light, Layout, Robot, and Noise.

SYNWorld V0.5 achieves an overall success rate of 87.78%, compared with 85.66% for Cosmos3 Nano and 85.41% for π0.5. Under this evaluation setup, the post-trained model initialized from SYNWorld V0.5 achieves a higher overall success rate.

Video generation: double-blind human preference evaluation

We also evaluate the base model's ability to model the visual world. Using Cosmos3 Nano as the baseline, we generate videos under identical conditions and randomize their order, then ask multiple reviewers to compare 60 randomly sampled pairs in a double-blind test across three dimensions: semantic adherence, visual quality, and physical consistency.

Across these 60 pairs, Good votes outnumber Bad votes in all three dimensions. Net preference rates are 46.7% for semantic adherence, 26.7% for visual quality, and 51.7% for physical consistency.

WAM demonstrations: four real-robot tasks

The following videos show SYNWorld V0.5 WAM results after post-training on 50 demonstration trajectories, covering object localization and placement, continuous pose control, alignment and contact with targets, and multistage manipulation of deformable objects.

ACWM demonstrations: predicting the future under action conditions

The following two videos show SYNWorld V0.5 ACWM post-training results: the input is actions performed by a human in real time, and the output is the corresponding video prediction. As the human's two-arm poses change, the model advances its predicted frames with the actions while maintaining the main object relationships. The videos also show good temporal continuity and visual consistency.

06Next steps: continued scaling and evolution

SYNWorld V0.5 establishes a foundation for joint visual and action pretraining and has demonstrated the ability to learn a range of real-robot manipulation tasks from a few demonstrations. The Scaling Engine supporting its development also provides infrastructure for further growth in data and model scale.

Our goal is for a model to complete a broad range of tasks in everyday work and life from a few demonstrations. To achieve this, we still need to introduce modalities such as force and touch so the model can learn a fuller account of physical interactions, while continuing to expand data, models, and training scale to improve its ability to handle complex tasks. Next, we will develop dexterous hands, multimodal data collection, and model training together, using real tasks to test the gains from each step of expansion.

Hand-and-brain coordination: joint iteration around real tasks

We will improve our in-house dexterous hands and models together around tasks that require multi-finger coordination, in-hand adjustments, and complex contact. We will optimize Scaling Engine for our own dexterous hands, for example by retargeting human hand actions in Ego data to the corresponding hardware. We will also integrate software and hardware into a productivity feedback loop, using execution feedback from real tasks to guide hardware design and optimization.

Multimodal data: expanding EMG-based collection

We will continue expanding our EMG-based multimodal data collection system to acquire large-scale multimodal manipulation data from real scenes. Combining vision, force, and tactile information will add contact states and force changes that are hard to observe directly in video, helping the model understand how actions affect object motion and why the same action can produce different outcomes under different contact conditions.

Continued scaling of intelligence: expanding scale and testing capability growth

We will continue expanding models, data, and training scale, as well as modality and task coverage. By comparing model performance on real tasks and the number of demonstrations needed to learn them, we will evaluate the benefits of different expansion directions. We will seek more effective directions under equal investment, gradually improving the model's ability to learn complex tasks from a few demonstrations.

Scaling Engine: exploring a more autonomous research and development loop

We will continue expanding Scaling Engine and explore using an agent harness to support more autonomous research and development iterations. The goal is for agents to connect experiment planning, data processing, training execution, and result analysis, then use the results to adjust the next round of experiments. We want agents to combine past experiments with current results to reduce repetitive research and development work and speed up validation of model and data approaches.