Chatting with Wang Xiaofeng on Gigaworld-1: When World Models Become the Physical Referees for Embodied AI

Evaluating language models is almost instantaneous. Once training stops, you feed the test set to a GPU, and within seconds you get a string of scores—good or bad, it's clear at a glance.

The situation is completely different for the "brain" of a robot. When a VLA model iterates to a new version, researchers can't just evaluate it on a screen—they have to flash the new policy onto real hardware, arrange test props in the arena one by one, stand by, watch it execute repeatedly with their own eyes, and record each time until they've gathered enough samples. If things go smoothly, they get a score after one or two days.

The inefficiency of physical evaluation is slowing down the iteration of embodied foundation models, becoming the critical bottleneck that determines overall capability. This isn't an issue unique to one company; nearly all leading firms are paying a significant time cost for "evaluation." Consequently, a new approach has emerged in the industry: using algorithms to bypass the physical bottleneck. Since real machines are too slow, why not let a world model that truly understands physical laws serve as the judge?

Flexiv (Ji Jia Shi Jie) is one of the earlier teams to embark on this path. As early as 2023, they began working on world models for autonomous driving scenarios; last November, they released GigaWorld-0 for generating embodied training data; and this July, they took another step forward. Flexiv, in collaboration with Tsinghua University, published the report titled "GigaWorld-1: A Roadmap to Build World Models for Robot Policy Evaluation" and launched GigaWorld-1, a world model designed to evaluate robot policies. Developers can use this world model to determine whether a robot's policy is excellent.

The goal this time is clear: to formally embed world models as evaluators into the development workflow of Embodied AI, using digital referees to minimize reliance on physical hardware as much as possible.

Yesterday, we spoke with Wang Xiaofeng, algorithm partner at Flexiv and co-author of the report, who has deep experience in the fields of AI and world models.

In the conversation, he unpacked the mechanics behind "letting world models act as judges," frankly explained why current VLMs struggle with physical evaluation, and clearly delineated the future divergence between 3D and video generation approaches. More importantly, he offered an increasingly clear judgment: over the next three to five years, beyond the hype of benchmarking, the ultimate competition in this track will be about system capabilities that truly integrate data, models, and evaluation.

Here is a conversation between 42 Hertz and Wang Xiaofeng, slightly edited:

Why Position World Models as Evaluators

Q: This time, GigaWorld-1 positions the world model as an evaluator. What core judgments or practical needs led you to choose this direction?

A: Our company has been working on World Models for quite some time; we started in 2023, primarily focusing on autonomous driving. We have been developing evaluators for a long period. In the field of embodied AI, we previously had a data engine—specifically, last November, our GigaWorld-0 was used to generate data for training subsequent VLA (Vision-Language-Action) models. However, this was at the front end, i.e., the data generation side. On the evaluation side, we had not previously engaged in large-scale World Model-based evaluation.

Recently, however, while iterating on foundational models for embodied AI, such as VLAs and World Action Models (WAMs), we encountered a bottleneck: iteration efficiency was heavily constrained by the evaluation process. For instance, with language models, once a new version is trained, it can quickly be run on existing benchmarks using GPUs, yielding a score that allows developers to compare it with previous versions. In contrast, for embodied VLA or WAM models, determining whether a new version is better or worse requires submission to our benchmark team—the real-machine testing team. They must queue up, conduct tests, and provide feedback, which typically takes one to two days. Consequently, the overall efficiency is low.

Therefore, we needed to seek faster and more convenient evaluation methods. Given our extensive prior accumulation in World Models, both in autonomous driving and embodied AI, it was natural to leverage World Models for this purpose.

Another catalyst occurred in early June this year, when CVPR hosted a competition in Denver, USA, featuring a World Model Track. This track aimed to assess whether participants' world models could serve competently as evaluators. Through this competition, we exchanged insights with many participants, combined them with our internal iterative experience and know-how, and incorporated submitted data to support the experimental information behind our GigaWorld-1 technical report.

We aim to rigorously advance the concept of "World Model as Simulator" and "as Evaluator." Behind this lies the question of current capability boundaries: how can we push further to potentially replace real machines? Currently, our goal is merely to reduce reliance on real machines, not to completely replace them. Thus, this outlines our general developmental journey.

Q: You are currently reducing reliance on real robots. How does this improve time efficiency, and what are the key areas of improvement?

A: We treat real-robot evaluation as the final checkpoint, not every model deserves to reach that stage. Before testing on a real robot, I use my World Model to score it. Only when it reaches a certain level and is deemed truly OK and problem-free do we proceed to the real-robot phase.

Q: In the long term, could a World Model Evaluator completely replace real-robot evaluation? What are the main current gaps?

A: Currently, some tasks are moving towards fully replacing real-robot evaluation. A small number of tasks remain difficult to replace entirely. However, through overall iterative optimization, we hope to gradually reduce reliance on real robots.

In fact, we have internally integrated 'World Model as Evaluator' into our benchmark team's workflow. Previously, the benchmark team relied more on real robots and simulation. As we use it now, I can clearly see the gap between scores from the World Model evaluator and those from real-robot evaluation. This gap serves as a metric for further optimizing the World Model. My World Model will increasingly operate and optimize to align with real-robot performance.

At the same time, I can observe the role played by the World Model as Evaluator in assessing whether previous strategies were good. These two aspects complement each other: I can use it to reduce reliance on real robots and obtain scores; conversely, the gap with real-robot results becomes a target for optimization, enabling my World Model to better substitute for real robots in these tasks.

Q: So, do you ultimately believe it can 100% replace real robots, or will there always be a gap of over 90%, or perhaps an irreducible gap?

A: I believe it will reach over 90%, but it cannot completely replace real robots.

Q: If the World Model takes on a central role in the evaluation system, what changes do you think will occur in the overall R&D workflow of robotics companies?

A: I believe it will change. Because at the model level, it actually plays the role of a referee and assumes the function of a benchmark.

We have already started using it internally, which has reduced our reliance on real-world robots. With further reductions, we may even see clients coming to us for collaboration in this area. Once several leading companies in the embodied AI industry have successfully streamlined evaluations—effectively resolving the bottleneck in testing—it becomes evident that we can iterate base models more rapidly. This leads to stronger capabilities in our base models.

I think all other leading companies will follow this path to accelerate the iteration of their internal foundational models.

Q: In terms of practical results, by what extent has efficiency improved in the development process? Could you provide a more intuitive comparison in terms of time or workflow?

A: While it might not be convenient to provide highly specific details, I can give an overview of the current industry situation and how the pipeline generally works.

Typically, after the model team, algorithm team, and data team deliver a batch of data to the model team, training begins. After training, there may be internal ablation studies, resulting in two or three versions of the model. These are likely pre-trained models. Subsequently, post-training processes are applied to enable the models to perform embodied operational tasks.

Finally, once these post-trained models are ready, we need to assess their actual capabilities. At this stage, the model team submits the trained models to the benchmark team for testing.

When the Benchmark team conducts tests, they prioritize based on submission order. The benchmark team takes your model, locates the relevant personnel, positions it in front of a real robot for testing, and then submits the results to the platform. After the platform is updated, the algorithm team can view your current results to evaluate which of the recently trained models performs better or worse, analyze potential reasons for these outcomes, iterate internally, and even provide feedback to the data team.

If the evaluation process is slow, this is essentially an issue that all foundation model companies face—it is indeed quite slow. Especially when you aim to make the evaluation robust, the model team will wait continuously for results from the benchmark team. If they cannot deliver results, the model team won't know whether the current version is good or bad, forcing them to train blindly. This leads to a backlog of benchmark tasks, creating a situation like this.

Q: Based on your assessment, how long do you think it will take for this new workflow to be widely adopted across the industry and achieve a substitution effect close to that of real robots (e.g., 90%+)?

A: From what I understand, entities like Physical Intelligence and NVIDIA's Cosmos abroad have already started collaborating on such initiatives, though I am unsure if they are driving this on a large scale internally.

In China, as far as I know, no other competitors are currently doing this. Or perhaps only a few individuals are conducting research to explore such possibilities. Our work is a serious, systematic attempt to analyze the feasibility of doing this and has been extensively piloted within our organization.

I believe that within two to three months, we could receive very strong market feedback. Information circulates quickly within the circle; people likely want to start using this tool since we have already begun implementing it.

What Kind of World Model Can Serve as a Referee

Q: In the WMBench you proposed, what is the core evaluation dimension? How does it differ from traditional World Model evaluation metrics?

A: Let me first explain the motivation behind our work on World Model Bench. Currently, the industry already has many metrics and benchmarks to evaluate the quality of World Models. However, this paper focuses more on whether a World Model can truly serve as an evaluator. To address this, I needed a "golden metric"—a metric directly relevant to the concept of "World Model as Evaluator"—to serve as my guidance. Other metrics could only be used as auxiliary indicators.

Our golden metric is also the one used in our CVPR competition. Simply put, the success or failure of rollouts within the World Model must perfectly match the success or failure of real-world robot rollouts. If it succeeds here, it must succeed there; if it fails here, it must fail there. This is the golden metric I require.

But this golden metric, which we currently use and have used at CVPR, is mainly evaluated by humans. Because for other methods where you let the model run, no matter how large the model is, it's actually unreliable. It might only have 80% to 90% accuracy, but I cannot accept 80% to 90%. I want it to get the most standard results, just like what a human eye sees. So we then employed a large number of people to annotate the results submitted by each team, and for one team, three people repeatedly conducted A/B tests and checks, finally obtaining a result.

As seen here, manual annotation played a significant role. However, I also hoped to reduce dependency on human labor. This led back to the question: Given that the industry already has so many metrics, which ones are truly correlated with the golden metric derived from subsequent manual annotations? This was a key issue and a central focus of our paper.

Because if you say there are 50 metrics in the industry, and I selected 8 of them, or say 8 to 10 that are most correlated with that subsequent golden metric. Then, can I reduce my reliance on manual annotation? Can I completely use those initial 8 metrics to judge whether this World Model as Evaluator approach is good? Or partially judge it. So this is a key focus of our article's analysis.

We reached a conclusion, which is detailed in the paper. For well-known benchmarks in the industry, such as Water Arena, which contain over ten or twenty metrics, we categorized these indicators into several types. Some types align very closely with our subsequent golden metric, showing linear correlation; others do not align well, and some may even be negatively correlated. We discuss these findings clearly, delineating the boundaries of various metrics to advise readers on which metrics to use to optimize the "World Model as Evaluator" capability.

Furthermore, since auxiliary metrics remain just that—auxiliary—and cannot replace the golden metric, while the golden metric currently requires manual annotation, a natural question arises: Can we develop something to directly replace the golden metric? In other words, can we obtain the golden metric more directly through automation?

For us, a favorable circumstance was that we hosted a competition where over a hundred teams submitted a vast amount of rollout data, which we had manually scored and evaluated. These scores were ideal for training a Judge model, or a multimodal large language model, to determine how the results of each round should align with the human scores. Consequently, we internally trained such a referee model to mitigate the impact of human annotation on this process.

To put it briefly, we approach this from two dimensions. First, the industry already has many automatic metrics—automatically calculated, without manual intervention. We measure their boundary capabilities to determine which align with golden metrics and which do not. Second, we use a large amount of human-annotated data. Internally, we have trained a VRM (Value Model) to replace the reliance on these golden metrics for human annotation. These are the two angles from which we constructed WMBench. This is the motivation behind creating this benchmark, and also its most important reason.

After building this benchmark, we needed to evaluate it as well. We believe this benchmark is the best in the industry for demonstrating the performance of 'World Model as Evaluator.' It relies on human annotations, and the results are not biased solely by our own efforts; they come from submissions from numerous competition models, allowing us to analyze unbiased outcomes. Based on this benchmark, it can effectively determine which World Model is best suited to serve as an evaluator.

On this foundation, we conducted extensive experiments, including open-source video World Models and our own, performing numerous ablation studies. The rankings on this leaderboard—who comes first, who comes second—better reflect whether the preceding World Models are more suitable for the evaluator role.

Q: In practical applications, have you observed discrepancies between scores provided by VLMs and actual physical results? How is this bias typically generated?

A: Very frequently. Some metrics are derived from VLMs, where the VLM calculates whether your interaction's physical consistency holds true. However, in our findings, there is a negative correlation. The score given by the VLM is negatively correlated with the golden metric for 'World Model as Evaluator.' They are fundamentally different things.

The root cause of this issue is that current VLMs lack robust discriminative abilities regarding common-sense physics in embodied intelligence. This is one of our key conclusions.

So, how did we handle this ourselves? We did not use VLMs to judge whether the physical consistency was correct, because VLMs currently may not be capable of doing this well, and solving this is not our immediate objective. Instead, what we aim to solve is whether a VLM can accurately judge, after a World Model rollout, whether the current result is successful or failed, and identify the reasons for success or failure.

We then take the results from real-machine rollouts and feed them into the VLM, asking the VLM to determine success or failure, along with the reasons. I hope these two elements can accurately reflect that the VLM can predict these outcomes sufficiently well, and that these predictions align between the real machine and my model. This is what we intend to achieve.

Video or 3D: The Route Debate for World Models

Q: Within world models, what are the respective advantages and disadvantages of latent space, 3D, and video-based world models when used for policy evaluation?

A: I believe the latent space route cannot serve as a model for policy evaluation; it can function more like a policy model or prediction model. It cannot be used for policy evaluation because policy evaluation must predict sensor signals to feed into our policy model, which then predicts subsequent actions. Therefore, it must be explicit. For instance, for a VLA (Vision-Language-Action) or a World Action Model, it needs to explicitly predict video signals.

Whether it is a video model or a 3D model, such as three-dimensional Gaussian Splatting (3DGS), it must ultimately be rendered into a video for downstream use. It must eventually converge onto a video sensor to be utilized by subsequent systems.

Q: Thus, the issue narrows down to comparing the two versions of World Models—video-based and 3D-based—and their respective advantages and disadvantages in policy evaluation.

A: Let's start with 3D. The advantage of 3D is that it offers the highest consistency. Since it is already constructed as a 3D asset, from any viewing angle, it represents a deterministic 3D physical entity stored on your hard drive.

However, the challenge lies in constructing a 3D world, which is quite complex. To illustrate, suppose we need to evaluate a scenario where a robot is cooking in a kitchen at home. If we use a 3D-based World Model for this task, we must replicate the home environment using the 3D world model. We need to show the model what each item in the kitchen looks like, including the background and foreground. Each object must be processed individually. For example, for each foreground object, we must perform image-to-3D generation to obtain 3D assets, then place them in appropriate positions and assign suitable physical properties. After adjusting everything, we must conduct tests by placing the robotic arm inside to ensure its physical properties are accurate and that the overall 3D reconstructed mesh is precise.

It is an explicit, module-based, pipeline-based solution. Everything is interpretable, which is its advantage. However, its disadvantage is that it is difficult to replicate the real-world scenario at a 1:1 ratio; it may not even be 1:1, but rather a 1:0.6 or 1:0.7 replication of the real-world scenario. There are numerous areas in this process that require manual intervention for adjustment.

I also had conversations with many people working on 3D and digital twin technologies during CVPR. They all mentioned that truly reconstructing a scene involves an enormous amount of manual labor in the 3D domain. This issue remains unsolved. Although there are so-called agent-based methods to replace human effort, currently, it is impossible to eliminate human involvement in this process.

In short, while a 3D approach is feasible if pursued, it inherently relies heavily on manual labor to construct the scene. If I want to evaluate another scenario, such as my bedroom, the same level of human labor must be applied again to build that specific scene. Therefore, it is a relatively labor-intensive solution.

The advantages and disadvantages of video-based World Models are essentially the opposite of those for 3D. The key advantage is that these scenes do not require extensive manual construction. As long as we have videos of the scene—dozens or hundreds of clips, such as those collected via teleoperation—we can directly use them to train our World Model to reconstruct the scene. There is no need for human intervention to sculpt the foreground into meshes or assign physical properties; it becomes entirely a data-driven and model-driven endeavor.

However, its disadvantage is equally apparent: it lacks interpretability. In contrast, with a 3D-based World Model, we can manually inspect certain physical attributes and 3D meshes to verify their reasonableness. We can adjust them until they are correct. For video-based World Models, however, everything is represented through hidden embeddings without any explicit representation.

Although current solutions allow us to constrain some physical attributes of the robotic arm or its position using pixel-aligned conditioning methods, these remain implicit constraints rather than explicit, interpretable ones.

Q: So, are these two approaches with completely opposing strengths and weaknesses?

A: Personally, I have long been optimistic about the video-based approach. If we compare three evaluation methods—the most traditional being real-robot evaluation, the second being 3D evaluation (or simulation-based evaluation), and the third being video-based evaluation—the proportion of model-based and data-driven content increases progressively, with the video-based method having the highest share. As models become more powerful and data accumulates, the latter approach will naturally outperform all preceding ones. The former methods are more labor- and capital-intensive, whereas the latter are driven by data and models.

Q: Therefore, will World Models diverge into different technical directions for evaluation and pre-training?

A: Yes. Actually, we divide world models into three applications.

The first is a data engine, which generates data. The second is a simulation platform, or an evaluator platform, which is what we are currently building with GigaWorld-1. The third is the World Action Model, where the World Model directly performs actions and generates strategies. For these three, the training paradigms and downstream use cases differ.

For data engines and simulation platforms, they require videos with high physical and geometric consistency that can accurately reconstruct scenes. They need pixel-level fidelity—highly detailed predictions with strong physical and 3D consistency.

In contrast, for policy models and the World Action Model, pixel-level requirements are less stringent. The key requirement is that if the model is pretrained on video, the knowledge learned from video prediction should transfer effectively to action prediction. Their objectives are fundamentally different.

Why Is Long-Term Rollout a Key Challenge for World Model Evaluators?

Q: In GigaWorld-1, you mentioned an important design feature aimed at solving error accumulation during long-term rollouts. Why is long-term temporal consistency considered the biggest challenge for current world models? Is it indeed a significant challenge?

A: It is a significant challenge. Because using a World Model as an Evaluator is not like previous systems such as Seedance or Sora, where generating a video of just over ten seconds is sufficient. For robotic manipulation tasks, completion may take 10 minutes, or even several hours for slower operations.

So for video generation, or World Model as Evaluator, it needs to be able to continuously rollout into the future. It may not require generating 10 minutes all at once, but I allow you to generate 10 seconds at a time, then feed this 10 seconds into the next generation, and then into the subsequent generation. It must support continuous progression like this. Only in this way can it evaluate those particularly long-horizon embodied manipulation tasks.

So this is a rigid demand for it. Behind this technical requirement lies my desire for my World Model to possess strong long-term rollout or generation capabilities.

The next three to five years will be about system capabilities

Q: Based on the GigaWorld team's development roadmap and your personal research achievements, what do you think will be the core focus of competition in World Models over the next three to five years?

A: I believe the core competitiveness behind this must lie in system capabilities.

If we view the World Model merely as a model itself, my goals are actually quite simple: one is to keep my World Model's capabilities in the first tier, and the other is to ensure its profitability. These two dimensions constitute my objectives.

To achieve this goal, optimization must begin at the data stage, extend through model training, model infrastructure, the model training team, and even reach the benchmarking team. All these elements must be integrated and driven by clear commercialization targets to maintain its position in the first tier over the long term.

In short, it comes down to model capabilities and commercialization落地 (implementation/landing) abilities. Underlying both is a comprehensive, systematically optimized technical stack for World Models, rather than isolated point solutions.

Q: So how do you define 'first tier'?

A: I think the first tier should be evaluated from two perspectives.

The first is the model itself. From an academic standpoint, you look at metrics like citation count, GitHub stars, and download numbers. Even less quantifiably, when you release a model, can your competitors or students and professors in other labs immediately recognize the work? Do they know it was produced by your team?

Looking toward the industrial sector, the key question is whether your customers validate the product. For instance, if they have needs related to World Models, do they think of you first? And once you secure the contract, can you meet their requirements effectively? Furthermore, are they willing to pay for it in subsequent phases? It’s similar to our continued willingness to pay for models like Anthropic’s or Codex.

I believe these are effective indicators for quantifying both model capability and commercial viability. It’s not simply about achieving first place on a specific benchmark. This is also why, although we created WMBench for GigaWorld-1, we explicitly stated in our paper that we do not encourage people to game this benchmark. It is certainly not a perfect tool yet; we intend for it to serve as a resource. If users find it useful, they can use it to iterate on their World Models and 'World Model as Evaluator' approaches.

We will also continue to update this benchmark by adding more tasks and robot data, enabling everyone to truly utilize it. That’s roughly the situation.

Q: Have you considered how many companies might successfully emerge in this space?

A: I believe there definitely won’t be just one company in China. However, I won’t name specific competitors. I’m confident we are among them, but the market certainly won’t be dominated by a single entity. We believe it’s important to maintain a healthy ecosystem, as a monopoly would likely not yield good results overall. That’s roughly how I see it.

Q: Aside from your own research, if you were to recommend a notable paper you’ve seen this year—one with significant impact or inspiration—what would it be?

A: One recent piece of work I found impressive is Qwen RobotManip, which represents Qwen's foray into VLA (Vision-Language-Action) models. The paper is highly detailed and far from a superficial promotional draft; it includes specific implementation details and provides very objective evaluations in the benchmarking section. Previously, the embodied AI community had been heavily focused on leaderboard rankings, which rendered those leaderboards unfair. In their paper, they conducted a similar analysis to assess which benchmarks are valid and which are not, proposing a setting to explore how these leaderboards can better reflect the true capabilities of algorithms. This article reads more like an industry perspective on "how base models should be optimized," rather than simply boasting about ranking first on a particular benchmark. It is one of the most impressive articles I have read recently.

On the other hand, there is a more theoretical piece. Around late 2025 or early 2026, I read an article titled 'Much Ado About Noising: Dispelling the Myths of Generative Robotic Control.' It offers a profound theoretical analysis of the current mainstream paradigm in VLA—specifically, the architecture combining VLMs with Action Expert Flow Matching. It is worth noting that the traditional route inherited from OpenVLA was not structured this way. Why did the industry shift toward this new strategy? What exactly makes this strategy theoretically superior? This article thoroughly combines theory and experimentation to analyze the underlying logic of why this approach works. Such research truly holds lasting value; it will be remembered by history and provide critical theoretical guidance for the progress of the entire embodied AI industry.

In summary, one category of work thoroughly explains principles through theoretical contributions, while the other accumulates extremely detailed know-how and experience through enterprise-level practice. I personally highly admire and appreciate both types of articles.