Alaya Lab
Game World Models

Predicting World States, Rendering Geometry,
Painting Appearance

Marionette predicts an explicit, interpretable 3D world state, renders its geometry with a graphics operator that has no learnable parameters, and asks a video-diffusion model for one thing only: appearance.
Zian Meng1,2,3,*, Zhen Li1,*, Chuanhao Li1, Qiang Li3, Kaipeng Zhang1,2
1Alaya Lab, Shanda AI Research Tokyo  ·  2Shanghai Innovation Institute  ·  3Huazhong University of Science and Technology
*Equal contribution
◉ Paper ◉ Code ● Weights (soon) ◉ WildWorld dataset
276dimensional world state
0learned parameters in the bridge
837FVD, against 975 for pixel-AR
+31%pose error from a wrong action token
673.8 hpaired RGB and state footage
System

One world model, assembled from four parts

Each stage does one job and hands the next an explicit object, never a hidden activation. Pick a node to see what it produces and watch it run on real data.

ActionGPT

decision · 2.5M params

picks one discrete action token per entity, per frame

hands over  →  a token

a real action stream, drawn live

The dashed leg marks where the world stops being a state and becomes pixels — not where generation starts. Everything here is generated. The model is handed two things and nothing else: a short seed of pose frames, and one reference image. From there ActionGPT invents the action stream, PoseGPT invents the articulated state, and only the bridge is not generative — it is closed-form geometry with no weights. In the demo film the first three stages are drawn from a ground-truth-aligned state so that one continuous moment can be followed across all four panels; that is a continuity device for the film, not how the system runs.

Insight

How much of a world is computed, and how much is generated

Every way of putting a virtual world on screen sits somewhere on one axis. At the left the world is a program and every pixel is the consequence of code. At the right it is a distribution and every pixel is a sample. The interesting question is not which end wins. It is where you put the cut — which parts of a world are exact by construction, and which are worth generating.

Read left to right and something is given up at every step; read right to left and something is gained. The middle is where a program and a generator each do what they are good at, and it is not obvious that any current system is there. We sit right of it because we generate the world's state and compute only its geometry. That is a claim about where the cut belongs, and the rest of this page is what it buys.

Inference

Drag to see what is computed and what is generated

Each clip is one autoregressive rollout. The right of the divider is the pose-control video the deterministic bridge rasterizes from the predicted state; the left is the RGB the observation model paints from it. The geometry the two sides share is never generated.

↔ drag the divider · clips play when they scroll into view

These are the first nine seconds of seventeen-second rollouts. Later chunks degrade in two ways. The characters sometimes walk past the edge of the scanned terrain, which is a coverage limit of our height field. Appearance also drifts: it is the one quantity carried forward only through the chunk-relay seed frames, so unlike geometry it has no anchor in the state. That is an open problem, and the paper's Limitations section sets out how we intend to attack it.

Comparison

Four systems, let run freely

No control is injected into any of these. Each system is given a first frame and simply allowed to continue, because the only question here is what the picture looks like — not how faithfully any particular moment is reconstructed. Whether a command is actually obeyed is a different question, and it has its own section below.

Ours and the pixel-autoregressive baseline are free rollouts of the same window: the decision model predicts its own action stream and nothing is forced. The commercial systems received a free-combat prompt with no schedule.

Control

Two channels: the action token, and the root

Steering this model is a tensor assignment on its own output stream. No gradient, no auxiliary conditioning network, no retraining. Each clip is cut at its injection points, so a marker switches to that moment instantly instead of seeking.

Forcing the action token

A flat constructed floor and a fixed camera, so nothing but the commanded action can change the picture. A new action id is written into the stream every five seconds and the body executes it within a few frames. Drag the divider and the division of labour is visible: the pose side contains the hunter and nothing else, while the RGB side invents the surrounding creatures and debris. Those inventions are exactly the un-conditioned content our data audit turns up, and they belong to the observation model, not to the state.

Scripting the root directly

The root is a regressed, metric quantity that the animation stage consumes, so it can be overwritten just like a token. The character spins in place, then takes a new heading every two seconds at a commanded 2.5 m/s, each turn exactly 90°. Marker times are measured back off the generated root track, not asserted.
Evaluation

Measured on two layers, which a pixel-only model cannot do

Because the state is a metric articulated skeleton, error can be scored in the world itself, in absolute units, before anything is rendered. The observation layer is the shared ground where any model that emits RGB can be compared.

LayerMetricMarionettePixel-AR
ObservationFVD16 ↓7411131
ObservationFVD, final chunk of a long rollout ↓8901424
StateRA-MPJPE @100f, hunter (m) ↓0.274no state to score
ControlRA-MPJPE, correct action forced (m) ↓0.272no state to score
ControlRA-MPJPE, wrong action forced (m) ↑0.357no state to score

Two findings worth stating plainly. Driving the same observation model with ground-truth pose instead of predicted pose changes nothing measurable, so the rendering stage, and not the dynamics, is what currently limits us. And per-chunk FVD stays flat for both models over a long rollout even while the pixel-autoregressive baseline is quietly swapping the conditioned monster for a different species, which is why we pair distributional scores with state-layer diagnostics and caution against reading a flat FVD curve as evidence of long-horizon consistency.

Data

Two synchronized views of the same footage

Every recording yields both an RGB stream and a per-frame 276-dimensional articulated world state, read out of the game engine, so an action label is the animation the engine actually played and not something a vision model guessed. That correspondence is what makes the dual-layer evaluation possible at all.

4,008RGB clips
673.8 hdecodable source footage
1,395state segments @ 20 fps
16.4 Mannotated state frames
27monster types
5weapon types
6hunter appearances
5stages
54 / 32 / 2joints: monster / hunter / weapon
173 / 689action vocabulary: monster / hunter

The two views, side by side

↔ drag the divider
RGB view
state view, rasterized
Twenty-five recordings at once. Both views come from the identical frames, which is why the skeletons on the right line up with the bodies on the left everywhere in the grid.

The corpus is our previously released WildWorld dataset, and it is richer than the slice we exercise here: 29 monster species with mutually incompatible skeletons, four weapon classes, parties of up to four hunters. We hold most of that fixed and study one-on-one combat so that the state layout, the action vocabulary, and every metric mean the same thing across all of our ablations. The scanned terrain height fields the harness consumes are new to this work.

Citation

BibTeX

@article{meng2026marionette,
  title   = {Marionette: Predicting World States, Rendering Geometry, Painting Appearance},
  author  = {Meng, Zian and Li, Zhen and Li, Chuanhao and Li, Qiang and Zhang, Kaipeng},
  journal = {arXiv preprint arXiv:2608.14530},
  year    = {2026}
}