这篇paper算是我学习世界模型的起点了，ok 开干！

(先叠个甲，很多内容是读到了就随性写下，正确性不做担保。本文的目的是尽量不涉及数学公式的前提下，把dino-wm的思路和方法讲清楚)

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning, ICML 2025

世界模型应该做到什么

作者开头就写出了，他们认为一个真正的世界模型应该是什么样子：

1. 能够基于离线预收集的轨迹进行训练(感觉是最基础的,不必多说)
2. 支持测试时的行为优化(在线优化)
3. 促进任务无关的推理(泛化性)

作者为了满足以上要求，提出了一种无需重建视觉世界即可建模视觉动态的新方法(好绕啊)。

把三条要求拆开看

我们来一条条拆开说明：

首先是第一条"1) 能够基于离线预收集的轨迹进行训练",作者文章中说是利用dinov2预训练的空间Patch特征，使其能够通过预测未来Patch特征从离线行为轨迹中学习。(把dinov2看作一个预训练的视觉编码器)

接着是第二条"2) 支持测试时的行为优化"，文章中说测试期间使用模型预测控制与推理时最优化来达成目标，而无需任何额外信息。感觉就是在测试时固定世界模型参数，只优化动作序列。

最后是第三条"3）促进任务无关的推理"作者在6个环境上进行了zero-shot评估，涵盖非常全面，完全可以支撑这个论点。

DINO-WM 的建模流程

作者接着在文章中定义了什么是世界模型，简单来说世界模型就是:

现在看到什么 + 做什么动作 → 接下来世界会变成什么样

文章中dino-wm具体是这样做的：

1. 图片送入dinov2变为潜在特征(Observation model)
2. 将历史特征和动作相结合预测未来的特征(Transition model)
3. 把未来特征变为RGB 图片(Decoder model)，这一步用于可视化，不是必须的。

上面是文章的方法图，左边和右边是dinov2为模型提供任务无关的状态表征。中间是Transition model使用vit架构，采用帧级注意力(当前帧只能读取过去帧，整帧所有 Patch 一起预测。其实llm也是这样，预测token时能看到之前的所有token)

动作经过MLP编码后拼接到每个Patch特征中，使模型能够学习动作如何改变环境。训练是直接最小化预测dinov2特征与真实下一帧dinov2特征的MSE损失。

如何搜索动作序列

在动作方面，作者使用MPC(Model Predictive Control模型预测控制)结合CEM(Cross-Entropy Method交叉熵方法)反复搜索动作序列，使预测最终状态尽可能接近目标状态。坦白来讲，交叉熵我是知道的，mpc让我想起来了去年火过一段时间的大模型mpc接口...

不扯了，这部分我感觉是一种最优搜索方法，有点抽象。在附录中作者说CEM是从分布中采样一组动作序列，初始分布设置为高斯分布，看起来是一种随机采样方法。

然后对每个采样的动作序列预测潜在空间中的结果轨迹，选择成本最低的前几个动作序列，并相应地更新分布的均值和协方差。一直这样更新循环直到满足条件或者达到迭代次数的阈值。

This paper marks the start of my journey into world models. Okay, let's get into it!

A quick disclaimer: much of this was written spontaneously as I read, so I can't guarantee that everything is correct. My aim is to explain the ideas and methods behind DINO-WM with as little math as possible.

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning, ICML 2025

What should a world model be able to do?

The authors begin by laying out what they believe a real world model should be able to do:

1. Train on trajectories collected offline. This feels like the most basic requirement, so there isn't much to add.
2. Support action optimization at test time, i.e., online optimization.
3. Enable task-agnostic reasoning, i.e., generalization.

To meet these requirements, they propose a new way to model visual dynamics without reconstructing the visual world. Quite a mouthful!

Unpacking the three requirements

Let's go through them one by one.

First, training on trajectories collected offline. The paper uses spatial patch features from a pretrained DINOv2 model to learn from offline action trajectories by predicting future patch features. Think of DINOv2 as a pretrained visual encoder.

Second, supporting action optimization at test time. The paper says it uses model predictive control and inference-time optimization to reach a goal without any extra information. My understanding is that the world model's parameters stay fixed during testing, and only the action sequence is optimized.

Finally, enabling task-agnostic reasoning. The authors conduct zero-shot evaluations in six environments. The coverage is very broad, and I think it supports this claim well.

How DINO-WM works

The authors then define what a world model is. Put simply:

What I see now + the action I take → what the world will look like next.

Here is how DINO-WM does this:

1. Feed an image into DINOv2 to obtain latent features — the observation model.
2. Combine past features with actions to predict future features — the transition model.
3. Turn future features into RGB images — the decoder model. This step is for visualization and is not required.

The figure above shows the method. On the left and right, DINOv2 provides task-agnostic state representations. In the middle, the transition model uses a Vision Transformer (ViT) architecture with frame-level attention: the current frame can only read past frames, and all patches in a frame are predicted together. In a sense, large language models do something similar — when predicting a token, they can see all the preceding tokens.

Actions are encoded by a multilayer perceptron (MLP) and concatenated with each patch feature, allowing the model to learn how actions change the environment. Training directly minimizes the mean squared error (MSE) between the predicted DINOv2 features and the actual DINOv2 features of the next frame.

Searching for an action sequence

For actions, the authors combine model predictive control (MPC) with the cross-entropy method (CEM), repeatedly searching for action sequences so that the predicted final state is as close to the goal state as possible. To be honest, I know cross-entropy, but “MPC” reminds me of those “MPC” interfaces for large language models that were popular for a while last year…

Anyway, this part feels like an optimal-search method to me, and it is a little abstract. In the appendix, the authors explain that CEM samples a set of action sequences from a distribution, initially a Gaussian. So it looks like a random-sampling method.

For each sampled action sequence, the model then predicts the resulting trajectory in latent space. The lowest-cost sequences are selected, and the distribution's mean and covariance are updated accordingly. This loop continues until the stopping condition is met or the iteration limit is reached.

下面是世界模型第二篇文章，这里同样用不涉及数学的方式记录下文章的思路和方法。

Learning Massively Multitask World Models for Continuous Control，ICLR 2026

Newt 想解决什么

这篇文章作者同样认为世界模型应该具有泛化性，发布了一个包含200个多样化的任务benchmark。

并提出了自己的模型Newt：先看大量已有的示范轨迹，让模型学会相关任务大概在干什么以及通常应该怎么动。然后让同一个Newt真正在很多任务里与环境交互，根据获得的新经验进一步迭代模型。 （感觉思想其实和dinowm很像，都是要提出一个泛化到多个任务的模型，思路也比较类似 离线预训练 + 在线优化，有意思）

对世界模型的要求

作者认为合理的世界模型方法应该满足：

• (i) 随着模型和数据规模的增加而扩展（scaling law）
• (ii) 对各种观测空间、动作空间、奖励函数和任务时长具有鲁棒性（模型支持state、RGB等多种输入。奖励尺度不同，有离散回归/交叉熵和log变换。不同任务有不同的折扣因子，就是对未来的奖励进行打折，）
• (iii) 能够充分区分不同任务
• (iv) 在合理的时间范围内完成训练

Newt 的模型结构

上面是文章的方法图。左边是输入，Newt同时接收很多不同任务的数据，包括RGB图像、状态信息和任务文字描述。图像先经过dinov2（上一篇文章也是用这个做编码器，看来想法是一脉相承的），文字经过CLIP编码（感觉可以对应(iii) 能够充分区分不同任务 这一点，毕竟不同任务自然语言输入不同）。

然后中间是Newt World Model。 把这些不同任务的信息放进同一个世界模型中，主要是学习[ 当前状态下做什么动作，可能会得到什么奖励 ]。因为不同任务动作维度不一样，所以无效部分会被掩码mask掉。

右边是具体的世界模型结构。基于TD-MPC2，把观测编码成隐藏状态，然后学习隐藏状态如何随动作变化，同时预测奖励和价值，用这些预测来规划动作（其实就是先学习一个能够预测未来的世界模型，再利用这个模型在内部规划动作）。

Newt 学习什么

Newt主要学习以下内容：

• Dynamics：执行动作后，下一状态会是什么；
• Reward：这个动作能获得多少奖励；
• Value/Q：长期来看这个动作好不好；
• Policy Prior：在当前状态下，什么动作比较值得尝试。

看起来Newt是在潜在空间中预测未来，为控制服务。

从专家示范到在线交互

在训练时，Newt 同时学习： 当前状态 + 动作 → 下一状态 + 奖励 + 长期价值（同时看这个动作后的长期奖励，和折扣因子对应）

然后测试时选择预计回报最高的动作执行。值得一提的是，Newt的训练流程是：

专家示范预训练 → 得到初始世界模型和动作先验 → 与环境在线交互 → 收集新经验 → 持续更新世界模型和策略 （dinowm训练开始时是先随机采样，Newt也算是先用专家进行冷启动了？正好对应了(iv) 在合理的时间范围内完成训练）

与 DINO-WM 的比较

整体看下来，Newt和dinowm一样，都选择在潜空间预测未来（看来潜空间目前是对的）。两者区别在于Newt还显式学习Reward和Value，直接套用了强化学习控制的框架，直接服务于控制。

This is the second paper in my world-model reading notes. Once again, I am recording its ideas and methods without getting into the math.

Learning Massively Multitask World Models for Continuous Control, ICLR 2026

What Newt aims to solve

The authors also believe that world models should generalize, and they introduce a benchmark containing 200 diverse tasks.

They propose their own model, Newt: first, it looks at a large collection of existing demonstration trajectories to learn roughly what the tasks involve and how to act. The same Newt model then interacts with environments across many tasks, using the new experience to keep improving.

(To me, the idea feels quite similar to DINO-WM: both aim to build a model that generalizes across tasks, and the recipe looks similar too — offline pretraining followed by online optimization. Interesting.)

Requirements for a world model

The authors argue that a reasonable world-model approach should:

• (i) Scale as the model and dataset grow — scaling laws.
• (ii) Be robust to different observation spaces, action spaces, reward functions, and task horizons. The model supports inputs such as state information and RGB images. Different reward scales are handled with discrete regression/cross-entropy and logarithmic transformations. Different tasks have different discount factors, which discount future rewards.
• (iii) Distinguish sufficiently between different tasks.
• (iv) Finish training within a reasonable amount of time.

Newt's model architecture

The figure above shows the method. On the left are the inputs: Newt receives data from many different tasks, including RGB images, state information, and textual task descriptions. Images first go through DINOv2 — the previous paper used it as an encoder too, so the ideas seem to follow a similar line. Text is encoded with CLIP. I feel this corresponds to requirement (iii), distinguishing between tasks, since different tasks have different natural-language inputs.

In the middle is the Newt World Model. Information from these different tasks goes into one shared world model, which mainly learns what reward might result from taking a particular action in the current state. Because different tasks have different action dimensions, invalid components are masked out.

On the right is the specific world-model architecture. Based on TD-MPC2, it encodes observations into hidden states, learns how those hidden states change with actions, and predicts rewards and values. These predictions are then used to plan actions. In other words, it first learns a world model that can predict the future, then uses that model to plan internally.

What Newt learns

Newt mainly learns the following:

• Dynamics: what the next state will be after an action is taken.
• Reward: how much reward the action will produce.
• Value/Q: whether the action is good in the long run.
• Policy Prior: which actions are worth trying in the current state.

It looks like Newt predicts the future in latent space to support control.

From expert demonstrations to online interaction

During training, Newt learns these together:

Current state + action → next state + reward + long-term value.

It also considers the long-term reward after the action, which ties back to the discount factor.

At test time, it chooses the action with the highest predicted return. One point worth noting is Newt's training pipeline:

Pretrain on expert demonstrations → obtain an initial world model and action prior → interact with environments online → collect new experience → continually update the world model and policy.

(DINO-WM starts with random sampling when training begins; could Newt be viewed as using experts for a warm start? That would fit requirement (iv): training within a reasonable amount of time.)

Comparing Newt with DINO-WM

Overall, Newt, like DINO-WM, chooses to predict the future in latent space. It seems latent space is the right direction for now. The difference is that Newt also explicitly learns rewards and values, directly adopting a reinforcement-learning control framework to serve control tasks.

最近实习比较忙，有些文章看了没及时整理（找补忏悔 ing）。这次来看世界模型的第三篇笔记。

论文是 Time-Aware World Model for Adaptive Prediction and Control，发表于 2025 年国际机器学习大会（International Conference on Machine Learning，ICML）。

实习中的问题

之所以选这篇文章，是因为我最近在实习里用 Seedance 做视频生成，碰到了一些和时间有关的问题。能指定视频总时长，和能让动作严格发生在指定时间，其实是两回事。

比如“2.3 秒内拿起杯子”。实际使用时，这类细粒度的时间要求依然很难被严格遵守。所以我想看看：世界模型会怎么处理“时间”这件事？

论文想解决什么

世界模型（World Model）可以直观理解为：在内部预测“现在做这个动作，接下来会发生什么”。作者指出，传统方法通常在固定时间间隔下训练，因此可能适应了训练时的观测频率，却不适应部署时的另一种频率。

论文引言举了一个例子：训练时每隔 2.5 毫秒观察一次环境，部署时却每隔 20 毫秒才能获得一次观测。虽然都叫“预测下一步”，这两步实际对应的时间跨度并不一样。

某种程度上，可以把它理解为时间间隔上的分布变化。不过要注意：这是用来说明固定步长训练局限的例子，不是说本文模型只在 2.5 毫秒上训练。

文章的问题可以概括为：在不增加训练样本数量和训练步数的情况下，能不能让同一个世界模型学习不同时间尺度上的动态，并支持不同观测频率下的控制？

为此，作者提出时间感知世界模型（Time-Aware World Model，TAWM）。直觉很简单：不要让模型猜“下一步到底有多长”，直接把这段时间告诉它。

方法图解

左边的 oₜ 是当前时刻的观测，比如机械臂看到的画面。经过编码器 h，它被转换成潜在状态 zₜ，也就是模型内部对当前环境的表示。后面的预测都在这个表示空间中进行。

中间的 M 代表时间感知模型，包含动力学、奖励、价值和策略先验等子模型。分别可以理解为：环境怎么变化、当前能得到什么回报、长期来看值不值得，以及哪些动作比较值得尝试。

图中汇入模型的信息有三类：zₜ 是当前状态，aₜ 是当前动作，Δt 是这一步经过的时间。相比只根据状态和动作预测固定的“下一步”，TAWM 把时间间隔也作为明确条件。

例如，同样的状态和动作，当 Δt 分别为 10 毫秒和 50 毫秒时，模型需要预测两个不同时间跨度下的结果。编码器本身不需要时间条件，其余上述子模型则显式接收它。

下方橙色路径对应未来状态预测。模型先学习潜在状态的变化率，再通过数值积分（Numerical Integration，也就是把变化率累积成一段时间内的变化），更新当前状态。图里的两种更新方式是 Euler 和 RK4。

补充一下这两个名字，跳过也不影响继续往下读。

Euler 是欧拉法（Euler method）。直观理解是：下一状态 ≈ 当前状态 ＋ 当前变化率 × 时间间隔。

RK4 是四阶龙格–库塔法（Fourth-order Runge–Kutta，第一次听说，名字还挺酷）。它会估计起点、两个中间试探状态和末端试探状态的变化率，再按规定权重合并。

相当于欧拉法只看开始时的变化趋势，RK4 还会探一下中间和末端的趋势。不过这里算的是潜在状态的变化率，不一定是物理意义上的速度或加速度，也不需要真的观察到那些未来状态。

上面的欧拉公式只是直觉示意。论文实际使用经过对数缩放的时间因子 τ(Δt)，而不是直接乘原始时间间隔，以稳定训练。

更新后得到的 ẑ(t + Δt)，就是模型预测的、经过这段时间后的状态。橙色旁路表示当前状态会直接参与更新：在原来的状态上，加上模型估计的变化。

最右边带帽子的 r、q、a，分别对应预测奖励、长期价值和策略先验给出的候选动作。后续还要通过规划，比较动作带来的预计回报，再决定执行什么。

还有一个很重要的点：不只是给模型多加一个时间输入，训练时也要让它见到不同时间间隔。作者会在每轮环境交互开始时抽取一个间隔，并在这一轮里保持不变。

比如在 Meta-World 这个机械臂任务测试环境中，TAWM 混合使用 1–50 毫秒的间隔训练。这样，时间条件才有对应的训练数据，而不是只在部署时突然改变它。

作者的对比里，Euler 在多数 Meta-World 任务中表现更好；涉及偏微分方程（Partial Differential Equation，PDE，这里用于描述一个量随空间和时间的变化）的控制任务中，RK4 的收益则随任务和时间间隔而变化，并不是始终更好。

这里“相同训练样本数和训练步数”也不等于“相同实际耗时”：RK4 的更新需要更多模型计算。

我的看法

基本这篇文章就到这里了。相对来说，感觉没有前两篇那么“惊艳”，但问题更具体，也更贴近我实际工程中碰到的困惑。

不过，读到后面发现，好像还是没有直接解决我遇到的问题（笑）。TAWM 主要回答的是“过了这么久以后，环境会变成什么样，以及应该怎么控制”；我想问的却是“能不能让生成视频里的动作，按指定时间发生”。

两者都和时间有关，但不是同一个问题。这篇论文没有直接验证视频生成里的动作时间约束。对我来说，它更像一个提醒：如果希望模型认真对待时间，训练数据和模型的条件输入里，也应该认真对待时间。

论文和演示可以在作者项目页查看。

My internship has kept me busy, and some papers I read never made it into written notes. Consider this a little overdue catching up (and confession). Here is my third note on world models.

The paper is Time-Aware World Model for Adaptive Prediction and Control, published at the 2025 International Conference on Machine Learning (ICML).

A problem from my internship

I picked this paper because of a timing problem I encountered while using Seedance for video generation at work. Specifying a video's total duration and making an action happen at a precise time are two different things.

Take “pick up the cup within 2.3 seconds.” In practice, these fine-grained timing instructions have been difficult to enforce. That made me wonder: how do world models deal with time?

The paper's question

A world model predicts, internally, “if I take this action now, what happens next?” The authors point out that conventional methods are often trained with a fixed interval between observations.

A model may therefore adapt to its training observation frequency but struggle with a different one at deployment.

The introduction gives an example: observations arrive every 2.5 milliseconds during training, but only every 20 milliseconds at deployment. Both predictions are called “the next step,” yet they cover different amounts of time.

In a sense, this is a shift in the distribution of time intervals. But this example illustrates the limitation of fixed-step training; it does not mean the proposed model was trained only at 2.5 milliseconds.

The question is: without increasing the number of training samples or training steps, can one world model learn dynamics at different timescales and support control at different observation frequencies?

The authors propose a Time-Aware World Model (TAWM). The intuition is simple: instead of making the model guess how long “the next step” is, tell it explicitly.

Reading the method figure

On the left, oₜ is the current observation, such as the view from a robot arm. Encoder h turns it into a latent state zₜ: the model's internal representation of the current environment. Predictions are made in this representation space.

The central M contains time-aware dynamics, reward, value, and policy-prior models. Roughly: how the environment changes, what immediate reward an action brings, whether it pays off in the long run, and which actions are worth trying.

Three kinds of information enter the model in the diagram: zₜ, the current state; aₜ, the current action; and Δt, the elapsed time for this step. TAWM explicitly conditions its predictions on the time interval.

For the same state and action, Δt values of 10 and 50 milliseconds call for predictions over different durations. The encoder itself does not need this time condition; the other submodels listed above do receive it.

The orange path predicts the future state. The model learns a rate of change in latent space, then uses numerical integration—accumulating that rate into a change over time—to update the state. The figure shows Euler and RK4 updates.

A quick aside on those two names; feel free to skip it and keep reading.

Euler refers to the Euler method. The intuition is: next state ≈ current state + current rate of change × time interval.

RK4 is the fourth-order Runge–Kutta method. This was my first encounter with the name, which sounds rather cool. It estimates rates at the start, two trial midpoint states, and a trial endpoint, then combines them with prescribed weights.

Euler looks at the initial trend; RK4 also probes intermediate and endpoint trends. These are rates of change of a latent state, not necessarily physical velocity or acceleration. The method does not require real future observations.

The Euler expression above is only an intuitive sketch. The paper actually uses a logarithmically scaled time factor, τ(Δt), rather than multiplying by the raw interval, to stabilize training.

The updated ẑ(t + Δt) is the predicted state after that interval. The orange bypass means the current state participates directly in the update: add the estimated change to the starting state.

On the right, the hatted r, q, and a represent predicted reward, long-term value, and candidate actions from the policy prior. Planning then compares the expected returns of actions before choosing what to execute.

One more important point: adding a time input is not enough. Training also exposes the model to different intervals. An interval is sampled at the start of each episode of environment interaction and stays fixed within that episode.

For example, in Meta-World, a robot-arm task environment, TAWM trains with a mixture of intervals from 1 to 50 milliseconds. The time condition therefore has corresponding training data; it is not suddenly changed only at deployment.

In the authors' comparisons, Euler performs better on most Meta-World tasks. For control tasks involving partial differential equations (PDEs), RK4's benefit depends on the task and interval.

Here, these equations describe how a quantity changes across space and time.

Neither method is always better. Also, equal sample counts and training steps do not imply equal wall-clock time: RK4 updates require more model computation.

My take

That is roughly it for this paper. It felt less “wow” than the previous two, but its question is more concrete and closer to a difficulty I have actually encountered in engineering work.

Although, by the end, I realized it still had not directly solved my problem (ha). TAWM mainly asks what the environment will look like after a given interval, and how to control it.

My question is different: can an action in a generated video happen at a specified time? Both involve time, but they are not the same problem. This paper does not directly test action-timing constraints in video generation.

For me, it is more of a reminder: if we want a model to take time seriously, its training data and conditioning inputs should take time seriously too.

The paper and demonstrations are available on the authors' project page.