--- license: apache-2.0 library_name: rynnvla pipeline_tag: robotics language: - en tags: - vision-language-action - vla - latent-action - latent-action-model - robot-manipulation - flow-matching - mixture-of-transformers - embodied-ai - pytorch # base_model is omitted on purpose: the hub warns about and drops an id that does not # resolve to a live repo, and the RynnBrain backbones are not published yet. --- # RynnVLA-Latent-2B Stage-1 latent-action pretraining checkpoint of **RynnVLA-Latent**, a Mixture-of-Transformers vision-language-action model. Given a language instruction and up to six camera views, it generates a chunk of **latent actions** by flow matching. - Parameters: **2,510,917,041** (2.511 B), 1,186 tensors, all `bfloat16` - Weights: **EMA** snapshot at the final training step **236,442** - Frozen during Stage 1: the vision tower and the language model (parameter regexes `visual\.`, `language_model\.`) - Text backbone: hidden 2048, 28 layers, 16 attention heads / 8 KV heads, intermediate 6,144 - Exported: 2026-09-28 Code, data format and training scripts: [github.com/alibaba-damo-academy/RynnVLA-Latent](https://github.com/alibaba-damo-academy/RynnVLA-Latent) --- ## Read this first: what this checkpoint is *not* This is a **Stage-1** checkpoint. Its flow-matching target is the latent action, not a robot action, so Stage 1 consumes no action labels at all. As shipped, `processor_config.json` carries an empty action/state schema (`"schema": {"action": {}, "state": {}}`, `"use_state": false`), and the eval adapters in the repository explicitly refuse a Stage-1 checkpoint for closed-loop control — `validate_libero_schema` raises *"Use a Stage2 action checkpoint, not a Stage1 latent-only checkpoint"* whenever `use_latent_actions` is true. So: **do not point LIBERO / RoboTwin / VLABench evaluation at this repo.** Use it as the initialization for Stage 2, which keeps the same flow-matching objective and retargets it from latent actions to a robot-specific action space. `action_dim = 81` and `action_chunk_size = 30` are declared in the config and carried in the architecture, but Stage-1 training does not use them. ## Latent-action protocol (608-dim `ktoken_zcam`) Stage 1 trains against latent actions produced by [RynnLAM](https://modelscope.cn/models/DAMO_Academy/RynnLAM), which labels raw RGB frame pairs and needs no action space — that is what lets real-robot trajectories, simulation rollouts and human egocentric video share one target representation. | Field | Value | | --- | --- | | `latent_action_dim` | 608 = 8×64 k-tokens (512) + 64 `z` + 32 camera-pose | | `latent_action_chunk_size` | 6 | | `latent_action_stride` | 5 | | Labeling cadence | `gap = 4`, `pair_stride = 4` — one latent action per 4 RGB frames | ## Architecture | Config | Value | | --- | --- | | `model_type` | `rynn_brain_vla` | | `architectures` | `Qwen3VLForConditionalGeneration` | | `action_head_type` | `expert` | | `expert_backbone` | `qwen3_vl` | | `expert_hidden_size` | 768 | | `time_conditioning` | `adaln` | | `num_view_slots` | 6 (`head`, `left_wrist`, `right_wrist`, `front_third`, `side_left`, `side_right`) | | `use_view_cond_slots` | true | | Vision encoder | `qwen3_vl`, depth 24, hidden 1024, patch 16, spatial merge 2, deepstack layers [5, 11, 17] | | Tokenizer | Qwen3 BPE, 151,936 embedding rows; vocabulary identical to the public Qwen3-VL tokenizer (151,643 tokens / 151,387 merges) plus one extra special token `<|state_pad|>` | | `transformers_version` | 5.2.0 | Up to 6 camera roles are carried per sample with a `slot_mask` marking the valid ones, so a source with one camera and a source with four train in the same batch. Images are normalized with `image_mean = image_std = [0.5, 0.5, 0.5]`, i.e. rescaled to `[-1, 1]` (`processor_class = RynnBrainVLAProcessor`). The separate `action_norm_type` field is `mean_std`; it governs action/state normalization and is inert at Stage 1, where `schema.action` and `schema.state` are both empty. ## Loading The custom `model_type` is resolved by the `rynnvla` package, so import it first: ```python import torch from rynnvla.models.rynn_brain_vla import RynnBrainVLAModel, RynnBrainVLAProcessor path = "./RynnVLA-Latent-2B" # a local directory, e.g. from snapshot_download below model = RynnBrainVLAModel.from_pretrained( path, dtype=torch.bfloat16, attn_implementation="flash_attention_2", ) processor = RynnBrainVLAProcessor.from_pretrained(path) ``` Verified on `transformers==5.2.0` / `torch==2.7.1` with both `attn_implementation="eager"` and `"flash_attention_2"`: the checkpoint materializes with **zero** missing and zero unexpected keys, the parameter count equals the `safetensors` header count exactly (2,510,917,041), and no tensor contains a non-finite value. The only exactly-zero tensors are the three action-head biases (`state_proj.bias`, `action_in_proj.bias`, `action_out_proj.bias`), which Stage 1 does not train; their weights are non-zero. Download from ModelScope: ```python from modelscope import snapshot_download model_dir = snapshot_download("DAMO_Academy/RynnVLA-Latent-2B") ``` ## Files | File | Bytes | SHA-256 | | --- | --- | --- | | `LICENSE` | 11,358 | `cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30` | | `chat_template.jinja` | 5,292 | `3636d0f0bd6bef02654cdffdc447b79cb2cef8ab02cc75267345946291a489e4` | | `config.json` | 3,208 | `8ff25f9a6a5c12661da19c270f8a6e4ba4a766e83820acde648ea59730496fd3` | | `export_provenance.json` | 594 | `8c314bd7eab7a1bb4f5f2b71a858f684b38478be72c3fa388aefc52f0f54221d` | | `model.safetensors` | 5,021,978,274 | `e3e7155dda69b38aa05fc6a107353d8fe3a7499bd755ea80db16ec3299992784` | | `processor_config.json` | 1,501 | `2d3f06f58b85154a00da0905c853e92fe760e2e3a6b066935812707024a95e58` | | `tokenizer.json` | 11,422,840 | `f9c0d33d111cb70baf6140ce788268a980ad8b6f70981c35b76193bfa5fb45aa` | | `tokenizer_config.json` | 492 | `e6ee285337a50ab8c6572ac8967fd02fc3da292bc50bc2835b0d873c4fbe61b7` | `export_provenance.json` records the source step, the EMA selection, the frozen-parameter regexes, the source weight digest and the tensor count. Internal absolute paths were removed from it before release, which changes only that file's own digest. ## Training Stage-1 latent-action pretraining on the full latent-action corpus, global batch size 512, run to completion at step 236,442. The released weights are the EMA snapshot of that final step. ## License Apache License 2.0. ## Links - Code & training scripts: [github.com/alibaba-damo-academy/RynnVLA-Latent](https://github.com/alibaba-damo-academy/RynnVLA-Latent) - RynnLAM (produces the 608-dim latent actions): [modelscope.cn/models/DAMO_Academy/RynnLAM](https://modelscope.cn/models/DAMO_Academy/RynnLAM) - Sibling checkpoints: [RynnVLA-Latent-2B](https://modelscope.cn/models/DAMO_Academy/RynnVLA-Latent-2B) · [RynnVLA-Latent-4B](https://modelscope.cn/models/DAMO_Academy/RynnVLA-Latent-4B)