The Hugging Face model is public. A token is normally unnecessary. If your environment requires authentication, pass it through the environment rather than putting it in source code:
The Hugging Face model is public. A token is normally unnecessary. If your environment requires authentication, pass it through the environment rather than putting it in source code:
"# Minimal OpenVLA with Qwen2-0.5B — End-to-End Reproduction\n",
"\n",
"This notebook reproduces the complete minimal experiment from environment setup through action generation.\n",
"\n",
"> **Scope:** this is an OpenVLA-style minimal reproduction. It preserves the central `image + instruction → autoregressive action tokens → continuous action` loop, but replaces Stanford OpenVLA's fused DINOv2+SigLIP and Llama-2 7B stack with LLaVA-OneVision SigLIP + Qwen2-0.5B.\n",
"\n",
"## Table of contents\n",
"\n",
"1. [Paper and references](#1-paper-and-references)\n",
"- **LLaVA/OpenVLA adaptation used here:** [IP127000/openVLA-Qwen2-0.5B](https://github.com/IP127000/openVLA-Qwen2-0.5B)\n",
"\n",
"The original OpenVLA predicts discretized robot actions with a pretrained VLM. Its characteristic architecture uses fused DINOv2 and SigLIP image features, a two-layer projector, and Llama-2 7B. Actions are normalized, discretized into language-model tokens, trained with action-only cross entropy, and decoded back into continuous robot commands.\n"
"+Only the multimodal projector, token embeddings, LM head, and final Qwen block are trained. The experiment is designed to verify the learning path on a few samples, not robotics generalization.\n"
]
},
{
"cell_type": "markdown",
"id": "ec3624ea",
"metadata": {},
"source": [
"## 3. Runtime configuration\n",
"\n",
"The defaults reproduce the reported 100-step experiment. Environment variables allow a fast smoke test or local assets:\n",
"\n",
"- `OPENVLA_STEPS=3` — shorten training for validation.\n",
"- `OPENVLA_INSTALL_DEPS=0` — skip package installation in a prepared environment.\n",
"- `OPENVLA_SKIP_DOWNLOAD=1` — use the Hugging Face cache/offline mode.\n",
"- `OPENVLA_MODEL_PATH=/path/to/model` — load a local model directory.\n",
"- `OPENVLA_LLAVA_REPO=/path/to/qwen2-openvla` — use a local source checkout.\n",
"- `OPENVLA_DATA_JSON=/path/to/data.json` — use local examples instead of the synthetic dataset.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "768bd9ec",
"metadata": {},
"outputs": [],
"source": [
"import os, sys, json, subprocess\n",
"from pathlib import Path\n",
"\n",
"REPO_ROOT = Path.cwd().resolve()\n",
"if not (REPO_ROOT / \"minimal_openvla_qwen05b.py\").exists() and (REPO_ROOT.parent / \"minimal_openvla_qwen05b.py\").exists():\n",
"This cell installs the tested runtime into the current Jupyter kernel. On a clean machine, create and select a Python 3.10–3.12 virtual environment before opening the notebook. Re-running the cell is safe because pip skips satisfied packages.\n"
" print(\"Dependency installation skipped by OPENVLA_INSTALL_DEPS=0\")"
]
},
{
"cell_type": "markdown",
"id": "f67285d5",
"metadata": {},
"source": [
"## 5. Source and weight download\n",
"\n",
"The model depends on the LLaVA adaptation source and a separate SigLIP visual tower. By default this cell clones/downloads them automatically. To use local assets, set the path environment variables in the configuration section.\n",
"\n",
"No access token is stored in this notebook. Public downloads normally work without one; if needed, set `HF_TOKEN` in the shell before launching Jupyter.\n"
"print(\"Weights are downloaded into the Hugging Face cache unless a local path is supplied.\")"
]
},
{
"cell_type": "markdown",
"id": "25639a5d",
"metadata": {},
"source": [
"## 6. Dataset preparation\n",
"\n",
"The default dataset contains three generated 224×224 images with colored squares and seven-dimensional normalized actions. To use local data, provide a JSON file with this schema:\n",
"CUDA uses BF16 when available and otherwise FP16. Apple Silicon uses MPS with FP32 training because real FP16 AdamW testing produced NaNs. Most model parameters remain frozen.\n"
"print(\"trainable parameters:\", f\"{sum(p.numel() for p in trainable):,}\")"
]
},
{
"cell_type": "markdown",
"id": "d4eb7f49",
"metadata": {},
"source": [
"## 8. Minimal overfitting\n",
"\n",
"Only action-token positions contribute to the loss. The default 100-step run takes a few minutes after weights are cached. Set `OPENVLA_STEPS=3` before launching Jupyter for a quick end-to-end smoke test.\n"
"Teacher-forced accuracy measures individual action tokens. The second part performs explicit constrained autoregressive decoding: every generated position is restricted to the 256 action tokens and then converted back to a continuous value.\n"
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
seeker