diff --git a/README.md b/README.md index 46a70ec..8775cd1 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,17 @@ **Read this in other languages: [English](README.md), [中文](README_zh.md).** + # OpenVLA Lightweight Version -OpenVLA的轻量化版本,基于llava-next框架 -特点: -1.主体模型仅0.5B,使用qwen2-0.5B -2.不使用RLDS格式数据集,使用mllm格式进行微调 -3.不占用LLM模型固有tokens,使用新增的271个tokens表示动作 +Lightweight version of OpenVLA, based on the llava-next framework + +Features: + +1.The main model is only 0.5B, using qwen2-0.5B. + +2.Does not use RLDS format datasets, instead fine-tuned using mllm format. + +3.Does not occupy inherent tokens of the LLM model, instead using an additional 271 tokens to represent actions. +# How to use ## Installation @@ -29,9 +35,24 @@ conda activate vla-qwen pip install --upgrade pip pip install -e ".[train]" ``` +### 4. Download the weights from huggingface +```bash +cd openVLA-Qwen/models +git clone git clone https://huggingface.co/lmms-lab/llava-onevision-qwen2-0.5b-ov +``` +### 5. Change the file path in scripts/train/finetune_ov_vla.sh +```bash +PREV_STAGE_CHECKPOINT="models/llava-onevision-qwen2-0.5b-ov" +--data_path scripts/train/vla.yaml +--image_folder /images \ +``` +### 6. Change the data paht in scripts/train/vla.yaml +```bash +json_path: /root/data/vla_llava.json +``` ## Acknowledgements -I would like to express my gratitude to the following repositories: +This project makes use of the following open-source projects, and I would like to express my gratitude to their creators: - [OpenVLA](https://github.com/openvla/openvla) - For providing the core structure and functionality that inspired OpenVLA-Qwen. - [LLaVA_OpenVLA](https://github.com/Darren-greenhand/LLaVA-Next) - For his contribution to data preprocessing techniques. diff --git a/README_cn.md b/README_cn.md deleted file mode 100644 index c016a5e..0000000 --- a/README_cn.md +++ /dev/null @@ -1,37 +0,0 @@ -**其他语言版本: [English](README.md), [中文](README_zh.md).** -# OpenVLA Lightweight Version - -OpenVLA的轻量化版本,基于llava-next框架 -特点: -1.主体模型仅0.5B,使用qwen2-0.5B -2.不使用RLDS格式数据集,使用mllm格式进行微调 -3.不占用LLM模型固有tokens,使用新增的271个tokens表示动作 - -## Installation - -Follow the steps below to install the OpenVLA-Qwen lightweight version. - -### 1. Clone the repository - -Clone the repository to your local machine: - -```bash -git clone https://github.com/IP127000/openVLA-Qwen.git -cd openVLA-Qwen -``` -### 2. Create a new conda environment and activate it: -```bash -conda create -n vla-qwen python=3.10 -y -conda activate vla-qwen -``` -### 3. Install the required dependencies -```bash -pip install --upgrade pip -pip install -e ".[train]" -``` -## Acknowledgements - -I would like to express my gratitude to the following repositories: - -- [OpenVLA](https://github.com/openvla/openvla) - For providing the core structure and functionality that inspired OpenVLA-Qwen. -- [LLaVA_OpenVLA](https://github.com/Darren-greenhand/LLaVA-Next) - For his contribution to data preprocessing techniques. diff --git a/README_zh.md b/README_zh.md new file mode 100644 index 0000000..0e631de --- /dev/null +++ b/README_zh.md @@ -0,0 +1,58 @@ +**Read this in other languages: [English](README.md), [中文](README_zh.md).** + +# OpenVLA Lightweight Version + +基于llava-next框架的OpenVLA轻量版。 + +特性: + +主模型只有0.5B,使用qwen2-0.5B。 + +不使用RLDS格式的数据集,而是采用mllm格式进行微调。 + +不占用LLM模型的固有token,而是使用额外的271个token来表示动作。 +# 如何使用 + +## Installation + +按照以下步骤安装OpenVLA-Qwen轻量版。 + +### 1. Clone the repository + +Clone the repository to your local machine: + +```bash +git clone https://github.com/IP127000/openVLA-Qwen.git +cd openVLA-Qwen +``` +### 2. Create a new conda environment and activate it: +```bash +conda create -n vla-qwen python=3.10 -y +conda activate vla-qwen +``` +### 3. Install the required dependencies +```bash +pip install --upgrade pip +pip install -e ".[train]" +``` +### 4. Download the weights from huggingface +```bash +cd openVLA-Qwen/models +git clone git clone https://huggingface.co/lmms-lab/llava-onevision-qwen2-0.5b-ov +``` +### 5. Change the file path in scripts/train/finetune_ov_vla.sh +```bash +PREV_STAGE_CHECKPOINT="models/llava-onevision-qwen2-0.5b-ov" +--data_path scripts/train/vla.yaml +--image_folder /images \ +``` +### 6. Change the data paht in scripts/train/vla.yaml +```bash +json_path: /root/data/vla_llava.json +``` +## 致谢 + +本项目使用了以下开源项目,在此向其开发者表示感谢: + +- [OpenVLA](https://github.com/openvla/openvla) - 感谢提供核心结构和功能,启发了OpenVLA-Qwen。 +- [LLaVA_OpenVLA](https://github.com/Darren-greenhand/LLaVA-Next) - 感谢他对数据处理技术的贡献。 diff --git a/scripts/train/finetune_ov_vla.sh b/scripts/train/finetune_ov_vla.sh index a47fed5..a197bd8 100644 --- a/scripts/train/finetune_ov_vla.sh +++ b/scripts/train/finetune_ov_vla.sh @@ -7,7 +7,7 @@ export NCCL_DEBUG=INFO MIXTURE="qwen" PROMPT_VERSION="qwen_vla" RUN_NAME="openvla-${MIXTURE}" -PREV_STAGE_CHECKPOINT="models" +PREV_STAGE_CHECKPOINT="models/llava-onevision-qwen2-0.5b-ov" echo "PREV_STAGE_CHECKPOINT: ${PREV_STAGE_CHECKPOINT}" echo "MID_RUN_NAME: ${RUN_NAME}" diff --git a/scripts/train/vla.yaml b/scripts/train/vla.yaml index d7bf631..69d076c 100644 --- a/scripts/train/vla.yaml +++ b/scripts/train/vla.yaml @@ -1,3 +1,3 @@ datasets: - - json_path: /root/code/openvla-llava/LLaVA-Next/vla_llava.json + - json_path: /root/data/vla_llava.json sampling_strategy: "all" \ No newline at end of file