coffee
65 lines
1.8 KiB
Markdown
65 lines
1.8 KiB
Markdown
**Read this in other languages: [English](README.md), [中文](README_zh.md).**
|
|
|
|
# OpenVLA Lightweight Version
|
|
|
|
Lightweight version of OpenVLA, based on the llava-next framework
|
|
|
|
Features:
|
|
|
|
1.The main model is only 0.5B, using qwen2-0.5B.
|
|
|
|
2.Does not use RLDS format datasets, instead fine-tuned using mllm format.
|
|
|
|
3.Does not occupy inherent tokens of the LLM model, instead using an additional 271 tokens to represent actions.
|
|
# How to use
|
|
|
|
## Installation
|
|
|
|
Follow the steps below to install the OpenVLA-Qwen lightweight version.
|
|
|
|
### 1. Clone the repository
|
|
|
|
Clone the repository to your local machine:
|
|
|
|
```bash
|
|
git clone https://github.com/IP127000/openVLA-Qwen.git
|
|
cd openVLA-Qwen
|
|
```
|
|
### 2. Create a new conda environment and activate it:
|
|
```bash
|
|
conda create -n vla-qwen python=3.10 -y
|
|
conda activate vla-qwen
|
|
```
|
|
### 3. Install the required dependencies
|
|
```bash
|
|
pip install --upgrade pip
|
|
pip install -e ".[train]"
|
|
```
|
|
### 4. Download the weights from huggingface
|
|
```bash
|
|
cd openVLA-Qwen/models
|
|
git clone git clone https://huggingface.co/lmms-lab/llava-onevision-qwen2-0.5b-ov
|
|
```
|
|
### 5. Change the file path in scripts/train/finetune_ov_vla.sh
|
|
```bash
|
|
PREV_STAGE_CHECKPOINT="models/llava-onevision-qwen2-0.5b-ov"
|
|
--data_path scripts/train/vla.yaml
|
|
--image_folder /images \
|
|
```
|
|
### 6. Change the data paht in scripts/train/vla.yaml
|
|
```bash
|
|
json_path: /root/data/vla_llava.json
|
|
```
|
|
|
|
### 6. start to train
|
|
```bash
|
|
cd /openVLA-Qwen2-0.5B
|
|
./scripts/train/finetune_ov_vla.sh
|
|
```
|
|
## Acknowledgements
|
|
|
|
This project makes use of the following open-source projects, and I would like to express my gratitude to their creators:
|
|
|
|
- [OpenVLA](https://github.com/openvla/openvla) - For providing the core structure and functionality that inspired OpenVLA-Qwen.
|
|
- [LLaVA_OpenVLA](https://github.com/Darren-greenhand/LLaVA-Next) - For his contribution to data preprocessing techniques.
|