0.任务描述

在普通的图片抽屉打开/关闭识别任务中，强如GPT-4o也只有50~60%的准确率，所以为了让我们的VLM模型能够准确的识别图中抽屉是打开还是关闭的，基于Qwen2_VL进行自定义数据集的微调。

原始数据集如下：
抽屉打开状态图片1109条（drawer_open文件夹）

抽屉关闭状态图片995条（drawer_close文件夹）

1.环境搭建

1.1 获取微调框架ms-swift的代码

git clone https://github.com/modelscope/ms-swift.git
cd ms-swift

1.2 构建微调的conda环境

conda create --name swift -y python=3.10
conda activate swift

1.3 安装依赖

pip install -e .[llm]
pip install pyav qwen_vl_utils deepspeed torchvision vllm modelscope 

到此，你就有了训练环境了！

2.模型获取

切到你自己的模型存放路径并获取Qwen2_VL模型

cd /home2/jianxiong/models/vlm

modelscope download --model=qwen/Qwen2-VL-7B-Instruct --local_dir ./Qwen2-VL-7B-Instruct

到此你就能获得本地模型的路径

/home2/jianxiong/models/vlm/Qwen2-VL-7B-Instruct

3.数据处理

3.1 新建一个datas/drawer目录，把上面的原始图片数据raw_data拷过去

cd ms-swift
mkdir -p datas/drawer

3.2 启一个jupyter lab来进行处理原始数据

# 安装jupyter lab
pip install jupyter jupyterlab
# 切到指定目录
cd ms-swift/datas
# 启动jupyter lab
jupyter lab --ip 0.0.0.0 --port 9528

能看到如下页面

3.3 新建一个notebook"构造VLM训练数据.ipynb"

代码内容如下：

import os
import csv
import json
import pandas as pd
from typing import List

def get_image_files(directory):
    """
    从指定目录递归读取所有文件，返回子文件夹的名称和每个子文件夹下所有图片的绝对路径列表。
    
    Args:
    directory (str): 需要递归读取的目录路径。

    Returns:
    dict: 子文件夹名称为键，图片绝对路径列表为值的字典。
    """
    # 定义支持的图片扩展名
    image_extensions = {".jpg", ".jpeg", ".png"}

    # 初始化结果字典
    result = {}

    # 遍历目录及其子目录
    for root, dirs, files in os.walk(directory):
        # 获取当前目录的名称
        current_folder_name = os.path.basename(root)
        print(f"[当前目录]:{current_folder_name}")

        # 初始化当前目录的图片列表
        image_paths = []

        # 遍历当前目录下的所有文件
        for file in files:
            # 获取文件的扩展名
            _, ext = os.path.splitext(file)
            # 如果文件是图片，将其绝对路径添加到image_paths列表
            if ext.lower() in image_extensions:
                image_paths.append(os.path.abspath(os.path.join(root, file)))

        # 如果当前目录里有图片，记录到结果字典中
        if image_paths:
            result[current_folder_name] = image_paths

    return result

def save_list_to_csv(data_list: List[str], csv_file_path: str):
    """
    将包含文件路径和描述的列表保存到CSV文件中。

    :param data_list: 包含文件路径和描述的列表，格式为 '文件路径,描述'
    :param csv_file_path: CSV文件的保存路径
    """
    if not data_list or not csv_file_path:
        raise ValueError("数据列表和CSV文件路径不能为空")

    # 将列表分解为行和列
    data_rows = [line.split(',') for line in data_list]

    # 写入CSV文件
    with open(csv_file_path, mode='w', newline='', encoding='utf-8') as file:
        writer = csv.writer(file)
        # 可选：写入标题行
        writer.writerow(['image_path', 'caption'])
        # 写入数据
        writer.writerows(data_rows)

    print(f'Data successfully written to {csv_file_path}')
	
	
directory_path = "drawer/raw_datas/"
image_files_dict = get_image_files(directory_path)


STATE_DICT = {
    "drawer_open":"抽屉是打开的",
    "drawer_close":"抽屉是关闭的"
}

contents = []
for name,image_paths in image_files_dict.items():
    caption = STATE_DICT.get(name)
    for image_path in image_paths:
        line = f"{image_path},{caption}"
        contents.append(line)
		
		
save_list_to_csv(contents,'dataset.csv')

## csv转json

df = pd.read_csv('dataset.csv')
# 随机打乱数据
df = df.sample(frac=1).reset_index(drop=True)

# 按 10:1 的比例划分数据
train_size = int(len(df) * 0.9)
train_df = df[:train_size]
val_df = df[train_size:]

def create_conversations(df):
    conversations = []
    for i in range(len(df)):
        conversations.append({
            "query": "<image>图中抽屉是打开还是关闭的？",
            "response": df.iloc[i]['caption'],
            "images": [df.iloc[i]['image_path']]
        })
    return conversations
	
# 创建训练集和验证集对话
train_conversations = create_conversations(train_df)
val_conversations = create_conversations(val_df)

# 保存为json文件
with open('data_train.json', 'w', encoding='utf-8') as f:
    json.dump(train_conversations, f, ensure_ascii=False, indent=2)

with open('data_val.json', 'w', encoding='utf-8') as f:
    json.dump(val_conversations, f, ensure_ascii=False, indent=2)

3.4 运行该notebook，即可生成训练数据

到这一步，就能得到训练数据和验证数据的位置

--dataset /home2/jianxiong/codes/ms-swift/datas/data_train.json \
--val_dataset /home2/jianxiong/codes/ms-swift/datas/data_val.json \

4.模型微调

4.1 新建微调脚本finetune.sh

CUDA_VISIBLE_DEVICES=0,1,2,3 NPROC_PER_NODE=4 swift sft \
  --model_type qwen2_vl \
  --model /home2/jianxiong/models/vlm/Qwen2-VL-7B-Instruct \
  --train_type lora \
  --dataset /home2/jianxiong/codes/ms-swift/datas/data_train.json \
  --val_dataset /home2/jianxiong/codes/ms-swift/datas/data_val.json \
  --deepspeed zero2 \
  --torch_dtype bfloat16 \
  --per_device_train_batch_size 4 \
  --per_device_eval_batch_size 4 \
  --learning_rate 1e-4 \
  --lora_rank 8 \
  --lora_alpha 32 \
  --target_modules all-linear \
  --eval_steps 100 \
  --save_steps 100 \
  --save_total_limit 5 \
  --logging_steps 1 \
  --output_dir output \
  --warmup_ratio 0.05 \
  --num_train_epochs 20

4.2 运行微调脚本

bash finetune.sh

GPU占用情况

微调结束后，会在output目录下生成如下文件

其中的images有相关训练过程可视化图片

5.模型推理

5.1 新建命令行推理脚本infer.sh

CUDA_VISIBLE_DEVICES=0 swift infer \
    --model /home2/jianxiong/models/vlm/Qwen2-VL-7B-Instruct \
    --adapters   /home2/jianxiong/codes/ms-swift/output/v0-20250110-121613/checkpoint-2300 \
    --stream true \
    --merge_lora true \
    --infer_backend pt

5.2 运行命令行推理脚本

bash infer.sh

输入如下内容

<img>/home2/jianxiong/codes/ms-swift/test_image.jpg</img>图中抽屉是打开还是关闭的？

推理结果

5.3 新建app推理脚本app_infer.sh

CUDA_VISIBLE_DEVICES=0 swift app \
    --model /home2/jianxiong/models/vlm/Qwen2-VL-7B-Instruct \
    --adapters /home2/jianxiong/codes/ms-swift/output/v0-20250110-121613/checkpoint-2300 \
    --stream true \
    --merge_lora true

5.4 运行命令行推理脚本

bash app_infer.sh

在你的浏览器上输入你的IP和对应的服务端口

进行推理

推理结果

6.模型部署

CUDA_VISIBLE_DEVICES=0 swift deploy \
    --model /home2/jianxiong/models/vlm/Qwen2-VL-7B-Instruct \
    --adapters /home2/jianxiong/codes/ms-swift/output/v0-20250110-121613/checkpoint-2300 \
    --merge_lora true \
    --infer_backend vllm

需要多卡部署用参数–tensor-parallel-size。

7.接口测试

7.1 接口测试方式一：curl

curl http://localhost:8001/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
          "model": "Qwen2-VL-7B-Instruct",
          "messages": [
            {
              "role": "system",
              "content": "你是一个有用的助手。"
            },
            {
              "role": "user",
              "content": [
                {
                  "type": "image",
                  "image": "/home2/jianxiong/codes/ms-swift/test_image2.jpg"
                },
                {
                  "type": "text",
                  "text": "抽屉的状态？"
                }
              ]
            }
          ]
        }'

7.2 接口测试方式二：python

import requests
import json

def test_api(image_path, text_content):
    url = "http://localhost:8001/v1/chat/completions"
    headers = {
        "Content-Type": "application/json"
    }
    
    data = {
        "model": "Qwen2-VL-7B-Instruct",
        "messages": [
            {
                "role": "system",
                "content": "你是一个有用的助手。"
            },
            {
                "role": "user",
                "content": []
            }
        ]
    }
    
    if image_path:
        data["messages"][1]["content"].append({
            "type": "image",
            "image": image_path
        })
        
    if text_content:
        data["messages"][1]["content"].append({
            "type": "text",
            "text": text_content
        })

    response = requests.post(url, headers=headers, data=json.dumps(data))
    
    print("Status Code:", response.status_code)
    print("Response Body:", response.json())

if __name__ == "__main__":
    # Example usage
    image_path = "/home2/jianxiong/codes/ms-swift/test_image2.jpg"
    text_content = "抽屉的状态？"
    test_api(image_path, text_content)

8.结果统计

8.1 统计脚本

import os
import json
import requests
from tqdm import tqdm

# 配置
STATE_DICT = {
    "drawer_open": "抽屉是打开的",
    "drawer_close": "抽屉是关闭的"
}

# API 端点
API_URL = "http://localhost:8001/v1/chat/completions"
HEADERS = {
    "Content-Type": "application/json"
}

def send_request(image_path, text_content):
    data = {
        "model": "Qwen2-VL-7B-Instruct",
        "messages": [
            {
                "role": "system",
                "content": "你是一个有用的助手。"
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "image": image_path
                    },
                    {
                        "type": "text",
                        "text": text_content
                    }
                ]
            }
        ]
    }
    
    response = requests.post(API_URL, headers=HEADERS, data=json.dumps(data))
    if response.status_code == 200:
        return response.json()["choices"][0]["message"]["content"]
    else:
        return None

def batch_test(state_dict, base_dir, error_log_file):
    total = 0
    correct = 0
    errors = []

    text_content = "抽屉的状态？"
    
    for state, expected_answer in state_dict.items():
        dir_path = os.path.join(base_dir, state)
        file_list = [f for f in os.listdir(dir_path) if f.endswith(('.jpg', '.jpeg', '.png'))]
        
        for image_filename in tqdm(file_list, desc=f"Processing {state}", unit="images"):
            image_path = os.path.join(dir_path, image_filename)
            
            # 调用接口
            answer = send_request(image_path, text_content)
            
            total += 1
            if answer and expected_answer in answer:
                correct += 1
            else:
                errors.append(f"Image: {image_filename}, Expected: {expected_answer}, Got: {answer}")

    accuracy = correct / total if total > 0 else 0

    with open(error_log_file, "w", encoding="utf-8") as f:
        f.write("\n".join(errors))
    
    print(f"Accuracy: {accuracy:.2%}")

if __name__ == "__main__":
    # 基目录，包含 "drawer_open" 和 "drawer_close" 文件夹
    base_dir = "/home2/jianxiong/codes/ms-swift/datas/drawer/raw_datas"
    # 错误日志文件路径
    error_log_file = "/home2/jianxiong/codes/ms-swift/datas/drawer/error_log.txt"
    
    batch_test(STATE_DICT, base_dir, error_log_file)

8.2 统计结果