Bug: Qwen2-VL weights not initialized with transformers==4.52.1

[hoyoung3724]

While reproducing results from the official Qwen-VL code, loading Qwen/Qwen2-VL-7B-Instruct now produces missing-weight warnings on transformers==4.52.1.
As a result, all reward outputs become 0, suggesting the model is not initialized with pretrained weights.

Environment
GPU: NVIDIA RTX A6000
CUDA: 12.4
PyTorch: 2.6.0 (+cu124)
Transformers: 4.52.1
Model: Qwen/Qwen2-VL-7B-Instruct
Other libs: accelerate==1.6.0, deepspeed==0.16.5
Python: 3.10

Observed Warning
Some weights of RewardModel were not initialized from the model checkpoint at Qwen/Qwen2-VL-7B-Instruct and are newly initialized: ['model.language_model.embed_tokens.weight', 'model.language_model.layers.0.input_layernorm.weight', ...]

[Jia-py]

Thanks for your interest in our work!
It seems that this is due to the backbone model's weights not being loaded correctly, rather than an issue with the provided checkpoint.
You can try the following points:

1. Checking whether the qwen2-vl-7b-instruct weights are correctly downloaded.
2. Testing the inference script to see qwen2-vl-7b-instruct weights can be loaded or not
3. Then test the inference script to see if the lora part works

[hoyoung3724]

Thanks for the quick response!

I’ve verified that the Qwen2-VL-7B-Instruct weights themselves load correctly when initialized alone, so the base checkpoint is fine.
However, the issue appears only when the model is wrapped as a RewardModel — the structure of the reward head seems to prevent proper weight loading.

I also tried several alternative model structures to bypass the warning, but those led to different (and inconsistent) reward values when running quick_start_im2gps3k.py.

import os
import sys
import warnings

warnings.filterwarnings("ignore")
os.environ["TOKENIZERS_PARALLELISM"] = "false"

import argparse
import os
import types
from datetime import datetime
from typing import Optional, Union

import deepspeed
import torch
import torch.distributed as dist
import torch.nn as nn
from datasets import load_dataset
from flash_attn.utils.distributed import all_gather
from peft import LoraConfig, PeftModel, TaskType, get_peft_model
from peft.tuners.lora import LoraLayer
from transformers import (
    AutoConfig,
    AutoModel,
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    Qwen2VLForConditionalGeneration,
    Trainer,
    TrainingArguments,
)
from transformers.integrations.deepspeed import HfDeepSpeedConfig

from utils import *


def reward_model_forward(
    self,
    input_ids,
    attention_mask=None,
    pixel_values=None,
    return_output=False,
    **kwargs
):
    outputs = self.original_forward(
        input_ids=input_ids,
        attention_mask=attention_mask,
        pixel_values=pixel_values,
        output_hidden_states=True,
        **kwargs
    )

    last_hidden_states = outputs.hidden_states[-1]

    values = self.value_head(last_hidden_states).squeeze(-1)
    reward = values[:, -1]

    if not self.training and self.normalize_reward:
        reward = (reward - self.mean) / self.std

    return (reward, outputs) if return_output else reward


def get_vlm_for_sequence_regression(
    base_model_name_or_path: str,
    model_name_or_path: str,
    model_type: str,
    *,
    bf16=True,
    lora_config=None,
    normalize_reward=False,
    use_flash_attention_2=False,
    ds_config: dict = None,
    init_value_head: bool = False,
    value_head_prefix="value_head",
    device_map=None,
    packing_samples=False,
    lora_path=None,
    **kwargs
) -> nn.Module:

    config = AutoConfig.from_pretrained(base_model_name_or_path, trust_remote_code=True)

    config._attn_implementation = (
        "flash_attention_2" if use_flash_attention_2 else "eager"
    )

    model_dtype = "auto"
    if bf16:
        model_dtype = torch.bfloat16
    elif use_flash_attention_2:
        print(
            "Warning: use_flash_attention_2=True requires bf16 or fp16. Forcing torch.bfloat16."
        )
        model_dtype = torch.bfloat16

    if ds_config is not None and ds_config["zero_optimization"]["stage"] == 3:
        dschf = HfDeepSpeedConfig(ds_config)
    else:
        dschf = None

    model = Qwen2VLForConditionalGeneration.from_pretrained(
        model_name_or_path,
        config=config,
        trust_remote_code=True,
        torch_dtype=model_dtype,
        quantization_config=None,
        device_map=device_map,
        **kwargs
    )

    value_head_layer = nn.Linear(model.config.hidden_size, 1, bias=False)
    value_head_layer = value_head_layer.to(device=model.device, dtype=model.dtype)
    setattr(model, value_head_prefix, value_head_layer)

    model.normalize_reward = normalize_reward
    model.register_buffer("mean", torch.zeros(1), persistent=False)
    model.register_buffer("std", torch.ones(1), persistent=False)
    if hasattr(config, "mean"):
        model.mean[0] = config.mean
        model.std[0] = config.std

    model.original_forward = model.forward

    model.forward = types.MethodType(reward_model_forward, model)

    model.supports_gradient_checkpointing = True

    if lora_path is not None:
        model = PeftModel.from_pretrained(model, lora_path, is_trainable=False)
        model.eval()
    elif lora_config is not None:
        model.enable_input_require_grads()
        model = get_peft_model(model, lora_config)

    model_config = model.config.to_dict()
    if "output_router_logits" in model_config:
        print("[MoE] set output_router_logits as True")
        model.config.output_router_logits = True

    model.config.use_cache = False

    if init_value_head:
        value_head = getattr(model, value_head_prefix)
        if dschf is not None:
            with deepspeed.zero.GatheredParameters(
                [value_head.weight], modifier_rank=0
            ):
                if torch.distributed.get_rank() == 0:
                    value_head.weight.data.normal_(
                        mean=0.0, std=1 / (config.hidden_size + 1)
                    )
        else:
            value_head.weight.data.normal_(mean=0.0, std=1 / (config.hidden_size + 1))

    return model

This code had no problem with loading the base model and the checkpoints, but ended up giving the following results

Rewards for all candidates:
Candidate 0: GPS=(46.470601, 11.623621), Reward=-3.6250
Candidate 1: GPS=(46.623701, 12.299613), Reward=-4.3125
Candidate 2: GPS=(47.329750, 11.259956), Reward=-5.2500
Candidate 3: GPS=(46.527441, 12.029183), Reward=-3.6094
Candidate 4: GPS=(44.173955, 6.697883), Reward=-5.6562
Candidate 5: GPS=(46.435313, 11.665763), Reward=-3.5000
Candidate 6: GPS=(46.378019, 11.577111), Reward=-4.2812
Candidate 7: GPS=(44.392579, 7.046012), Reward=-5.1562
Candidate 8: GPS=(44.242555, 6.711788), Reward=-3.9062
Candidate 9: GPS=(46.592254, 12.014923), Reward=-4.0000

Predicted GPS (max reward): (46.435313, 11.665763) with reward -3.5000
Ground Truth GPS: (46.167286, 7.099698)
Geodesic Distance: 352.99 km

[hoyoung3724]

After further testing, I downgraded the environment(transformers==4.51.3) and retried loading the model.
This time, the base model loads correctly, but I get the following message:

Some weights of RewardModel were not initialized from the model checkpoint at Qwen/Qwen2-VL-7B-Instruct and are newly initialized: ['value_head.weight']
You should probably TRAIN this model on a downstream task to be able to use it for predictions and inference.

The resulting rewards are all exactly 0,

Rewards for all candidates:
Candidate 0: GPS=(46.470601, 11.623621), Reward=0.0000
Candidate 1: GPS=(46.623701, 12.299613), Reward=0.0000
Candidate 2: GPS=(47.329750, 11.259956), Reward=0.0000
Candidate 3: GPS=(46.527441, 12.029183), Reward=0.0000
Candidate 4: GPS=(44.173955, 6.697883), Reward=0.0000
Candidate 5: GPS=(46.435313, 11.665763), Reward=0.0000
Candidate 6: GPS=(46.378019, 11.577111), Reward=0.0000
Candidate 7: GPS=(44.392579, 7.046012), Reward=0.0000
Candidate 8: GPS=(44.242555, 6.711788), Reward=0.0000
Candidate 9: GPS=(46.592254, 12.014923), Reward=0.0000

Predicted GPS (max reward): (46.470601, 11.623621) with reward 0.0000
Ground Truth GPS: (46.167286, 7.099698)
Geodesic Distance: 350.00 km

This indicates that the value_head parameters are not being loaded from the checkpoint, even though the rest of the backbone weights are correctly initialized.
It seems the checkpoint may not include the value head weights, or the mapping between the stored checkpoint keys and the RewardModel structure has changed.

[Jia-py]

Hi, thanks for the follow-up.

For value head weights, you may use the following code to see the parameters in the safetensors file:

from safetensors import safe_open
with safe_open("model.safetensors", framework="pt") as f:
    for k in f.keys():
        print(k)

Then you will find there is a parameter named base_model.model.value_head.weight.

[Jia-py]

For the warning

Some weights of RewardModel were not initialized from the model checkpoint at Qwen/Qwen2-VL-7B-Instruct and are newly initialized: ['value_head.weight']
You should probably TRAIN this model on a downstream task to be able to use it for predictions and inference.

it should be fine, please refer to the code script in geo_ranker.py, where we define:

model = cls_class.from_pretrained(model_name_or_path, config=config, trust_remote_code=True, torch_dtype = torch.bfloat16 if bf16 else "auto", quantization_config=None, device_map=device_map, **kwargs)

if lora_path is not None:
    model = PeftModel.from_pretrained(model, lora_path, is_trainable=False)
    model.eval()

[Jia-py]

For the environment issue, I uploaded the conda environment at huggingface. You can try downloading it and following the instructions:

mkdir -p ~/conda_path/envs/your_env_name
tar -xzf georanker.tar.gz -C ~/conda_path/envs/your_env_name

Hope this helps!

[hoyoung3724]

Thank you for your swift response.

I’ve encountered issues when running this code on multiple GPUs (DDP / DeepSpeed).
The model works fine on a single GPU, but multi-GPU setups can sometimes lead to unexpected behavior — possibly related to how the custom value_head and device_map interact under distributed settings.

I haven’t fully confirmed the exact cause yet, but it seems safer to stick to single-GPU or disable device_map when using DDP / ZeRO for now.

Despite this, i'm still having some small discrepancies in the reward values.

Rewards for all candidates:
Candidate 0: GPS=(46.470601, 11.623621), Reward=-6.6875
Candidate 1: GPS=(46.623701, 12.299613), Reward=-6.5312
Candidate 2: GPS=(47.329750, 11.259956), Reward=-7.2500
Candidate 3: GPS=(46.527441, 12.029183), Reward=-7.0312
Candidate 4: GPS=(44.173955, 6.697883), Reward=-4.1875
Candidate 5: GPS=(46.435313, 11.665763), Reward=-6.6250
Candidate 6: GPS=(46.378019, 11.577111), Reward=-6.4375
Candidate 7: GPS=(44.392579, 7.046012), Reward=-5.0625
Candidate 8: GPS=(44.242555, 6.711788), Reward=-4.3750
Candidate 9: GPS=(46.592254, 12.014923), Reward=-6.7500

Predicted GPS (max reward): (44.173955, 6.697883) with reward -4.1875
Ground Truth GPS: (46.167286, 7.099698)
Geodesic Distance: 223.77 km

I'm not sure whether this would harm the model performance or not, as the orders stay the same.

[Jia-py]

1. Yes, I only used a single GPU for inference. And it will work if you use the provided script to fine-tune the model with multiple GPUs.
2. The accuracy of the rewards may be affected by different GPUs, which should be a normal phenomenon. We have also conducted experiments on the A100, and this should not have a significant impact on the results.