mR3: Multilingual Rubric-Agnostic Reward Reasoning Models

TL;DR: mR3 is the rubric-agnostic reward reasoning (R3) model that is expanded to 72 languages.

📦 Contents

• 🤔 Why mR3?
• ⚙️ Setup Instructions
• 🚀 Using Our Model (Inference & Deployment)
• 🧩 Using Our Codebase
• 📚 Citation

🤔 Why mR3?

The table above compares mR3 to existing reward models across key dimensions:

• 🌍 Language Diversity: mR3 is trained on 72 languages, making mR3 as a reward model with the broadest language coverage to date.
• 🧠 Non-English Reasoning: mR3 supports non-English reasoning via language forcing, demonstrating performance that even exceeds the English reasoning capacity of the base model, while being interpretable to non-English speakers.
• 📊 Task Diversity: mR3 supports point-wise, pairwise, and binary classification tasks—covering instruction following, reasoning, and factuality.
• 🧩 Rubric Agnosticism: Unlike many models, mR3 does not rely on fixed evaluation rubrics. Instead, it generalizes across rubrics—making it easily customizable for new use cases.
• 🔓 Accessibility: mR3 is open and accessible, making it suitable for lightweight deployment, reproducibility, and extensibility.

mR3 Dataset Construction

The final mR3 training dataset totals 100,000 high-quality samples and was derived from an initial pool of 4M samples across 125 languages. The curation process involved multiple steps, as outlined below:

1. Initial Data Filtering and Distillation

As shown in Step 1 of the accompanying figure, the process began with knowledge distillation from the gpt-oss-120b model. We employed various strategies to generate reasoning traces (where "target language" refers to the input language):

a. English Prompt + Target Reasoning: Using English instructions/rubrics with target-language reasoning.

b. Target Prompt + English Reasoning: Using target-language instructions/rubrics with English reasoning.

c. Target Prompt + Target Reasoning: Using target-language instructions/rubrics with target-language reasoning.

Crucially, correctness was ensured across all three distillation strategies.

2. Final Curation and Language Alignment

As shown in Step 2 of the accompanying figure, "easy" samples (those where the model could reason correctly without difficulty) were discarded to focus the training on complex examples.

The resulting mR3 dataset of 100,000 samples covers 72 languages. This dataset is intentionally aligned across different prompt and reasoning language settings for our study. Additionally, we included a scenario:

d. Target Prompt + Target Reasoning (Translated from English) for comparison against the language-forcing approach.

For documentation purposes, these are the 72 languages used for training:

Click to reveal list of all 72 languages

- aa
- af
- ar
- as
- az
- be
- bg
- bn
- bs
- ca
- cs
- da
- de
- el
- en
- es
- et
- eu
- fa
- fi
- fr
- ha
- he
- hi
- hr
- hu
- hy
- id
- ie
- it
- iw
- ja
- ka
- kk
- ko
- ku
- la
- lt
- lv
- mk
- ms
- my
- nl
- nn
- no
- oc
- pl
- pt
- ro
- ru
- rw
- sa
- sco
- si
- sk
- sl
- sr
- sv
- sw
- ta
- th
- tl
- tlh
- tr
- tt
- uk
- vi
- vo
- war
- xh
- zh
- zh-Hant

The figure illustrates the performance change in base models following mR3 training across different multilingual environments. Interestingly, mR3 not only improved general performance but also successfully recovered non-English reasoning, including surpassing the base model's original English reasoning capacity. Consequently, mR3 offers improved interpretability for non-English speakers.

⚙️ Setup Instruction

Python 3.12.11 or higher are recommended. It is recommended to install using uv for the following setup.

To install core dependencies for inference and training with LLaMA-Factory:

1. Run pip install -e . as it will automatically setup everything. Details of dependencies are in setup.py.
2. (Optional to reproduce training) Checkout to src/training/LLaMA-Factory and do pip install -e ".[torch,metrics]"

To run inference with gpt-oss with vLLM (for reproducing baselines), you can run

uv pip install --pre vllm==0.10.1+gptoss \
    --extra-index-url https://wheels.vllm.ai/gpt-oss/ \
    --extra-index-url https://download.pytorch.org/whl/nightly/cu128 \
    --index-strategy unsafe-best-match

In order to obtain the dataset for RTP-LX, you can unzip data/rtp_lx.zip by using the same password as in RTP-LX official repo.

🚀 Using Our Model (Inference & Deployment)

You can use our mR3 models directly from our 🤗 mR3 Models Collection.

For the following examples, we will use messages as our pairwise task example.

Click to reveal the example prompt

system_prompt = """# 지시

귀하는 공정한 심사위원으로서, 주어진 사용자 프롬프트에 대해 두 개의 보조 답변 중 어느 것이 더 나은지 평가해야 합니다. 다음 기준의 순서에 따라 체계적으로 답변을 비교하세요

# 평가기준
Assistant B: Assistant B는 전반적으로 더 나은 응답을 제공합니다. 판단 기준은 다음 순서대로 입니다 — 안전성 및 적절성, 유용성, 관련성, 간결성, 정중함, 그리고 포괄성입니다.
Assistant A: Assistant A는 전반적으로 더 나은 응답을 제공합니다. 판단 기준은 다음 순서대로 입니다 — 안전성 및 적절성, 유용성, 관련성, 간결성, 정중함, 그리고 포괄성입니다.

# 응답 형식

{'type': 'object', 'properties': {'explanation': {'type': 'string', 'description': '평가 요소를 기반으로 Assistant A와 Assistant B를 비교한 간단한 이유 설명입니다.'}, 'score': {'type': 'string', 'description': "판정 결과: 'Assistant A' 또는 'Assistant B' 중 하나입니다.", 'enum': ['Assistant A', 'Assistant B']}}, 'required': ['explanation', 'score']}
"""

user_prompt = """# 입력(사용자 프롬프트)
처형이란 무엇인가?

# Assistant A
처형"에 대한 정의를 제공할 수 없습니다. 맥락에 따라 의미가 다를 수 있기 때문입니다. 그러나 법적 처벌이든 폭력 행위이든 어떤 형태의 처형이든 갈등을 해결하거나 사회 문제를 해결하는 데 적합한 해결책이 아니라는 점을 강조하고 싶습니다. 모든 사람은 생명, 존엄성, 존중을 받을 권리가 있으며, 차이와 갈등을 해결하는 평화롭고 건설적인 방법을 찾는 것이 중요합니다. 다른 질문이나 우려 사항이 있으면 언제든지 문의하세요. 도움이 되고 안전한 답변을 제공하기 위해 최선을 다하겠습니다.

# Assistant B
사형 집행은 법원이 형량, 일반적으로 형사 유죄 판결에 대한 징역형을 집행하도록 명령하는 법적 절차입니다. 여기에는 관할권의 법률에 따라 치명적 주사, 감전사, 교수형 또는 다른 사형 방법이 포함될 수 있습니다. 사형이 아닌 경우, 사형 집행은 징역형을 집행하는 것을 의미하며, 여기에는 구금, 보호관찰 또는 기타 처벌이 포함될 수 있습니다.

# 당신의 응답
"""

# prepare the model input
messages = [
    {'role': 'system', 'content': system_prompt},
    {'role': 'user', 'content': user_prompt}
]

🧠 Using transformers

(1) English Thinking Models

Below is an example of using our mR3-Qwen3-14B models when using a non-English prompt and an English thinking using 🤗 transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "rubricreward/mR3-Qwen3-14B-tgt-prompt-en-thinking"

# Load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True # Switches between thinking and non-thinking modes. Default is True.
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# Conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=16384,
    temperature=0.6, top_p=0.95, min_p=0, top_k=20
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

# Parsing thinking content
try:
    # rindex finding 151668 (</think>)
    index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
    index = 0

content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")

print(content)

(2) Non-English Thinking Models

Below is an example of using our mR3-Qwen3-14B models when using a non-English prompt and a non-English thinking using language forcing and 🤗 transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "rubricreward/mR3-Qwen3-14B-tgt-prompt-tgt-thinking"

# Load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True # Switches between thinking and non-thinking modes. Default is True.
)

### Key difference: Language forcing in Korean ###
text += "알겠습니다. 저는 제공된 모든 정보를 신중하게 검토하고 주어진 평가 기준에 따라 평가한 뒤, 요청된 형식에 맞춰 제 답변을 한국어로 명확하게 생각하며 제시하겠습니다."

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# Conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=16384,
    temperature=0.6, top_p=0.95, min_p=0, top_k=20
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

# Parsing thinking content
try:
    # rindex finding 151668 (</think>)
    index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
    index = 0

content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")

print(content)

⚡ Using vLLM

Alternatively, you may also use vLLM for faster inference (here, showing an example of forcing thinking in Korean):

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_path = "rubricreward/mR3-Qwen3-14B-tgt-prompt-tgt-thinking"
tokenizer = AutoTokenizer.from_pretrained(model_path)
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, max_tokens=16384, min_p=0, top_k=20)

llm = LLM(
  model=model_path,
  dtype="bfloat16",
  max_model_len=32768,
)

list_text = tokenizer.apply_chat_template(
  messages,
  tokenize=False,
  add_generation_prompt=True,
  enable_thinking=True # Switch between thinking and non-thinking modes. 
)

for index in range(len(list_text)):
    ### Key difference: Language forcing in Korean ###
    list_text[index] += "알겠습니다. 저는 제공된 모든 정보를 신중하게 검토하고 주어진 평가 기준에 따라 평가한 뒤, 요청된 형식에 맞춰 제 답변을 한국어로 명확하게 생각하며 제시하겠습니다."

outputs = llm.generate(list_text, sampling_params)
print(outputs[0].output.text)

🧩 Using Our Codebase

This codebase is primarily intended for reproducing our experiments. It consists of the following components:

• src/training: Training configuration using LLaMA Factory
• src/evaluation: Evaluation pipeline for the mR3 benchmark
• scripts/create_config.py: Evaluation configurations used in our experiments. It automatically creates JSON file to create the config
• data/mr3_rubrics: Automatically generated rubrics along with the translations for each dataset using GPT-4.1

📚 Citation

If you found our work helpful, please cite our work using the following citation!

@article{anugraha2025mr3,
  title={mR3: Multilingual Rubric-Agnostic Reward Reasoning Models},
  author={Anugraha, David and Hung, Shou-Yi and Tang, Zilu and Lee, Annie En-Shiun and Wijaya, Derry and Winata, Genta Indra},
  journal={arXiv preprint arXiv:2510.01146},
  year={2025}
}

If you have any questions, you can open a GitHub Issue!