Reward Model Generalization: A Cross-Domain Analysis

摘要

We present the first systematic study of reward model generalization across domains.

Introduction

Reward models are critical components in Reinforcement Learning from Human Feedback (RLHF) pipelines.

Background

Previous work has focused on reward model training but neglected cross-domain generalization.

Methodology

We train reward models on single-domain datasets and evaluate zero-shot transfer.

Experiments

Our experiments span 4 training domains and 5 evaluation domains.

Results

We observe significant generalization gaps, with performance drops exceeding 30%.

Discussion

These findings suggest the need for more generalizable reward model architectures.

Conclusion

We provide recommendations for building robust reward models.