Inverse Reinforcement Learning (IRL) aims to recover expert rewards that transfer across environments with differing dynamics. Prior work has shown that such transfer is possible when gathering expert demonstrations across sufficiently different dynamics. In practice, however, expert demonstrations are costly, making it critical to select the right dynamics to reduce the quantity of expert data required for learning transferable rewards. We propose a two-phase approach for doing so: we cycle through the available dynamics to obtain reliable initial reward estimates, before we actively choose them by minimizing uncertainty about the expert reward. Crucially, this leads to new sample complexity guarantees in terms of transferability. Empirically, our method significantly reduces the amount of expert data needed to learn a transferable reward.