Although recent work has provided insights into how individual deep RL algorithms respond to different hyperparameter values, statements about the general behavior of an algorithm with respect to its hyperparameters are largely based on intuition rather than structured analysis. As hyperparameter sensitivity is a well-known obstacle in RL application and research, we want to enable systematic cross-algorithm comparisons across metrics that measure the response of an RL algorithm to its hyperparameter settings. We propose a set of such metrics on algorithm performance, hyperparameter configuration transfer and training stability with respect to algorithm hyperparameters in a broad analysis of existing performance data of PPO, DQN and SAC. Our results show clear differences between these algorithms and the conditions under which they are best applied. We also find that some effects we tend to attribute to poor hyperparameter configurations, like learning stability, arise mainly from the algorithm-environment interaction rather than from hyperparameter configurations. Our results provide an extended set of meta-algorithmic metrics for developing more well-behaved RL algorithms and offer important insights for applying hyperparameter tuning methods in RL.