Plasticity-Based Analysis of Recent CNN Encoders in Pixel-Based RL
Poster E: Wednesday -- 11:00 - 12:30
Thomas Delliaux, Vincent Francois-Lavet, Emmanuel Rachelson
Keywords: deep reinforcement learning, plasticity loss, CNN encoders, Hadamard product, MaxPool, Hessian eigenspectrum, feature rank, Atari
Recent advances in pixel-based Deep Reinforcement Learning have introduced CNN encoder architectures that achieve strong empirical performance, yet the mechanisms behind these gains remain poorly understood. While plasticity-based diagnostic tools have been used to evaluate post-hoc interventions such as network resets or normalization layers, they have rarely been applied to understand why recent architectures succeed. We apply this analytical methodology to the Hadamax encoder on 10 Atari games, using an ablation and plasticity metrics to disentangle the contributions of MaxPool and the Hadamard product. Our analysis reveals that these two components make orthogonal contributions: MaxPool improves generalization independently of plasticity, as evidenced by superior performance on supervised visual tasks, while the Hadamard product mitigates plasticity loss by maintaining lower dormant ratio, higher feature rank, and a flatter loss landscape under TD non-stationarity, without any explicit intervention.