Ensemble methods are a common strategy for improving the stability and performance of value-based reinforcement learning (RL) algorithms. However, conventional ensembles operate in the output space and require maintaining multiple models during inference, resulting in an inference cost that grows linearly with the ensemble size. In this work, we study the geometric structure of the objective landscape in value-based RL through the lens of mode connectivity and analyze how independently trained Q-networks relate in parameter space. Our analysis reveals conditions under which weight-space ensembling becomes effective, namely when Q-networks lie within flat and linearly connected regions of the weight space. Building on these insights, we propose Q-Soups, a weight-averaging approach that merges multiple Q-networks into a single model. Empirical results on Atari benchmarks show that Q-Soups achieves performance comparable to traditional output-space ensembles while maintaining a single model at inference time. To the best of our knowledge, this work presents the first systematic investigation of mode connectivity and weight-space ensembling in value-based RL.