We study the reproducibility of experiments in Deep Reinforcement Learning (RL). Our main contributions are to i) test the standard experimental workflows for Deep RL and assess their reproducibility, ii) give some partial explanations as to why reproducibility is hard to attain in Deep RL, and iii) introduce new tools and guidelines to maximise reproducibility. We show that using Conda or frozen Pip requirement files does not guarantee reproducibility. We illustrate this on standard RL tasks, including training and evaluation of a PPO agent on MuJoCo and Atari environments. We then explain and empirically demonstrate why reproducibility is more challenging in RL: the discrepancy between different machines grows increasingly large as training progresses, due to the feedback loop of RL. These issues motivate the proposal of a new experimental workflow based on Guix, a functional package manager designed with reproducibility as a goal. Using Guix, we can obtain results that are bitwise reproducible across different machines on several Classic Control and MuJoCo environments (but not on Atari).