Autonomous air combat represents one of the most demanding challenges in artificial intelligence, requiring agents to operate under uncertainty, partial observability, and adversarial dynamics. Reinforcement Learning and Multi-Agent Reinforcement Learning have recently emerged as promising approaches for enabling adaptive decision-making and coordination in this domain. This survey provides a structured overview of Reinforcement Learning-powered autonomous air combat, with emphasis on open-source environments, algorithmic frameworks, and hierarchical control architectures. We systematically compare three aspects: (1) single-agent and multi-agent settings, (2) full-control and hierarchical abstractions, and (3) the treatment of sensors and observability. Furthermore, we analyze the reproducibility of recent contributions, highlighting the tension between fidelity and accessibility across open and closed-source platforms. Beyond a methodological review, we identify persistent challenges related to scalability, transfer to real-world platforms, non symmetrical scenarios robustness, and computational requirements. By consolidating these advances and limitations, this survey aims to clarify the current state of the field, highlight open problems, and outline pathways toward more robust, scalable, and operationally relevant autonomous collaboration in future air combat systems.