In hierarchical reinforcement learning (HRL), a decision process is decomposed into subproblems, while a semi-Markov decision process (SMDP) chooses among subproblems. Researchers have shown empirically that HRL improves aspects such as exploration, sample efficiency, credit assignment, long-term planning and generalization. This talk discusses several design choices related to HRL, and presents two novel results. The first result is to prove theoretically that HRL is more sample efficient than standard reinforcement learning when the subproblem policies and SMDP policy are learned in parallel. The second result is to develop a theory that enables HRL to compute the globally optimal policy in a sample efficient manner.

Reinforcement learning has gone from being a largely academic field to an established engine of the AI boom in a span of 2 years. This talk will describe this “industrialization of reinforcement learning” happening in private AI labs, from technical, organisational, and economic perspectives. What are RL practitioners doing in AI labs, and how different is it from more traditional RL? We’ll cover research questions, scalable systems, and infrastructure hiccups encountered along the way. We’ll also discuss the organisational challenges of modern RL, where thousands of concurrent agents may interact with sandboxes, external services, reward models, and distributed inference systems. A particular focus will be on crafting useful RL datasets and environments. How do we know if an RL sample is really useful? How can we generate such samples synthetically at scale? We’ll also discuss the growing RL ecosystem: new economic actors, data vendors, human annotation, and how they combine to support successful RL runs. Finally, many important questions remain open, some of which are very hard to investigate at scale. We will highlight several of the open problems that still stand in the way of the growing RL industry.