We study the finite-sample convergence of Q-learning algorithms for the average-reward formulation of Reinforcement Learning (RL) with options. While options allow temporarily extended and hierarchical actions, the average-reward formulation incorporates continual tasks beyond discounted and episodic RL. *We establish the first* $\tilde{O}(1/\mathrm{iterations})$ *convergence guarantees, in the mean-square sense, for average-reward option Q-learning algorithms in both the inter and intra-option settings.* First, we prove that both inter- and intra-option Bellman operators satisfy multi-step span contractions under unichain assumptions. Then, we propose synchronous and asynchronous estimators of these multi-step option Bellman operators leading to four variants of option Q-learning. Finally, we extend the stochastic approximation framework to prove convergence rates of these four algorithms under the minimal assumptions used in the literature. The analysis further yields a guidance for tuning the stepsizes of each option Q-learning variant. As a result, we obtain the first non-asymptotic convergence rates for these four Q-learning algorithms in average-reward RL with hierarchies and reward machines. Our experimental results across RL environments corroborate the theoretical findings.