Hierarchical inference improves on-device classification accuracy by offloading low-confidence samples from a local model to a more accurate remote classifier. When offloaded samples are held in a finite-buffer queue, the offloading decision must account not only for the confidence of each sample but also for the state of the queue, since a full buffer forces local classification regardless of confidence. We study this finite-buffer setting under the assumption that the local accuracy is an unknown, monotone-increasing function of the confidence score and that the confidence distribution is unknown. We formulate the problem of maximizing the long-run fraction of correct classifications as an average-reward Markov decision process and show that the optimal admission policy is a sequence of queue-length dependent confidence thresholds. As learning all B+1 thresholds online is costly, we propose Naive, a low-complexity algorithm that instead learns the best single, queue-independent threshold. Through simulation, we compare Naive against the optimal queue-dependent policy, the best static threshold, and the deep reinforcement learning method DeepTOP. For a range of simulation settings, a single learned threshold recovers almost all of the achievable reward, and Naive attains it substantially faster than DeepTOP, which learns a separate threshold for each queue length.