We study the minimal oracle complexity of exact \(q^\pi\)-realizable reinforcement learning. Existing upper bounds for online RL with linearly realizable value functions already use contextual policy-optimization primitives, but whether such planning is intrinsically necessary in the generic stochastic regime has remained open. We answer this question affirmatively. For every finite offline contextual policy-planning instance, we construct a stochastic horizon-\(2\) MDP that is exactly \(q^\pi\)-realizable and whose root Bellman backup equals the planning objective. We then prove an exact transcript simulation theorem: the full online interaction and all fixed-policy evaluation queries for the encoded MDP can be reproduced from the planning instance alone. Consequently, any algorithm that solves generic stochastic online \(q^\pi\)-realizable RL using only fixed-policy evaluation yields a solver for offline policy planning. Combined with known planning-based upper bounds, this implies that policy planning is a minimal sufficient standard oracle for the generic stochastic \(q^\pi\)-realizable regime. In finite-action contextual settings, cost-sensitive classification inherits the same interpretation up to polynomial-time oracle equivalence.