Standard approaches to robust reinforcement learning assume that feedback sources are either globally trustworthy or globally adversarial. In this paper, we challenge this assumption and we identify a more subtle failure mode. We term this mode as Contextual Sycophancy, where evaluators are truthful in benign contexts but biased in critical ones. We prove that standard robust methods fail in this setting, suffering from Contextual Objective Decoupling. To address this, we propose ESA, which learns a high-dimensional trust boundary for each evaluator, and we prove that it achieves sublinear regret $\tilde{O}(\sqrt{T})$ against contextual adversaries. ESA recovers the ground truth even when no evaluator is always (in every context) reliable.