The mechanical response of glasses remains challenging to understand. Recent results indicate that the oscillatory rheology of soft glasses is accompanied by a sharp non-equilibrium transition in the microscopic dynamics. Here, we use simultaneous x-ray scattering and rheology to investigate the reversibility and hysteresis of the sharp symmetry change from anisotropic solid to isotropic liquid dynamics observed in the oscillatory shear of colloidal glasses (D. Denisov, M.T. Dang, B. Struth, A. Zaccone, P. Schall, Sci. Rep. 5 14359 (2015)). We use strain sweeps with increasing and decreasing strain amplitude to show that, in analogy with equilibrium transitions, this sharp symmetry change is reversible and exhibits systematic frequency-dependent hysteresis. Using the non-affine response formalism of amorphous solids, we show that these hysteresis effects arise from frequency-dependent non-affine structural cage rearrangements at large strain. These results consolidate the first-order-like nature of the oscillatory shear transition and quantify related hysteresis effects both via measurements and theoretical modelling.