The free energy of hard-sphere systems provides a direct link between the particle-scale structure and macroscopic thermodynamic properties. Here, we employ this framework to investigate the shear-induced structure of a colloidal glass, and link it to its macroscopic mechanical and thermodynamic state. We measure the nonequilibrium free energy under shear from the free volumes of the particles, and monitor its evolution with the applied strain. Unlike crystals, for which the elastic energy increases quadratically with strain due to affine particle displacements, for glasses the free energy decreases due to non-affine displacements and dissipation, reflecting the ability of the glass to reach deeper free-energy minima. We model this decrease using the nonaffine shear modulus and a standard viscous dissipative term. Our model and measurements allow us to disentangle the complex contributions of affine and nonaffine particle displacements in the transient shear deformation of glasses.