The medical decision of return to play (RTP) after a sport concussion is largely based on symptom status following a graded exercise protocol. However, it is currently unknown how objective markers of brain structure and function relate to clinical recovery. The goal of this study was to determine whether differences in brain structure and function at acute injury remain present at RTP. In this longitudinal study, 54 active varsity athletes were scanned using magnetic resonance imaging (MRI), including 27 with recent concussion, imaged at both acute injury and medical clearance, along with 27 matched controls. Diffusion tensor imaging was used to measure fractional anisotropy (FA) and mean diffusivity (MD) of white matter and resting-state functional MRI was used to measure global functional connectivity (Gconn). At acute injury, concussed athletes had reduced FA and increased MD, along with elevated Gconn; these effects remained present at RTP. Athletes who took longer to reach RTP also showed elevated Gconn in dorsal brain regions, but no significant white matter effects. This study presents the first evidence of altered brain structure and function at the time of medical clearance to RTP, with greater changes in brain function for athletes with a longer recovery time.