The turbulent fluctuations of the solar wind plasma control the propagation of energetic particles in the heliosphere. Solar Energetic Particles (SEPs), accelerated at the Sun and observed in situ typically at 1 AU from the Sun, propagate through the solar wind, scattering off the magnetic field fluctuations. The latter cause the magnetic field lines to meander across the mean field direction. Recent studies have shown that field line meandering results in non-diffusive, fast spreading of SEPs across the mean magnetic field early in the SEP event history, instead of the often-assumed diffusive SEP propagation. In this work, we use full-orbit particle simulations in prescribed turbulence to study the transition from the early non-diffusive phase to the time-asymptotic diffusive phase of particle propagation. We show that particles are initially strongly tied to the meandering field lines, and the that time scales for the transition from the non-diffusive to the diffusive propagation regime depend strongly on the particle energy and the turbulence amplitude. We discuss the physical phenomena and the relevant length scales that affect the particle decoupling, and the subsequent effects due to field line random walk.