Understanding galaxy formation and evolution requires studying the interplay between the growth of galaxies and the growth of their black holes (BH) across cosmic time. Here we explore a sample of Halpha-selected star-forming galaxies from the HiZELS survey and use the wealth of multi-wavelength data in COSMOS (X-rays, far-infrared and radio) to study the relative growth between typical galaxies and their black holes, from z=2.23 to z=0.4. We find that the relative black hole to galaxy growth is approximately constant for star-forming galaxies since z=2.23, and that the black hole accretion rates typically fall from high to low redshift following the decline in star formation rate density. Typical star forming galaxies at z~1-2 have BH accretion rates of ~10^(-2) Msolar/yr, and thus grow their stellar mass much quicker than their BH mass (3-3.5 orders of magnitude faster). However, ~2% of the sample (the sources detected directly in the X-ray band) show a significantly quicker growth of the BH mass (up to 1.5 orders of magnitude quicker growth than the non-detected samples). Such relatively short but intense periods of growth likely play an important role in establishing the local M-sigma relation.