There is still much debate surrounding how the most massive (log Mstar>11.5), central galaxies in the local universe have assembled their stellar mass, especially the relative roles of in-situ growth versus later accretion via mergers. We make use of two important observations: 1) that spectral analysis shows that massive galaxies are old (cosmic downsizing). 2) that massive galaxies form in massive dark matter halos. By combining these two pieces of evidence, we find that the local stellar mass is comparable to, if not greater than, the total baryonic mass at the galaxies formation epoch, z_form and it is on average a factor of about ten larger than predictions from abundance matching at z_form. We reason out that it is unlikely that all the galaxy's stellar mass was formed in one burst at high redshift. We test late assembly scenarios, making use of a state of the art, semi-empirical model. This model is constructed on top of dark matter merger trees and utilises empirical relationships to connect these dark matter halos to galaxies. The mass and size are evolved mostly through galaxy mergers therefore, by construction, this model relies on very few input assumptions. We demonstrate that this model can fully explain both the mass and size evolution of massive galaxies with a late assembly driven by later dry-minor mergers.