Semi-Analytic Models (SAMs) are a crucial tool for understanding galaxy evolution and formation. One major advantage is that these models are able to follow the same galaxy population through redshift, so can offer an insight into galaxy formation not possible with observational data alone. In this talk I will present work from the 'nIFTy comparison of galaxy formation models' project, in which I compare 5 SAMs to observational data from near infrared surveys between z=0 and z=3. I will examine the ability of the models to reproduce the observed number density of galaxies over this range and show that whilst the models fit the data fairly well at low redshift, they produce too many low mass galaxies at high redshift. In the models the number density of low mass galaxies changes very little from z=3 to z=0, showing that this population was in place at z=3. This contradicts the downsizing effect seen in observations where low mass galaxies are formed much later with high mass galaxies in place first. I will present data from 5 leading SAMs, discussing the differences between the models and observations, and highlight potential reasons for these discrepancies.