Do you know which strain they are ? (JA757, JA787, CY etc). I have the parent breeding flock of Master Gris (bred for free ranging) which look nothing like the resulting young that are the end product for slaughter. There are many different genes in the terminal youngsters, segregating them will take many generations. I would be wary using any cocks to breed from as if the dam was an RBM all cocks will be carrying the sex-linked recessive dwarfing gene, the hens will not.
I've never had any problems with growth rate, feeding and exercising are key.