I think you have partly answered your own question
and identified the cause. With many rare breeds though, if they were only kept on their original lands they would have died out. I think it's something which we should be very wary of and try to develop a strategy to prevent the general decline into puniness. As with all animals, it needs to be a good sheep before we start to select for breed characteristics (but see below about Soay).
It was I think the Captain who said that surely we should get the numbers up before we start to be too selective, and I agree with that - we just have to decide how many we should have before we start selecting, and what is 'too' selective. So it's not quite straightforward.
The multi-horned Heb is now quite a rare beast, having dropped from about 60% of the national flock when numbers were low, to about 8% now and this may be because selection for 'Jacob-type' even and upright horns began when numbers were low, and many excellent animals were, and still are, culled for less than perfect horns.
It is noticeable with Hebrideans that the big fat ones from the lush grasslands of England really struggle when they head north and the extra weight drops off them. There is now a growing movement to have a Scottish type. One of the difficulties is that English judges tend to judge Scottish shows and continue to judge according to the English norm.
Amongst Hebrideans, there is no actual measuring done, there is just a tendency to disapprove of overly tall individual animals, especially in the show ring. However, there are always people who want the biggest Hebs there are, maybe those breeding for meat. On the other hand there are those who like very small and neat Hebs, and plenty of those who take the middle road. Because as a breed society we have a Breed Description, not a Standard, there is room for this variation and personal preference within the breed, to the extent that one would probably only select for smaller size if all other things were equal. The problem seems to be when small size is the main point selected for.
With Soay, they have selected themselves naturally whilst isolated on their distant island. Their ideal size seems to be governed by patterns of food availability (which in turn depends on fluctuating flock numbers) and on fighting amongst tups for breeding rights. In good years biggest is definitely best, with the most successful tups going on to breed, and strong ewes producing strong lambs. However, once the flock size outgrows the available grazing, or there is a bad winter, it is the bigger individuals which die. Puny males which have small horns (scurs) and so invest fewer resources in maintaining those horns and in fighting for a prime position, are the ones which survive to breed another year. So what I'm saying is that puny doesn't necessarily mean bad - there is a place for both in a normal wild population.