My stepdaughter has a masters in equine science so, as you can imagine, we've talked about this quite a lot over the years.
Essentially, it's mostly down to the individual physiology. In human terms, think Michael Owen and Wayne Rooney who both started very young. Owen was essentially knackered by the time he was what, 27, but Rooney continued well into his thirties with only occasional fitness issues.
For NH thoroughbreds, while they generally stop growing at around 4, they are not considered physically mature until 5 or 6. By and large, softer ground is potentially liable to lead to soft tissue injuries but fast ground has a cumulative percussive effect on the joints which, as they get older can tend towards arthritis. Just like humans, the individual physiology will emphasise or lessen these tendencies but it's a reasonable hypothesis that every horse only has a certain amount of miles in its legs even though that number will vary substantially from horse to horse. Following on from that, it seems natural that potentially, the earlier a horse starts jumping the more pressure is being put on immature joints which in some cases will shorten the racing career but, in all fairness, that isn't always the case.
As a breeder, you're naturally keen to see your horses on the track as early as possible but, as an owner, I specifically try to avoid having my horses run when the word 'firm' appears in the going. They can't tell you if they have a small niggle which might be made worse by running.
I don't have a problem with loose schooling 2yos and, while some flat-breds might be suitable for breaking early in their 3yo year, I'd be much more inclined to look at 3yo stores in the spring/summer who, whenever they were broken, wouldn't see any serious training until they were 4.