[user104658]

Honest answer is even the ones who might have come from a "tough background" are not going to stay "hardmen" once they've made it big in the industry.

Combination of legal implications, softer lifestyle, keeping different company, etc.

Plus for the ones who look huge it's mostly just 'roid glamour muscle . In all seriousness, a lot of the actors who look absolutely jacked are not actually as physically strong as, say, a bloke in his late 20's whose been working a heavy construction job since he was 18. Actors are "gym tough". Generally.