You essentially have this backward: it is the "hit" that relies more on size and strength, as its success depends entirely on generating enough momentum (mass * velocity) to offset that of the ball-carrier - as you correctly observe, this approach is less geared toward accuracy, so there is less chance that the impulse will occur at an ideal location relative to the runner's center of mass. The wrap-up tackle, on the other hand, is designed to ensure that contact occurs well below the runner's center of mass, leading to substantial torque being applied. (It also prevents the ball-carrier from continuing to generate forward momentum with his legs - as my rugby coach used to put it, it doesn't matter how big a player is, he can't run without his legs.) As such, it is more a matter of technique than of pure physical prowess.
If what you say were correct, tackling in rugby would be impossible without a size/strength advantage, which is clearly not the case.