Mac provides a baseline of performance you can win with even if there aren't tremendous pieces around him, which is incredibly valuable in today's NFL. Of course he isn't a failure and he's a top 15 QB even if he doesn't develop a lick and continues to play at his current level, which is highly unlikely to happen. People are just antsy because they watch Mahomes, Herbert, Allen, Jackson and Burrow and think unless Mac somehow someway makes a jump into that category the Pats won't be contenders, and most don't believe he has the physical attributes to make said jump. It's hard to disagree with that notion, but he can potentially give the Pats similar value through mental acuity and decision making, which is why the coaching decisions are so frustrating: we can't afford to screw with what he does best because the athleticism isn't there to compensate for it.
I think the comparisons to Brady are a bit off, Tom has at least 15lbs and two inches of height on him and a significantly stronger arm at 45 than Mac has at 24. Brady is limited in mobility, but there isn't a window he can't hit and he makes his living on the intermediate part of the middle of the field, where arm strength matters more than anywhere else.