So, we just ignore his performances in the past 5 postseasons. Got it. From where he started, he was on track to be one of the top 2 or 3 QB's all time. He's not there, and his performances in the playoffs are a huge reason why. He couldn't put up more than 3 point-scoring drives in either of the past two Super Bowls. That matters to his legacy.
I'm not sure who you were putting into Top 2 or 3 QB's of all time, but taking a closer look at some of the more recent QB's whom some have put into that category:
Elway: 7-8 in his 1st 15 playoff games, and QB ratings of 83.6, 36.8, and 19.4 in his first 3 Super Bowls. Even in that final 7-0 playoff stretch, he passed for >225 yards once. Of course, his legacy is helped by saving his best for his final game, for which he deserves credit.
Marino: 8-10 career record in the playoffs with a 77.1 QB rating (compared with Brady's 87.6). Yes, his team's weren't always good enough, but that's the point a lot of the Brady defenders (myself included) are trying to make.
Montana: If you're saying that Brady isn't Montana, I won't argue that point. But there's only one Montana, and even he had 3 straight playoff losses (QB ratings of 65.6, 34.2 and 42) with 0 TD's, 4 INT's, and was 3-3 in his final 6 playoff games.
Brett Favre: An example of what good marketing can do for you. Similar playoff stats to Brady; actually slightly worse, and only 1 ring, and that was due to big assists from both Desmond Howard and Bill Parcells.
Peyton Manning: Arguing Brady vs. Manning (Peyton) can quickly turn into an Orr vs. Gretzky type argument. But their playoff records and stats are quite similar l if you care to look that up.
I won't look up Eli, because it will make my head hurt right now. But, to be fair to him, he's still in his prime for a few more years, so it's too early to "close the book" on him. Same with Drew Brees and Aaron Rogers. Also, my apologies to the great ones that came before Montana; too hard to compare against eras.
So, maybe Brady ain't Montana, but he's still pretty damn good. I'm still not sure why we're always so quick to say his legacy is "taking a hit". Certainly shouldn't take any more of a hit than Peyton.