I went back to pull DYAR and DVOA as well, and they don't change anything, he is on average around 9th in the league for both over his career, and adding them into the annual average still puts him around 10th best QB every year. You can't say that he's incredibly durable and acknowledge that he is on a better than average team, while also saying he isn't a counting stat fluke, since those are literally the two things that help you put up counting stats. Also, if the stat that you are pulling to argue against the stats I used (Int%) puts him in the Romo/Rivers category, then I would say that just proves my point.
Ben is good, and durable, but that's about it, and that doesn't get you anywhere near the GOAT, whether or not you remove Brady from the universe. If you have to argue to make a case that a guy is in the bottom half of the top ten QBs most of his seasons, that would seem to shut the door on the idea. For reference, Ben's BEST season by QBR is ranked 32nd all time. Brady has SEVEN seasons better than Ben's best. Brees has 3, Peyton has 6, Rodgers has 2, Romo has 2, MATT SCHAUB has 2.