Kickers are a finicky breed. Perhaps Ghost felt most confident with Brady holding for him rather than Mallett, Edelman, etc. If the kicker isn't confident the ball is gonna be on the spot, laces out, when he gets there he won't be in the right frame of mind.
EDIT: For example (SSS, my experiences aren't universal, etc) I held for PATs/FGs all four years in college. We always had a backup kicker and holder, and those two guys worked together.
When our starting kicker missed a potential game winner at the gun my sophomore year, we went with the backup the next game. He felt more comfortable with the other holder than me, and the coaches let that guy hold the next game.
And I've heard all the "laces out" jokes. So don't.