As you point out, we ultimately don't know the Red Sox's thinking on this matter and assumptions are often wrong. Still, the not-so-good defense argument/assumption strikes me as the best explanation for what happened.
You really think demoting a top 25 prospect who basically skipped AAA due to injuries on the ML roster is best explained by Swihart having "not-so-good defense"?
He was a full time catcher for the back half of 2015 on a club that actually pulled itself out of the muck for much of that run. No one was losing their shit over his defense then. In fact, most assessed it to be average. Not good but not bad either, middle of the road with potential to improve, just like the slightly above average (for the position) bat.
He wasn't replaced by the not yet beast mode version of Sandy Leon. He was replaced by someone many scouts and stats guys agree looks like a generational talent behind the dish. That was the play here. Getting the supposed next great pitch framer behind the plate to stabilize the pitching while Swihart could resume his development track in AAA.
His moving off LF was exactly what it looked to be - a club understanding that their best hitter in AAA happened to be a catcher with the athletic pedigree to play elsewhere, at a time when LF production looked pretty dubious as Holt went into slump and Young was scuffling mightily out of the gate. Knowing that if they were to keep both Swihart and Vaz one was going to need to find PT elsewhere it only made sense to start using the athletic guy somewhere else.
There is no great flaw in Swihart as a defensive catcher. He just isn't Christian Vazquez. Or the very good Ryan Hanigan. Or hell, he's probably not even Sandy Leon who has always had a reputation as a very good glove but no bat catcher until these past few weeks. Just because the Sox currently have three VERY GOOD defensive catchers doesn't make Swihart BAD by association, only in comparison. He's an acceptable catching option who happens to be in an organization stacked with ML level catching depth.