Otto, I'm very surprised to see someone as mathematically proficient as you are attempting to make that point. I know you are aware that in the long run, optimal lineups will score more runs than less that optimal lineups.
Of course you know in real life that the optimal lineup on paper might score fewer runs than the least optimal.
This is a reason not to optimize your lineup? Thinking outside the box and challenging assumptions is what drives innovation. Last time I checked, what the other 31 managers do with their lineups has no effect on the Red Sox' run production. Taking advantage of market inefficiencies gives you just that, an advantage over your competition, and it could very easily be the difference in making the playoffs or winning the division and not.
The other point is that the truly optimal lineup is one that has never been used by any manager ever.
This is absolutely not true. Can you predict exactly what a player's performance will be on a particular night? No, of course not, but you can clearly come up with expected production for each player and optimize the lineup according to that. Come on, this stuff should be obvious.
Anyway, even putting that aside, no manager will, or can, put together an "optimal" lineup every day. And even if he did, to actually produce the additional wins that you talk about, the players in the optimal lineup must perform optimally every time out.They can't have slumps. They can't get hurt. The have to perform in a consistent, predictable fashion. Which obviously does not happen in real life.
This is completely irrelevant as far as lineup optimization. Let's say that, in order of importance, you have the following things you can do to score more runs:
The point isn't that a manager should not try to optimze his lineup. The point is, of all the tactical decsions in a game, the bating order is probably the least important. Who is in the batting order is much more important than where each player in the order hits in that order. So Jimy Williams getting Darren Lewis 300 ABs at the expense of Troy O'Leary and (yes) Carl Everett was a far worse decision that letting Lewis and his .300 OBA get those ABs from the leadoff spot - bad as that decison was.
1. Acquire good players
2. Actually put these good players into your lineup
3. Optimize the lineup, whoever is in it
Clearly, if you don't acquire good players, you will score less runs. Even if you do acquire good players, but don't put them in your lineup and play lesser players instead, you will score less runs. And regardless of whether you do 1 and/or 2, optimizing the lineup that is actually playing in the games will still score more runs. Yes, it is the least important of the 3, but that doesn't change the fact that it can increase run scoring.