It's not my automated car or other automated cars on the road that would worry me. It's the manual cars with distracted drivers that would worry me. Until such a day that there are more automated cars than manual cars and the risk from riding in an automated car is lower than the risk of controlling the car myself, I will be controlling the car myself.
What if the data showed that automated cars handled "opposing" human drivers better than other humans did, regardless of perception?
The question is if the rate and severity of "automated vs. human" accidents is worse than that of "human vs. human" accidents. It might "feel" safer to be in charge when other humans are also on the road, but that feeling might be an illusion once we see the data.