This was the rule in the American Association of Affiliated American Football of America earlier this season. They didn't stick around long enough for there to be very big sample sizes or even for their to be websites that gather stats in an easy way. But it would be interesting to know the conversion rate.
One thing I don't like about the rule are all the ticky tack fouls that can give a team an automatic first down even when they had no hope. You don't want to encourage teams to commit fouls but other than line of scrimmage penalties virtually everything that the defense does is an automatic first down. Hands to the face, illegal contact, holding. I get that in many of these cases you could arguably say that the foul prevented the team from having a potential chance to get its 15 yards. Like holding that keeps a receiver from running way past you. Still, I might like the rule better if they suspended just for this one play the automatic first down on many defensive fouls, other than pass interference and personal fouls.
I think that giving the team a free play, and either 5 or 10 yards if they do not convert on the free play and another chance, is plenty.
I might even suspend it on roughing the QB. In virtually every case roughing the QB will be enough for the first down anyway, but in the situation in which the offense commits a penalty either at the line of scrimmage or on a play where they succesfully got the first down, so that now they are facing a 4th and longer than 15, I would not like the QB protection rule to mandate a first down automatically.