I've done a fair amount of shooting with only a front sight and the rear sight removed. Out to 15 yards its no problem. I would say though that not having a rear sight, at least for me, is no faster than having one. And if the speed is not increased, why not have one just in case you need it.
