I have no qualms with a manual safety whatsoever. I've never found myself forgetting to thumb down a safety lever on a 1911, my USP or M&P45 just to name a few. It's all about muscle memory and training. If you train like you should, it's second nature. If you don't, yes, you are going to have problems. Then again, if you don't train like you should when it comes to a Glock, M&P or XD to name a few, what makes you think that you are going to leave your finger off the trigger if that hasn't been built into muscle memory either?
As far as the "this is my safety" finger wavers go, alright, why don't they advocate a 2 pound match trigger for carry? If they're waving their booger hook in the air making that old claim, it shouldn't matter if it's a 20 pound trigger or a 2 pound trigger because they will never have their finger on the trigger until they are ready to shoot....or will they???