There are no degrees of death. Shot placement trumps caliber in too many cases.
Some people, because of their career choices, are more likely to face an armed encounter than others. A CCW holder is less likely to have to fight for his/her life than most LEOs, as they have no duty to act. Can you defend yourself from attack with a .380? Yes. Will it be as effective as something heavier against the "300 lb. assailant"? No. Then again, a .454 Casull won't be overly effective against a body-armored 300 pounder, either. Especially if he's high on speed or crack.
All too many people "what if" occurrences. The area that we've settled in has little crime. Armed robberies are rare here, as are assault. That may have something to do with the high rate of CCW. Knowing this, I usually carry a Kel-Tec P32 in a pocket holster. I can shoot it well under stress, and the little weapon is reliable. Should I venture into a more criminally active area, I usually opt for something in a more serious caliber, .40 S&W or .45 ACP. The little .32 is still going to be there, though. I, for one, am going to walk away from trouble. Failing that, I'll run away from trouble. The last thing that I want to do is fight my way clear. I have no desire to absorb even a .22 Short round.
The .380 has long been capable of working in the normal statistical gun-fight for the CCW. Less than ten feet, frontal shot, unarmored opponent. Living in Georgia, the heavy clothing scenario of the North doesn't apply, either.
If you like it, use it. Practice until hitting your target becomes second nature. You may find that the 9mm locked-breech weapons actually recoil less in like-sized weapons, though.