I've been reloading since 1965, and never had an accident, or a problem. I've never cleaned primer pockets.
I weighed every charge for about a month, and then bought a powder measure. In the old days, I would weight every tenth charge, to verify the measure. For about the last 20 years, I weigh every hundredth charge. (when I refill the primer tube, I weigh a charge)
Most handgun ammo today is reloaded on progressives, so primer pocket cleaning is a moot point for most of us.
Benchrest shooters do not weigh every charge, so I am convinced it's a waste of time. And not by any stretch of the imagination is it a safety issue.
If you follow proper procedures, your powder charges, thrown from a measure will never be off by more than plus od minus .2 grain for pistol powders and ball powder, and never more than .5 for extruded powders.
OTOH, if it feels good, do it!