Unc, your bullets have to be crimped. Even down at 800 fps, you still have .45 ACP level recoil. Trim and you get a jump from the case to the throat. Some guns will tolerate that with accuracy and some won't, at least when it comes to shooting .38 Special in .357 chambers.
I have not worked at reduced loads in .41 Mag real seriously, but it is something I have piddled with off and on for 25 years. Never really got anywhere. If I give it another go, I'll look for the fluffiest mid speed pistol powder I can find. Should be pretty easy to find with the Lee manual's density chart.
Likely to be someone along who knows more about this than I do. I've only had a couple of .41 Mags and one was a Model 58 for which I was only interested in police level loads that shot to the fixed sights.