Eastbank:
First, there is a nice forum with a lot of FAQ on the Handi, including making it more accurate.
Graybeard FAQ
Second, if you are a member of ASSRA, which you should consider if you own a Handi or any other single-shot rifle, there's a huge archive that you can order from. There's an article (okay, a series of 4 articles) that Rudy can send you for $10 that details making a Handi into a 32-40 Schutzen rifle that was like 1/4 MOA when he got done... extensive gunsmithing required, which you'd expect with a barrel blank with no chamber cut. That's not a recommendation, merely to say that they can be taken to incredibly accurate for a lot less money than, say, a Dakota...
Anyway, some Handi rifles are more accurate than others out of the box. With some small amount of work they can be made more accurate. Rather than give up entirely, you might want to look at the forum. I don't know the inherent accuracy of .357 mag at 100 yards, being as how that is a pistol cartridge. Don't jump on me, please, about the one-shot stops, etc. I know that a grizzly has been taken with one shot from a little over 100 yards. I'm saying that I don't personally know, or remember reading, the inherent accuracy. I know it ain't that of a 6 PPC. Maybe it's about like a 30-30? Worse? Is 4" a good group for that cartridge? If it isn't, then you might tweak the rifle. If it is, you can get a different barrel for about 90 bucks and a long (3-12 month) wait under Remington's auxiliary barrel program for the Handi's.
Later,
Keith