The SKS can be pretty accurate. My late model Rusky did 2 MOA and my current Yugo will do 2 MOA as well. With TechSights, the Yugo has a remarkably stout rifle feeling to it that actually does resemble the M1. In fact, I call it "the poor man's M14." They are reliable in typical Russian fashion and durable enough that you'd have better luck trying to break and anvil. Plus you can buy half a dozen SKS for the cost of your basic M1. However, the cartridge just doesn't have the range or the power of the 7.62x51 or the .30-06. The M1s are accurate and powerful enough to reach out and touch something, and their excellent ergonomics, triggers, and sights make it absurdly easy to do it. The SKS is somewhat unrefined, and, well, it just isn't all the rifle the M1s are. As a rifle, I feel my M1A is just about perfect.
If you have the money, then without a doubt you should own an M1, be it an M1A/M14 or the grandfather M1 Garand. They are sweet shooting rifles. If you want to know how good of a rifleman you can be, and how rewarding an afternoon at the range can be, you owe it to yourself to get one.
If you are just looking for a beater rifle--something that can maybe take a little neglect, something you can throw behind the truck seat or under the bed or wherever, and largely ignore until it is needed, the SKS is your baby. However, if you are looking for detachable magazines, it might be a worthwhile investment in your case to get an AK. In my experience, detachable mags for the SKS are hit-or-miss, and mostly miss. The .308 Saiga you mentioned is an great choice.