Unless you are shooting the same bullet and powder charge as your factory load, getting a different POI is not surprising. Since most factories do not sell the exact bullets they use for their cartridges, and since most factories have special lots of powders for those cartridges that are probably not available to the public; Again it is not surprising to find different POI.
When you change bullets, there are several factors which will change accuracy and POI. The bullets diameter, bearing surface, alloy, shape, crimp, seating depth/OAL (affects cartridge volume) all affect accuracy.
Unless you are mixing factory and reloads, the goal you should be trying to achieve is accuracy, then adjust your sights accordingly.
