Any holster material will eventually wear on a gun. Its kind of a toss up in my opinion. I would think leather as it is softer, but at the same time that would also increase the movement of the gun in the holster as you were walking causing it to rub against the gun. Kydex and plastic are stiffer and might cause more wear when holstering and un holstering but cause less wear when walking around due to holding them more stiffly. pick your poison.
Or better yet, this is the perfect excuse to buy a new gun! Seems perfectly justified to me to preserve the bluing.