The guy you talked to is right, and very wrong also.
The .25-06 is right on the edge of being "overbore" meaning there is more powder capacity than is easily used for the bore size (.25 cal). In past decades, using primarily IMR type powders, overbore cases had a reputation as barrel burners, specifically the leade in the rifling, the "throat" of the barrel. When this part of the barrel gets worn/burnt out, the accuracy suffers. However, how fast (how many rounds) and how much the accuracy goes downhill are quite variable. More modern powders, with cooler burning temperatures go a long, long way towards taming a "barrel burner".
The .220 Swift had (and still mostly has) the same kind of reputation, and in a varmint rifle, which is shot quite alot, a change from a rifle that groups 3/4" to one that groups 1 1/2" is a disaster, and the barrel ought to be replaced.
In a deer rifle, the same change is not really a serious concern. The .25-06 will wear out its fine accuracy before many less intense rounds, but a single box of ammo simply isn't going to do it. Unless you do the rapid fire thing, it will take hundreds, if not thousands of rounds to "burn out the barrel" to the point it will no longer group well enough for big game hunting.
Everything wears out. High performance machinery wears out faster than lower performance machinery. But a single box of ammo isn't even really breaking the gun in. Under normal conditions, and with proper maintenance and cleaning, I would expect 3-5,000 rounds before the barrel has degraded to the point where it needs to be replaced. Every gun is different, so there can be no hard and fast number, but many guns have gone this many rounds and more without critical failure. It is heat that destroys the steel, and pressure that erodes the bore. The .25-06 has more than many rounds, but many others match it, or come very close. Under normal hunting or sighting in conditions, the rifle should not get hot enough to suffer accelerated wear, unlike rifles used in target matches and rapid fire applications, and since few hunters actually shoot enough rounds in their life time to approach the point of wearing out a rifle barrel, the reputation that guns will last forever has resulted. Use a gun enough and you will wear it out. But "enough" is a huge number of shots, several times the cost of the gun in the cost of the ammo.
Your rifle isn't worn out, and isn't going to be worn out for a very long time (measured by number of shots fired), and even when it is (IF it ever gets that way) the cost of another barrel is much less than the cost of another rifle. Look at it this way, if you drive the same car everyday for 30 years, how many engines will you have to go through before your car isn't worth driving anymore? (note, I am not talking market resale value in dollars, I am talking about the value as working transportation, which is a different thing that what it might sell for)
In short (and I seldom do short) that guy was blowing smoke up your ......He took a single fact and made it sound like waaaay more than it is. And, yes, I do have a .25-06, a custom built 1903 Springfield. Accuracy is stellar, and I have only put a few hundred rounds through it. But the gun was built by someone decades ago, and nobody knows how much it has been fired before I found it. Don't worry about your Ruger, it will group as well as it is capable of for a long time.