Too fast, in my book, is where a bullet flies apart on the way to the target. Up to that velocity, with a perfectly uniform bullet (the Berger's are among the best in this regard) that is going straight through the barrel, there should not be a problem. But if you tilt even a perfectly symmetrical bullet in the bore, then the lateral drift shows up, and the faster it is spinning, the greater that drift will be.
The late Harold Vaughn demonstrated this with some high-quality bullets using a special gun built into a machine rest. With straight ammunition, it shot one-holers in the 1s or below. So he intentionally tilted the bullet tips on 8 rounds by 0.004", marked the high side on his cases, then proceeded to fire them starting with the tilt at 12:00, then 3:00, then 6:00, then 9:00 in the chamber, and then went around the clock again with the remaining four rounds. His group is idealized in the image on the left, below. The actual group tore the middle of the paper out, so it is a little harder to see, but a scanner image of it is on page 134 in
this reproduction of the first edition of the book (I highly recommend reading the whole book when you have time). You can see from the slightly out-of-round shapes, that each hole is a double. This is because the orientation and degree of tilt matched.
The second group on the right is what would have resulted from firing 30-06 with the much longer M1 Type bullet tilted the same amount and in a 10" twist barrel. The data to support that comes from A. A. Abbatiello's article in the 1981 NRA book, HANDLOADING.
In both instances, the faster the twist, the further from the center of each group the bullets would be. For any given velocity, it is directly proportional and, as Abbatiello demonstrated, you can calculate exactly how much drift away from the group center a tilted perfect bullet will produce if you know the bullet muzzle velocity, the barrel twist rate, the time of flight, how far from the middle of the bearing surface (center of the tilt) the tip of the bullet is, how far the center of gravity of the bullet is from the center of the bearing surface. You can download
my Excel file for making this calculation.
A couple of other things happen as spin increases. One is that the bullet's yaw of repose, which is the combination of pitch and sideslip yaw it settles in on the way to the target increases. This causes a decrease in ballistic coefficient and an increase in the aerodynamic jump when you have to make a windage adjustment. But neither effect is large.
Do you mind if I ask what bullet weight(s) and barrel length are you using it in? Also if chonoed what FPS you got?
