Regardless of velocity, the factory loads NOT locking the bolt indicates the load (whatever it is) generates LESS gas pressure at that late point when the bullet passes the gas port. So, reasonably, the hand loads keep the pressure level up longer indicating a slower powder.
I know in Garands and M14 rifles (I presume the M1A replica) the condition of a high gas port level can bend operating rods and peening some parts - over time - and just cause wear and tear more quickly. The muzzle velocity - when you have or can use a chronograph - is NOT a reliable predictor of internal pressure.
I do not know what model rifle you are using. That bit of information might or might not provide a clue.
I would suggest you, after consulting reloading manual, switch to a faster powder and therefore use less of a charge. Barring that, have a chat with someone who has the same rife as you and compare notes.