I don't think 4" of barrel will make 200+ fps difference in any rifle. You should see NO MORE than 100 fps if comparing apples to apples.
It is perfectly normal for 2 barrels of the same length to show very different velocities. Berger is probably using a match grade barrel cut to very close tolerances for their data. I doubt if most mass produced 24" barrels will get their advertised speeds.
I have multiple rifles in 308, 30-06, and 6.5 CM. Some with the same barrel length, some different. There is 90 fps difference between the two 30-06 rifles with the same loads from the same 22" barrel length. A friends 30-06 rifle is 130 fps slower than my fastest rifle with ammo from the same box, both with 22" barrels. The 308's with 22" barrels vary between 25-30 fps, and the 18" rifle is only 50-60 fps slower than the 22" guns.
My guess is that you have a barrel that is about 100 fps slower than normal. That combined with the shorter barrel will account for the 200 fps difference. The fact that you're .8 gr over a listed max charge and not getting pressure signs is another clue. The ammo you've loaded may be safe in your rifle, but may well come close 2700 fps and be an overload in another rifle.
FWIW In my admittedly small sample, all of the guns that I've encountered that were significantly slower than expected were Remington's. I've not seen enough to draw any conclusions, just making an observation.
I'd hunt with it. Most bullets still expand well down to at least 2000 fps, some as slow as 1800 fps. Even with a 2575 fps start at the muzzle I'm betting you're over 1800 fps farther down range than you're willing to shoot. At 400-500 yards it might be an inch or 2 lower but that is easy enough to account for.