You don't have the requisite information or calculations to draw these conclusions.
The pressure inside an ignited cartridge exerts a normal force on all interior surfaces.The pressure at any given time divided by the cross sectional area of the bullet base will give you the force on the bullet, divided by the cross sectional area of the inside base of the case will give you the force acting on the rifle axially through the bolt. Subtracting the friction forces from the bore, you can get the net forces on the bullet, acceleration equals Force divided by Mass. In the case of the rifle, you have a linear acceleration into the shoulder, but also an angular acceleration due to the moment arm between the force in line with the bolt, and the pivot point, your shoulder.
The acceleration of the rifle starts at the same instant as the acceleration of the bullet. On one hand, the acceleration of the rifle is far lower than the bullet due to much greater Mass, on the other hand, the motion of the rifle has a angular component and is much less constrained in direction than the bullet while it's in the barrel. Without actually doing the math on a specific rifle, load, stock geometry, etc, you can't say that the recoil induced movement of the rifle (and irregularities therin) will not have an impact on bullet POI.
Where are all these shooters that hold 0.25" groups with .338-378 in 7 lb rifles? Perhaps you could make a rifle regularly shoot like that from a rest, where forces can be uniformed, but I suspect that the increased rifle acceleration of a rig like that is going to prove my point when applied to squishy, human shoulders applying inconsistent forces to the stock.
Why do benchrest shooters use stocks that are designed to recoil straight back on specialized rests and bags?
Why do light rifles tend not to shoot as well in free recoil as heavy rifles?
Why do larger magnum revolvers tend to hit higher with heavier bullets than light, even when velocities are similar?
Here's an interesting article with some FEA models that suggests that recoil and recoil induced deformation affect bullet POI. Even with a 10lb 6mm PPC.
https://www.varmintal.com/alite.htm