The OP has discovered exactly what gunmakers, gunsmiths, and accuracy shooters have known for quite a while - you can install the sights perfectly aligned to the OUTER dimensions of the barrel, but it really has nothing to do with the INTERIOR bore and the dynamic placement of shots fired.
Just because a barrel bore doesn't exactly duplicate the outer circumference doesn't make it a lemon. It does, in fact, make it entirely normal and it's the reason why sights have been adjustable since the invention of guns. The first barrels bored proved to the barrel makers that their skill at getting them concentric wasn't as exemplary as they wished. Since then things have gotten a lot better - for the most part it's only bespoke shotgun makers who actually bend the barrels to change the point of impact.
Think about that for a minute.
The custom of the day and still practiced to this day is to literally put the barrel in a vice and then bend it in the direction needed to get the shot group placed. On shorter thicker barrels that was much more difficult, so they simply moved the sights to align the point of aim to the point of impact.
One other important factor in aligning impact to aim is the muzzle - where the barrel touches the bullet last on exit. Given two identical barrels shooting identical ammo, having the muzzle identical is nearly impossible. That alone can cause a bullet to vary it's path on exit creating a measurable difference. The simple solution for one far out of tolerance is to recrown it. Hopefully the results improve, but their is no guarantee. How the round chambers can also make a difference, as the leade - the portion of the barrel in front of the brass where the bullet engages the rifling - can cause the bullet to line up coaxially, or not - which results in the bullet nose being off axis and that results in it being tilted on exit at the muzzle. It has to stabilize in that flight path which can result in a consistent impact off the point of aim. Reaming the chamber often corrects that.
There's a lot more to it than just the sights sitting on the outside of the barrel. As was discovered long ago.