This sounds like a simple ammo related problem. If your rifle is shooting fine and accurate with your handloads, and all over the map with factory loads, clearly the rifle isn't at fault.
When I take a new rifle out for the first time I almost always run a box of factory ammo just in case something goes wrong (I'd rather not have it blamed on my handloads). I did this same procedure when I first got my Accuracy International AX260. That's a very high-end rifle, and a very accurate rifle. I started out with a box of factory Remington Core Lokt ammunition, and was dismayed at how inaccurate this stuff was in my new fancy rifle. I want to say that it was grouped around 2-2.5 inches, but I'd have to grab my logbook to check for sure.
Anyway, after about twelve rounds I gave up and moved over to my handloads. Even using test ammo that didn't end up being my final selection for this rifle, my handloads started putting the shots right on top of each other... what followed was a single hole group of ten rounds (a bit smaller than a dime).
My point? Some rifles just don't like some ammo!