I checked both the fore end torque and the crown, both of which are fine. I tried free floating the fore end from the barrel as suggested by some, and that only made it worse, so I put an unmodified fore end back on.
The only thing I didn't change is the base. I was using a Redfield 1 piece...