165-168 is the perfect length bullet for a 30-06 with a 1-10 rifling, it just doesn't get any better for pin point accuracy with that combo. It's like that with most rifle calibers, there is always one length bullet that is optimized for accuracy with a given caliber because of it's rifling twist and diameter.
I also agree that the heavy recoil of shooting longer (heavier) bullets may be affecting your accuracy, especially if your sitting down at a bench. You have to spin those longer (heavier) bullets faster to get the same accuracy in the same rifling that shoots 165-168gr bullets well. ie- 1-10 rifling that most 30-06s have.
That means heavier recoil that your shoulder has to eat while your sitting at a bench, half laying down to shoot.
When your standing upright, or for that matter, sitting upright, it's a whole different set of ergonomics on your shoulder because your shoulder can move with the recoil instead of getting mashed.