I was faced with this choice a few years ago. A .243 or .25-06 for hunting over a pasture.
I choose the .25-06, and with 115 grain Winchester Supreme BT, I put two down @ 300 yards before you could count to 5. As far as varmints you can load 75 or 87 grain bullets, or you can find a good 100 grain bullet that will do it all from rats to deer, even big deer in the .25-06.
It doesn't have a huge bullet selection, but the caliber doesn't need them. It works, and works well.
I am glad I did not choose the .243. There isn't anything wrong with the .243, but a 100 grain bullet from a .243 makes 3000 fps where in a .25-06 makes 3400+ using a 24" barrel. Recoil? What recoil? Yes, there is a report from shooing 50+ grains of powder, but there isn't enough recoil to say it is more than the .243. A 30-30 shooting a 170 grain bullet has more recoil than a .25-06 shooting anything from 115 gr to 120 gr.
Elk? Dead with a 115 grain Nosler Partition or BT. Probably even a 100 grain Barnes Triple Shock.
The only way I would take a .243 over a .25-06 is if I was shooting 1000 yard competitions, but for hunting, the .25-06 is hands down more gun than the .243. Not only on paper, but for real world shock and awe.