When I was a kid, one could not use a rifle smaller than .25 caliber in Texas on deer. The .243 had just come out. Weren't long before the law was changed as the .243 was popular on whitetail. I shot my first deer at age 11 with my grandpa's M722 Remington in .257 Roberts. I still have that gun.
I hear LOTS and LOTS of folks using .223 in a semi auto for their kids first deer rifle. I always figured if the kid couldn't handle a .243, he shouldn't be hunting deer. JMHO, though. Recoil is something you just get used to. I've shot everything up to .375 H&H Magnum. I don't find belted magnums "scarey".
Sure, they kick, but I'm an accomplished rifleman. I can ignore the gun and concentrate on trigger and sight picture. Heck, goose hunting, I shoot a 10 gauge H&R single shot with 3.5" steel Ts.
If you hunt with HUNTING rifles, you get used to recoil and .308 is NOTHING to worry about even in a little lightweight like my Remington M7. When I shot that first deer with my .257, they hadn't adopted the M16, yet, don't know that it was even out of development. .222, .222 magnum, and .22-250 were considered prairie dog calibers. You hunted DEER with a .30 caliber. My grandpa got kidded mercilessly about his "pop gun" .25 caliber.
And, think about this...I hunt doves and ducks with shotguns. I shoot a 12 on ducks. I now shoot a 20 on doves, but have shot 12s in the past. I can go through five boxes of 12 in 4 days of a good hunt. 12 gauge, even dove loads, is probably as rough as .308 in a light rifle.