You disagree, I assume? Do you know of someone who only fired a non magnum firearm less than 5 times/ year without hearing protection that now has significant (that's bolded for a reason) hearing loss that can be attributed to said firing?
Hmm, sounds like you are searching for a definition of "significance" and on that we will undoubtedly differ. When you say significant, do you mean statistically significant? Do you mean for 'significant' to comprise more than half of the hearing available hearing being lost? Maybe 25% lost? Is significance when it impacts your lifestyle? Is significance when somebody is legally considered deaf (Generally occurs at the 'severe' level of loss)? The reason why I ask is because "significant" isn't a classification for hearing loss. Mild, moderate, severe, and profound are, however. Some groups also add in a 5th, 'moderately severe.'
If it is measurable loss, then it is significant, according to my audiologist. She sees it that way because hearing doesn't recover. You don't test at a Moderate loss level, drink some power drinks for 6 weeks and come back and retest at the Mild level. It is gone.
I would argue that if you can't hear any of the tones that you should be able to hear for your age bracket, you have significant loss. You may have significant loss even if you can hear them, assuming you had better than average hearing to start. Until about 10 years ago, my hearing was one age bracket better than average for my age, though with slight right ear deficit (could not hear one of the tones I could hear in the left ear). Now, I have mild hearing loss
To answer your query, I certainly can't match the exact parameters you have stated and neither do you. Neither your or I have a way to control for a sample of people that incur no other auditory insults during their lives other than 1-5 times per year while hunting. Even if we did, neither of us has the data from these individuals as to what their hearing was like (fully tested) before they got into guns. If you don't know the starting point for each, then you can't say for certain how much loss has occurred. That is a reality. All that they can determine is how a person hears at the time of the test and compare that with demographic averages.
However, shooting a centerfire non-magnum rifle without hearing protection will damage you hearing with every shot. You likely won't recognize it is even happening. I have known several deer hunters who only fire a few times a year (until tagged out on deer) and who don't wear hearing protection while hunting. All of them have measurable hearing loss. None of them shoot magnums that I know of, one shooting a 30/30 and the other shooting .308 that I know of. The last guy shoots a Grendel now but using to shoot a .243. Two wear hearing aids. Two others, I have no idea what they shoot, but I don't think they are magnums. One guy told me that he didn't realize how much hearing loss he had until he had trouble hearing the coos of his grandbaby. Despite the fact that he doesn't wear hearing aids and generally has normal conversations with people, not being able to hear his granddaughter very well has definitely been significant for him.