The HazMat fee was started by UPS and Fedex back in the early 1980's. If I remember correctly is was $2.50 at the time. I had my FFL during that period and you could get a rifle shipped for about $5.00 to your door, and the same for a pistol.
The HazMat fees apply to other commodities as well, but the ones we notice are the fees for primers and powder. There is no HazMat fee for primed cases or loaded ammunition.
The fees have gradually gone up over the years, and I've yet to see a documented case of either primers or powder causing a problem during transit. The fees have probably been for the most part to pay for the HazMat training required by DOT for the carriers, since it's mandated, and expensive.
Hope this helps.
Fred