50 grains is the absolute maximum you should use in a GPP, and even that's too stout for me. The problem isn't the perceived recoil, it's the effect of the actual recoil on the stock. Shoot more than 50 grains and you will eventually crack the stock. And in my opinion, even 50 grains is risky. I know, it hasn't happened yet, but I think you're just marking time.
As regards using shot pellets in the GPP, it can be done but effective range is about 10 feet. The GPP has a rilfed barrel; shot pellets will exit the barrel as a rotating column and form a donut-shaped pattern with a large hole in the center. After about 10 feet the pattern becomes diffuse and is essentially no more lethal than individual bb's. That's the reason shotguns are smoothbores.
But, I suspect you're going to try it anyway, so here's what you do: load 30 gr fffg and cover it with a thin card made of heavy paper, like an index card; this is called an overpowder card. Then cover that with a 1/8" thick dry felt wad, called a cushion wad. Load about 1/2 to 5/8 oz of No. 6 lead shot and cover that with another card, called an overshot card. This load will be effective for snakes at close range. Beyond that you're just causing painful injury.
Finding the proper diameter cards and wads will be a bit difficult. Some sutlers do carry .50 cal rifle wads that will work for the cushion wad; you'll probably have to make the overpowder and overshot cards yourself. Nobody makes a .50 cal shot cup, so that's not an option.
Personally, 40 gr is not a load I'd try for whitetail, even at longbow ranges. You'd need a nearly perfect heart or lung shot for a one shot kill and anything less is unacceptable to me. I'm not a good enough shooter to take that shot and risk just maiming the animal. As far as rabbit is concerned, .50 cal will leave you with little edible meat unless you take a head shot only. .36 cal is really the maximum rabbit gun, and .32 cal is the optimum size. I do know a guy who uses a .40 cal custom long rifle, but he's good enough to always get a head shot even at 50 yards; very few of us are that good.