I have carried both (one at a time, on different hunts).
I have purchased several different guns with the excuse of using them as a belt gun while hunting. Here where I live, big game hunting usually invovles driving hundreds of miles and then camping at your hunting spot. So, I carry a handgun the whole time I am there. At the end of the day, I often lay my rifle down on the front seat of my truck, but obviously I carry the handgun everywhere I go. When you are hunting out here, you are usually WAY off the beaten track, but I have still had people show up after dark at my camp fire. It was definitely a comfort to know I had a pistol on my belt. It is also comforting in the tent at night to be armed. We do have mountain lions in addition to people here. I also have been known to take a shot at jackrabbits, coyotes, etc. It is more fun IMO to shoot them with a pistol.
I actually had to use my "back-up" handgun to shoot a deer once. This is a good story IMO:
I walked up on 4-5 mulies standing in an open area at the distance of about 50 yards. I picked out the biggest buck and shot it with a .30-06 right behind the shoulder. When the rifle came out of recoil, the deer was gone. The others were still standing there looking at me, but the one I shot just vanished. So, I put down my rifle, took off my pack, took off my coat, rolled up my sleeves, got out my knife+paper towels and all that stuff and then walked over to where the deer fell. I got about 10 yards away and the deer jumped up and started diagonally away from me. So........my rifle was lying back there with all my other stuff. I drew a Ruger P89 out of a shoulder rig and shot twice. Again the deer dropped in it's tracks after the second shot. I congradulated myself on the excellent shooting and the awsome power of the 9mm cartridge.
When I gutted and skinned the deer, my first shot with the rifle was right on the money: it's lungs were distroyed. The 9mm rounds however played no role at all in the animals death. Both my shots hit the deer in the back hip and didn't come near the vitals of the deer. The deer fell over dead because it's lungs were made into pudding by the rifle. The fact that I happened to fire a pistol at it just prior to it falling over was a coincidence.
My current belt gun for big game hunting is a S&W Model 629 with a 3" barrel. This goes into an El Paso Saddlery Threepersons holster. The load is an max loaded LBT 300 grain cast bullet.