If your guns can safely accept the 3" shells, they will provide more payload (additional pellets) and/or more velocity and, of course, more recoil.
Sounds like what you have is "double ought" or 00 buck. A typical 2 3/4" 12ga shell would contain 9 pellets of 00 buck which are each 0.33" in diameter. You could go to larger pellets, that would be "triple ought" 000 buck which are 0.36" pellets. Of course your payload (number of pellets) will decrease because fewer of them will fit into the shotshell. Figure on about eight 000 pellets in a 2 3/4" shell and maybe 10 pellets in a 3" shell.
The larger pellets are also heavier, and that means penetration will tend to be better than with smaller pellets. That said, I would think 00 would provide decent penetration out to the ranges that it should be used on big game.
If you go with .20ga, you'll probably have to go with something smaller than 00 buck. I don't think anyone loads 00 buck in 20ga.
To determine the effective range, you're going to have to pattern the load you pick in both guns at various ranges. I'd say that in terms of pattern size, your effective range will be the distance at which you and the gun can consistently put 3/4 of the pellets on a paper plate.
To stretch the range, you can choose the copper plated pellets as they tend to hold a tighter pattern--I think Federal and Hornady have some loadings that have a good reputation in that respect. Just keep in mind that shotgun pellets are not very good projectiles and will lose energy rapidly as they go downrange compared to a more aerodynamic projectile like a bullet. So even if you get a very good pattern at, say 50 yards you need to keep in mind that you're still talking about "0.33 caliber bullets" that each weighs only 54 grains. By 50 yards, the energy has dropped to about half of what it was at the muzzle.
