I am generally a fan of .30 caliber service rifles, but I have owned smallbores too. Never have owned a rifle with a "4" or "5" as the first number, but I do want a .45-70 Marlin lever someday.
I think there is no hard and fast correlation between big bore pistols and rifles as there are different qualities present as regards velocity and ranges involved.
The momentum that makes big bore pistol cartridges attractive for CQC work makes for a rainbow trajectory in a rifle. That is why the ~.30 cal rifles represent one of the best compromises between trajectory and hitting power downrange compared to both big bore and small bore rifles.
Since hydrodynamic effects aren't present at pistol velocities, I see no inconsistency between being a big bore pistol fan and a small bore rifle fan if the small bore rifles float one's boat. Small bore service pistols really only have capacity to recommend them, and sometimes even that advantage is very small when considering high capacity big bores like the XD-45 or the G21.