You'd need a .38 or .39 caliber rifle to use the balls that will fit a '51 Navy. Those Colts use .375-.380 balls, a .36 muzzleloading rifle would use about a .350" ball which would not be safe to fire in a Colt.
But yes that IS a good idea, and I and others have oft wished those calibers would be more popular so we could use such balls in our rifles too. Once cool thing though is you can buy the bulk Hornady 5 lbs of 000 buckshot and it is the perfect size for a .36 muzzleloader and is super cheap, between 3-4 cents per round, making it a very affordable caliber to shoot. That's one of the reasons why I am getting one.