How much swabbing one does depends on different factors such as:
1. How much powder is being loaded.
2. The type or brand of powder being used.
3. If the fouling is causing difficulty loading.
4. If accuracy is suffering noticiably.
The way bore butter (or other patch lubes work), is that it keeps the fouling soft enough so that the next loaded ball can push the residue back down the bore where it can be shot out along with the next load, in effect swabbing while you load & shoot. But this does often require shooting with moderate powder loads and having an adequate (or extra) amount of lube impregnating the patch.
Using more powder than needed (as in shooting heavy hunting vs. light target loads), or using a patch with not enough patch lube, can allow more hard fouling to accumulate than is able to be "shot out" by the subsequent loading.
Then the heavier fouling may need to be pulled out with a wet patch followed by a dry patch instead of simply using a dry patch which can push dry, hard fouling into the breech.
I use a lot of BB on my patches and much of the excess gets squeezed out at the muzzle during loading. Longer barrels may also require more BB.
Then it becomes the amount of powder that's being loaded that determines how long shooting can continue before swabbing is necessary.
A .50 caliber gun loaded with about 50 grains of Pyrodex, can be shot for hours without noticing very much accuracy change or fouling build up.
Some shooters who live in arid climates do report a totally different phenomenon though. The dry air seems to produce a much harder, drier cake like fouling which causes more problems than in the more humid eastern states.
So depending on where one lives and the powder type, the fouling may have different characteristics which can affect swabbing frequency.
Best target accuracy is often achieved with more frequent swabbing though.