I think it's probably several factors all at once.
As previously mentioned, a 20" A2 is big gun for a .223, while a 16" carbine is not.
Collapsible stocks offer some advantges in handling a rifle, and there are many more companies offering collapsible-stocked 16" carbines versus 20" rifles.
The abundance and popularity of the optical sights eliminates the negative of a short sight radius.
For shooting inside 300 yards, the drop off in ballisitic performance in going from a 20" rifle to a 16" carbine is not so great.
As was stated earlier, most people who are buying guns are recreational shooters. A shorter, lighter 16" carbine AR that can fit you better (collapsible stock) that will shoot just as well as a 20" rifle (flattop) at the ranges your are likely to shoot at, will tend to attract buyers.
ARs are now being recognized as excellent CQB/home defense weapons, and is beginning to start challenging the shotgun for use in home protection. If you're keeping something by the bedstand, it is preferable for it to be short and light. This regime is where the carbine really shines when compared to the rifle.
The one place that the carbine stands no chance against the rifle would be in offical Highpower competition. There, you ARE shooting to 600 yards, ARE required to use iron sights, and the longer and heavier 20" rifle is much superior.
At any rate, I don't see the 20" rifles going anywhere soon. They're just being complemented by the carbines.