Numerically, that chart seems odd to me. From that chart, the Swiss have twice as many unregistered/illegal arms as legit ones. Now, rather a lot of arms in Switzerland are in the "free" (not needing a license) category--but the inference is that the Swiss are harboring a huge cache of illegal arms.
Swiss have to adhere to the same
Firearms Directive as the rest of the EU countries. The Directive set deadline for compulsory registration of all Firearms by the end of 2014 (with exception of single shot smooth barrel long guns which fell into that category in September 2018). I don't know the particulars of current Swiss law in this regard, but given how eager Swiss government was to implement the 2017 EU Gun Ban I'd not expect any exceptions in the regard of registrations.
So, those are not illegal firearms in the sense they are understood in Greece (i.e. guns smuggled from ex-Yugoslavia and bought on the black market) but firearms that used to be legal and owners simply didn't register them, thus becoming illegal gun owners.
The same happened in late 2000s in Belgium and a few laters later in Austria. I've seen estimates according to which registration rate of firearms in Austria was only 20%.
Meanwhile the Czech Republic had mandatory registration of all firearms in 1998 when there were still not that many after the end of communism. Due to balanced legislation nobody is forced here to go to the black market - and low demand leads to low overall black market size. While number of legal firearms nearly doubled since 1998, the black market remained a niche (similar in Scandinavia).
Meanwhile in places like Germany and France with a lot of red tape and no chance in hell to own/carry firearm for self defense legally black market becomes the go to place also for otherwise law abiding citizens.
Anyway, to sum it up, Austria, Belgium and Switzerland are special apples due to recent legislatory changes. It's not that they have less law abidingcitizens, it is that government turned them into criminals.