To swedes, taking your shoes off when entering a private home, is as natural as the concept of being shoeless itself

No one have a problem with it.

And why would anyone?

Here is the "Take your shoes off/Ta av dig skorna" doormat.

People don't buy the mat because they fear anyone would want to keep their shoes on. They buy it because they want to mention the title of the classic Powel Ramel song Ta av dig skorna.