This is because it's usually merely the retort of a person who can't defend an idea they believe and have to resort to feeble insults to shore up their defenses.
If wanting good reasons for thinking something makes me "elitist" then so be it.
If putting forward my own ideas in a reasoned argument makes me "arrogant" then I'll own it.
If wanting clear evidence for my significant beliefs about reality makes me "narrow minded" or "skeptical" then I'm very happy to be so.
However, none of the three labels above are actually correct, fair, or reasonable.
My suggestion to others? Don't take yourself too seriously, don't be afraid to say "I don't know", and become your own worst critic. Everything else will come out in the wash.
Here are some quotes from some of history's "narrow-minded":
"Two things are infinite: the universe and human stupidity; and I'm not sure about the universe." ~ Albert Einstein.
"Most people would sooner die than think; in fact, they do so." ~ Bertrand Russell.
"I respect faith, but doubt is what gets you an education." ~ Wilson Mizner.
"Skeptical scrutiny is the means, in both science and religion, by which deep thoughts can be winnowed from deep nonsense." ~ Carl Sagan.
"Ignorance is preferable to error; and he is less remote from the truth who believes nothing, than he who believes what is wrong." ~ Thomas Jefferson.