Yeah, I suppose it could be called selfish, but I have a strong phobia of buzzing insects around me... especially if they fly in through a window and they won't seem to go back out. Also if it's at night, I have a fear of being asleep and insects flying around.
Having said that, most insects I'd just leave be, it's only the really buzzy ones that annoy me.