Free speech is not the same thing as consequence-free speech. Other people also have free speech rights to label you, as well. It may not be accurate, or even fair, but free speech works in all directions.