Sometimes it's necessary to distinguish somebody's race, i.e. in a discussion like this, so no, it's not always 'racist' to refer to someone by their skin colour. It's descriptive, not racist. If you want to press on with that logic then we can't call people fat if they're fat, even if it's that one of the first things you'd think of when you were describing someone who was overweight.
I don't think anyone in this thread has said it's "unfair" that white people "can't" use the n word. The problem is that if a black person says it to another black person, obviously it's not (well, usually not) intended as a racial slur, it means something else. What exactly it means hasn't been officially defined by any dictionaries as far as I know, but it doesn't mean what it used to mean. If a white person said it to a black person, they might intend it in the same way but by virtue of their race, it will come across as a racial slur. That's why I'm of the opinion that the word should either be put to rest or there should be some kind of drive to change what the word means, to properly reclaim it. Having it floating about in this limbo stage where it means something else to some people but not to everyone means that the world at large cannot move on from the historical past of the word. It's up to black people where to go from here with the word. Stay as things are and racism will never go away, it'll always be around us. Try and progress and things can only get better. The civil rights movement didn't just stop when slavery was abolished.