It's pretty obvious he's threatened by Raph's extensive knowledge of Big Brother and could be partly down to Raph's VT saying he doesn't care about anyone and he will manipulate anyone he can to win - Tom will have told Joe about it.
I don't see the issue really, if somebody said the things Raph did going in I think anyone would be justified in being wary
