I know when I saw George walk off in the kitchen and he was just sat there alone it was so hard to see! And having James etc. telling him it's annoying when he enters a conversation and can't understand must knock his confidence to engage with others too.
I just really hope it all goes over Gary's head and doesn't hurt his feelings. Sometimes he seems not to take their sh*te to heart but he has mentioned a few times that he's sad that everybody hates him