I don't find him annoying nor patronising at all, and he is always spot on when giving analysis on the other HM during diary sessions. He is one of the only three reasonable HM in that house.
I'd wish that freddie would purposely annoy them, although he is not that type of person so unfortunately that won't be happening..
Blows a bunch of kisses to my sweet Freddie.