If he had remained as a replacement for Louis in Series 4 and beyond? I think it would've been interesting to see, but of course Louis is better.
I think had he stayed Sharon would've got the Groups for the first time ever that series as well. They always lump the new judges with the best category, so Dannii got the Boys, Brian would've probably got the Girls [Sharon's category that year], there's no way he would've got the Over's [Louis's category] and Sharon had the Over's in Series 3 so it would've had to have gone to Simon. Leaving Sharon with the Groups [the only category she never had].
Maybe if he had stayed we would've ended up with a Brian - Dannii - Cheryl - Simon line-up? Funny how things work out...