Obviously it peaked with Leona winning, and as bad as 2007's winner was, the show was still decent and had a lot of controversy. Series 5 was entertaining despite no Sharon and also probably the best year for talent and most successful year for the finalists altogether, with quite a few having Top 40 entries, plus the first charity single.
Series 6 is where it started to become really crap for me, with the live audience for auditions. It removed all the classic judge moments. Then last year, I didn't even want to watch, but did because it had become such a tradition by then.
This year it will properly decline, but I think it went downhill after Sharon left. She isn't the best judge but she was entertaining and without it the show became less interesting.