Well I went to school in Scotland so the systems are a bit different but I think our situations are similar - we're told we have to take 5 Highers in 5th year at school, Highers being the results that Scottish universities look at for admission. I took 4 Highers (English, German, French, Modern Studies) and 1 Intermediate 2 (Maths) because a teacher bullied me into taking Maths, saying how important it was to have a Maths qualification and I would do the Maths Higher in 6th year... Scottish unis look at your 4 best Higher results for admission, so the pressure was on me to do well in my other four subjects which I was quite good at, but Maths I was always rubbish at and even though it was a level lower than my other subjects, I got so much homework and it really took up my time. In the end I stopped caring about Maths and focused on the other four subjects, got 4 A's in them (and in Maths too
but the exam was piss easy, our teacher had been more or less scare mongering us that it would be awful, so props to him for working us so hard I guess..) and got four unconditional offers to uni and one rejection (from St Andrew's, they rejected me because I didn't do 5 Highers in one sitting)...
So to answer your question - if you want to go to the most prestigious sorts of universities then I wouldn't recommend dropping any subjects. The Oxfords, Durhams etc of this world won't look twice at your application. I'm at the University of Glasgow which is one of the better Scottish universities and I got in with four Highers no problem, so I'd say just try your best with your other subjects and if your Psychology teacher doesn't like it then so be it, you shouldn't be working yourself that hard.