I expect it's a mixture of both, years of being subjected to be media narratives about how they're to be feared, and seeing something out of the ordinary that took you aback.
I too had an experience along these lines a few days after the Paris attacks and I was so unbelievably disappointed in myself. I was getting on a ferry to go back to uni, and walked up the aisle to try and find a window seat as I usually do, but to no avail. I kept walking and the first available seat was one next to a woman sat fully covered as you described, but with even smaller holes for her eyes. 9 times out of 10 I would just sit down, but in that split second I carried on walking and took a spare seat a few rows up. This woman remained sat on her own and as soon as I had walked past her I immediately thought to myself 'what the **** am I doing? I've fallen victim to the very thing I condemn', but it was too late to turn back
I've never in my adolescent or adult life ever done anything like that, it's totally out of character but these things happen. I still know my opinions on these things and I'm sure you do too, try not to take it to heart.