A couple of studies seem to indicate that the gender of a performer has an impact on the way we hear his/her music. From Pacific Standard magazine:
One study reported that knowledgeable listeners described the same performance as more “precise” when performed by a man and more “dramatic” when performed by a woman. Another found that people told (accurately) that they were listening to a Brahms symphony conducted by a woman, Marin Alsop, judged the performance as less powerful and more delicate than those who were told the conductor was a man.
Music may be an abstract art, but our reactions to it are shaped by the assumptions and prejudices of our society.