Sometimes it's the Howard Roark problem

New York Times writer Rachel Donadio explores the “It’s not you, it’s your books” problem in romantic love — a problem very familiar to many admirers of Ayn Rand’s novels.
From her perspective, though, it’s the like (rather than dislike) for Rand’s novels that indicates middlebrow tastes:

Pity the would-be Romeo who earnestly confesses middlebrow tastes: sometimes, itâ??s the Howard Roark problem as much as the Pushkin one. â??I did have to break up with one guy because he was very keen on Ayn Rand,â? said Laura Miller, a book critic for Salon. â??He was sweet and incredibly decent despite all the grandiosely heartless â??philosophyâ?? he espoused, but it wasnâ??t even the ideology that did it. I just thought Rand was a hilariously bad writer, and past a certain point I couldnâ??t hide my amusement.â? (Members of theatlasphere.com, a dating and fan site for devotees of â??Atlas Shruggedâ? and â??The Fountainhead,â? might disagree.)

Indeed.
Sounds like they both dodged a bullet. Two cheers for being unable to hide your own literary tastes!
And I’ll second Donadio’s suggestion to fall in love with someone who shares your taste in art. It makes a lot of other things go more smoothly.