I agree that while writing you shouldn't think about the potential readers too much. But I do think you want to speak to as many people as possible with your art. At least, that would be something I would like to be able to do.
But off course there is the possibility that you want to be understood only by those you feel are 'worthy'. And a way of realizing that is by making a book at least a bit difficult to read. You don't need to see Ulysses as difficult or pretentious, but at least admit it's no ordinary book, written for everyone. And it's not even like he used to write. So it might be a masterpiece, but he did alienate a big part of his audience by writing it this way.
And as I said, you might be fine with that, but I prefer reading more comfortable stories and I think Nabokov is a great example of that.
Art has always been a way for the elite to distinguish themselves from the 'normal' people, and I think Ulysses does an excellent job at this.