I think Andy has a point, as in sometimes I read something and think (for example) "oh, so that's where so and so gets its name from...". E.G, until I read To Kill a Mockingbird, I had no idea why the pop group The Boo Radleys were so-called.
That's not exactly an example of what Andy's trying to say, but you get the gist - often things are referenced from other books which one doesn't 'get' if one hasn't read the book. It doesn't necessarily spoil the enjoyment, but perhaps means more if you 'get it'.
Personally I quite like those lightbulb moments!