Well, I'm an exception to your theory, Raven! I pretty lucking that I can remember the plot of most of the books I've read going back for years, and last year I definitely whizzed through books at a rate of knots!
In fact, I'm sometimes amazed at how good my memory of books is, right down to usually being able to find a quote or section quickly within a book I haven't read for years. I was thinking about a book I read the other day, thinking I must have only read within the last couple of years, as I remembered clearly a particular plot thread, but when I looked it up, I realised it was at least two years before I started keeping a record of my reading, so must have been about 2004!