Sometimes there may be the odd book that I think, Oh I REALLY must read that. I would like to get those essentials read, before, its too late I guess. But there are not many books on my TBR like that.
Even then, I don't think it would bother me that much. Like FA says, I would be more worried about the things I hadn't done I expect.