I rarely give up on a book, unless it's utter crap. For example, (I'm loathe to bring up Dan Brown, but meh), I read The Da Vinci Code years ago and thought it was a ripping good yarn (it damn well was), but then when I picked up Angels and Demons... yeah well I read about 20 pages in, realised it was the exact same story as the Code, and have never (nor will I ever again) read Dan Brown again.
Currently I have a book called Funeral Rites by Jean Genet on the go... it has no chapters... the characters morph into others. Genet jumps from being himself to someone else entirely in the space of a sentence... I can't do it right now, it's too confusing and, quite honestly, a bit of a drag. So it's on hold with all good intentions of being finished.
In other words, I'll give up a book if I think it's trite and dull, I will put a book on hold (and feel the guilt every moment I think of it sitting there waiting for me) if I think it's a worthwhile read, but a little too obscure for what I'm feeling at that point.
(It took me a year to read Mervyn Peake's Titus Groan, and that has got to be one of my favourite books ever, it just had to be put on hold every few chapters)