I would never be in favour of banning it. Kell's Robinson Crusoe point is an excellent one both in that it's not a racist book, and in that it's an illustration of cultural change, something I don't think should be censored either.
Overall, I tend to be of the opinion that you can't say ANYTHING anymore or you're racist or sexist or some other 'ist', and it's hitting a point of ridicule. If people don't want to read anything that might be construed as racist, don't read it. If you read something and suddenly hate black people, you were probably doomed to begin with.
Personally, reading a book I later felt was purely racist without dealing with the subject objectively, it would make me LESS racist because I would think, 'I don't want to be like that'.
I think it's an incredibly subjective topic, and that it's too subjective to really draw a definitive line.