Almost any kind of book can inspire a science fiction game, from a science fiction novel, to a book on phone phreaking. Sometimes I will get an idea while reading a history book or a social science book. The inspiration almost always takes the form of a question, from the dumb, "What if this happened in spaaace?" to the more complicated, "What would a democracy look like if everyone could vote on hundreds of little issues every day through the internet?" Even when I am dealing with a licensed scifi game like Star Wars, the basis tends to come from a question.
To a certain extent, this is true for my fantasy games as well, except that the question is almost always, "What is over there?" I am not a big fan of epic, world-saving fantasy. I find that it all tends to be very derivative of Lord of the Rings, and very backwards looking. Epic fantasy tends to ape Lord of the Rings without thinking about what it is doing, romanticizing pastoral, rural, medieval England even when written by an American. I do not find this to be fun. The events of fantasy are very, very boring. Exploring fantastical locations, on the other hand, is interesting stuff.
Because of this, my fantasy games tend to be site based, not event based. I get excited by the big world maps that come in campaign settings. I trace the rivers and roads with my finger and look for places that call out to be explored. I like my campaign setting books to come with brief, evocative descriptions of the locations on the map, and more detailed maps of those areas. I am not going to read the painful elf history or list of dead kings. Just give me the map and a few lines of description.