Anyone heavily involved in SF/F fandom will have encountered something that shattered their "WSOD" – Willing Suspension of Disbelief – and kicked them out of their immersion in the story to say, in one way or another "What the heck? That made no SENSE!"
As an author, of course, I have to be very sensitive to this; I don't want my readers cranking along happily and then suddenly having their train of thought derailed. This is not, of course, something it's possible to avoid in a universal fashion; things that won't bother 99% of readers will jolt the hell out of that last 1%. But it is possible to avoid for most people, reading most books, but only if you understand what tends to do this… and how that changes depending on the book.
To a great extent, the flexibility of a reader's WSOD is inversely proportional to the realism of the setting. Much of this is inherently obvious; on the one extreme, watching a documentary of the life and times of King Louis XIV of France could end up destroying a viewer's WSOD by simply putting the wrong dress style front-and-center, while on the other most viewers don't even bat an eyeball when watching Bugs Bunny draw a round black hole in midair and then drop it in front of his pursuers, to send them plummeting to who-knows-where. We don't expect much logic in such a cartoon, just (hopefully) clever sight gags, so we almost totally disengage the critical function of the brain while watching.
In writing (or reading) speculative fiction of different types, this sliding scale of disbelief is something that has two major dimensions affected by the reader's personal experience and beliefs. The major dimensions are world (facts and beliefs about how the world functions – science, magic, etc) and people (whether people act in a believable manner to the reader). The latter tends to be more personal to a reader, but even the first is strongly affected by individual reader's knowledge and experience.
"Realism" is also, itself, variable. You can be writing a story that contains bizarre, fantastic elements, but if you set it in a world that is ostensibly very much like our own (as many urban fantasies, such Paradigms Lost or Jim Butcher's Harry Dresden series, are), readers will still apply much of their real-world knowledge and experience to the parts of the world you haven't clearly changed. Thus, even in such a series, a reader's WSOD may be shattered by the author getting a particular real-world fact wrong (say, describing the use of a gun incorrectly).
This can be quite a pain if you intended the "mistake" as a clue. This happens in the first part of Paradigms Lost, in fact; a careful reader who understands cameras will notice that anyone looking through an SLR camera at a vampire should immediately notice something's wrong, because an SLR uses a mirror to send the image to the viewfinder and, of course, vampires don't show reflections. Naturally, this turns out to be a clue rather than a mistake; the guy who was taking the pictures, Elias Klein, was a vampire and thus wasn't surprised by the results. Readers who hadn't gotten that far, however, often complained about what I had "missed".
Part of the technique for reinforcing WSOD on this sliding scale is, basically, to get the reader's trust. In a hard-SF or technothriller setting, this requires getting the reader to "buy in" to the setting in some fashion. A very effective technique is to include a character or sequence early on that draws on something the author knows cold – something they are if not an expert at least well-informed on, and can write about with some authority. Clive Cussler did this with his Dirk Pitt series of technothrillers by always having some scenes having to do with underwater salvage, something Cussler has real-world experience in; the details of those sections were always rock-hard and rang true, because he really did know what he was talking about. This gave a sort of aura of believability around Cussler's prose so that the reader would be willing to stretch their WSOD a bit when they came to the more outré sections of the book.
I've used this in the Boundary series frequently. I know a fair amount about sensors and data processing for remote sensing, so A.J. Baker became a mouthpiece through which I could insert some pretty-darn-accurate science and make the rest of the text, by extension, seem believable and solid.
In a less-realistic setting, the key to maintaining WSOD is almost always consistency. People will nod and keep going through whatever bizarre magical world you create, as long as you don't ask them to believe that all magicians are female on one page, and then introduce a male magician on another page without having some in-universe reaction to the apparent contradiction.
This is where the great paradox of "which is easier to write, hard SF or fantasy" really lives. On the one hand, doing accurate research and addressing all the details of a good hard-SF setting well is no small task. On the other, well, you have the same resources as your readers – you can ask the real world for the details. I could directly contact Dr. Phil Moynihan and ask him about the NERVA project, or Dr. Robert Sheldon about the dusty plasma sail concept.
The writer of an epic (but less scientifically valid) space opera, or a grand-scale fantasy, on the other hand, can make up whatever they like for their universe, but then they have the increasingly huge burden of trying to keep it a working universe, one with consistency that allows a reader to believe all the ridiculous stuff you throw at them – dragons, wizards, disembodied intellects, some guy getting superpowers from being bitten by a radioactive spider, whatever.
Maintaining consistency in a large fantastic universe is a very non-trivial effort. Speaking as someone who has, now, written everything from multiple hard-SF novels to straight-up fantasy, I don't think either end of the spectrum is more difficult to write in and of itself. Each has its own advantages and disadvantages, but in neither of them is ensuring the reader's "buy-in" an easy thing, and in both it requires a lot of work by the author to make it reliably likely that a given reader will accept their world as they read.
Characters, of course, are much more variable. What seems a perfectly believable character reaction to one person may throw another completely out of the story. The example I always recall (without precisely detailing it) is that there is a sequence of events in Digital Knight (later Paradigms Lost) which one of the readers told me was totally unbelievable – that no real-world person would act that way. The first part of the joke there is that the sequence of events was based on my own personal experience, so I knew for absolute fact that a real-world person would act that way. The second part was that the reader in question was my own brother, who one would have thought could have recognized his own sibling's likely behavior.
Because of this, I tend to be a lot less worried about "realistic" behavior in characters. An author can only write characters that behave as they understand makes sense. Certainly an author can try to stretch their understanding of other people and their behaviors, and should if it's feasible, but ultimately we, ourselves, are going to be the limits of our characters. We can't make them any more believable than we are, and as is oft-pointed out, reality is sometimes stunningly unrealistic, and what seems perfectly normal and natural for someone raised in one location with one set of assumptions may seem anywhere from odd to utterly crazy to another person raised elsewhere with another set of assumptions.
Character consistency then becomes the author's only reliable tool here (well, and character appeal, but that's also a pretty darn variable thing). However odd your character's behavior, if it maintains some kind of regularity – the reader can build up expectations of how they'll behave, and find them (mostly) met, or any variations of their behavior explained in some fashion – then most readers will go along with it.
This is helped if any oddities in the characters are reinforced by details of the world – a religion they follow, events that have or are happening to the character, and so on. Knowing that a character was heavily traumatized by seeing some particular type of creature at an impressionable age will let the reader anticipate their negative reaction to an alien that looks rather like that creature, for instance, even if the character is usually an enthusiastic greeter of new species.
This is one of the hardest overall parts of writing in the speculative fiction arena, in great part because there is no certainty in it. Readers have different levels of WSOD, different "hot buttons", different preferences in the amount of detail and consistency they demand.
And, as always, the author's worst enemy is what they don't realize they haven't considered: "It ain't what you don't know that gets you, it's what you do know that ain't so," to slightly paraphrase Mark Twain. The unconsidered knowledge and assumptions – that we all have – are the beartraps waiting in our writing.
In the end, however, the effort to maintain WSOD is one of the most important that an author in the SF/F field can expend, because that suspension of disbelief is what allows a reader to stay in our world, to connect with it, in a way that makes it worth their while to do so.