I'm starting to think that the shorter the fiction, the more important it is to twist it. I'm using "twist" here in its broadest sense as something unexpected (as opposed to a twist that changes the meaning of everything that has happened in the story), so heck, maybe all fiction needs a twist.
One problem is predictability. If the reader sees it coming, then every time you hint at it or try to misdirect, they'll get increasingly annoyed. "Look, I already know the mysterious stranger is his father. Get on with it." This problem is exacerbated by the fact that so many twists are cliche (e.g. see Strange Horizons' list of stories they see too often, #9).
This can be tempered by using a smaller twist. Imagine twists on a continuum; on one end, there are the big twists: "OMG, Bruce Willis is dead!" or "OMG, Kevin Spacey is Keiser Soze!"* On the other end are smaller twists: Frodo decides not to destroy the Ring or Marlin sees Nemo in a plastic bag and thinks he's dead.
The bigger the twist, the harder it is to pull off. Readers are pretty good at figuring out what's really going on. But when a big twist does work, it's mind-blowing. Smaller twists, on the other hand, won't blow anybody's mind, but they're still interesting and much easier to pull off.
But what does it take to pull them off? Here are some ideas:
- Practice. Try different things and see what works.
- Read a lot. Among other things, this will help you know what's cliche.
- Avoid hiding information from the reader artificially. If the assassin is the POV character, and he knows all along that the target is his daughter, hiding the info from the reader will only annoy them. (Exception: the unreliable narrator; e.g. give the assassin a reason for hiding it from the reader).
- Misdirection. Set up something else that appears to be the truth. This is tricky, though, because it can't be too obvious. Like in Scooby-Doo, you knew the ghost was never the mean old janitor who hated the museum curator.
- The twist should be better than the straightforward ending would have been. Like I've said before, don't twist just to be unpredictable. This is one reason why the "it was all a dream" ending always fails.
- The twist should make sense. It'd be nice if this went without saying, but...
* If you haven't seen these movies, I'm sorry. On the other hand, they're old - way past a spoiler moratorium, I think.