I learned how to tell stories by running role-playing games.
I've run Call of Cthulhu for fifteen years. A friend of mine bought me the book at some point in high school (thanks, Jeremy) and I fell completely under it's spell. I played with a group all through college, I've done six years of convention horror events and I got really, really good at this stuff. I've never met anyone who runs horror games better than I do.
This is how you do it.
1: Most of the ideas a lot of GMs have for increasing player tension (taking their gear, taking their character) aren't so much scary as GM Fiat. Yes, most role-players, especially people who favor combat games like Dungeons and Dragons, rely too much on weaponry. This shouldn't be a huge issue for games like Call of Cthulhu as most of the monsters are barely effected by human weaponry, but plenty of groups run under the philosophy that anything dies if given enough rounds. This is a problem for most horror fiction in general. When you look at stuff like Buffy the Vampire Slayer, it's really an action show with people beating up monsters instead of criminals or soldiers or whatever. Most RPG characters are at least competent in combat.
The shortcut to good character conflict is to put your PCs in situations that your characters AREN'T already equipped to deal with. There's a reason so many great stories involve meek characters who rise to challenges or strong characters learning compassion from being forced to interact with softer counterparts. Figure out who your characters are and then create situations that take them out of their comfort zones.
2: A lot of GMs believe that horror role-playing is primarily about off a bunch of player characters. Horror role-playing isn't about amassing a body count but it's about creeping out the people at your table. A dead character is at best a distraction. It's not that hard for a GM to get a body count. It's a lot harder to get people emotionally invested enough to scare them.
In that vein, don't get too in love with descriptions of gore. Good gore can be evocative, but too much gets fappy.
3: (the big one): You HAVE to control the environment you play in. You cannot scare people in a brightly lit room with music playing and cell phones going off. You can maybe evoke the trappings of the horror genre in your game but without the proper physical environment you're just playing an action game dressed in fangs and a taffeta cape.
You have to play in the darkness. Because I favor the simplicity of Call of Cthulhu's rule system, I play in a small circle, either by candle light or by a single very dim light. My character's most important scores are written very large on index cards so they can be seen in dim light.Have your players use their cell phones so they have just enough illumination to read the cards for their skills. Have dice corrals so they don't hop all over the place in the dark. Distractions are kept to a minimum and I take a ten minute break every hour so people can pee and fiddle with their phone. If you're doing your job right, people will need breaks in tension.
4: You have to cultivate a ghost story voice. I tend to view horror role-playing as an interactive campfire ghost story, which is why I take such effort to control the environment. Keep descriptions very short, but with strong central imagery for the players to work off of. For scenes of high tension, where I have my characters creeping through an old house, I drop my voice low and make it soft and feminine. When my players encounter a scene of awful violence, I break up my descriptions and raise my voice in hysterics. When I come to a point of obvious danger, I stop speaking abruptly and force the characters to make their choices from a point of imbalance.
And, yes, I occasionally slam my hand on the table. I used to scream, but that shit was corny. A heavy book slammed down gives you the jump scare you occasionally need without being comical.
5: As much as you need to cultivate the right physical environment, you also have to cultivate the right group.
There are a bunch of people I play with in other genres who I'd NEVER put in a horror game. They like goofing around, they undermine mood, and they don't engage with the in-game world in a serious way. They're basically playing Grand Theft Auto in any game they're put in. That's totally fine and they're a lot of fun when I run superhero games, but I need someone who's willing to buy in to the mood. A lot of people simply can't.
6: Combat is the hardest thing to pull off in horror gaming. Most RPG combats are either tactical by nature, where you have to problem-solve as much as fight, or they're like a football game where two groups of bruisers whale on each other. Combat takes the GM's role from active to reactive, where the players and their decisions are in charge and you are bouncing off what they do. It turns atmosphere and storytelling into a series of numbers.
Horror is about powerlessness. Most gamers don't like that feeling. If you want to get that feeling across, have your player's goal be less about killing or subduing the monster and more about escape.
Example. Most character-to-character fights are like Jason Bourne vs. some other Treadstone assassin. They're both highly competent and evenly matched and it's a skill-vs-skill thing until the hero triumphs. A horror fight should be like Leatherface trying to capture a frightened teenager. She crawls into someplace small to hide, he's reaching for her, she's kicking his hand away and hitting his arm with a wrench she grabs off the ground. He's stronger than she is, she can't do much damage to her, but she might be able to fend him off.
7: As an example, I ran a CoC event every year at a gaming convention. I requested a private room so I could control mood and set the scene for the players before starting. You have to create fairly conservative scenarios when you're running convention games and I ran a nice simple story involving Cthulhu.
The hook of the story involves the ghost of a little girl. The girl's father was a well-known and successful artist who started having dreams about R'lyeh. As his visions became more apocalyptic, he drowned his daughter in his bathtub to spare her from the second coming of Cthulhu before hanging himself. The players all knew these facts before entering the family's abandoned house in search of some Evidence.
When they got to the house I turned ALL the lights in the room off except for my tiny central one. I described the house in very simple terms, basically that it looked like a normal for-sale property but knowing the sad history of the place gave it an ominous feel.
When they said they approached, I paused. Without saying anything like "are you sure", I made it clear by slowing down the way I spoke and pausing at points that they were entering hostile territory.
After screwing around and searching a couple of rooms, I had them make listen checks. One made it and I whispered in their ear that they heard splashing and the sounds of struggle from the rear of the home. The player passed the information onto the others (it works better than a general address to the group, which prevents the game players feeling like a hive mind and casting doubt on the bearer of the information)
They find the bathroom that the artist drowned his daughter in. I make it a point to describe it as antiseptically white and clean but that the tub is full and there are lots of strands of jet black hair (my ghost child had long black hair.) As they're standing in the doorway, I dropped a heavy book on the table to symbolize the bathroom door slamming shut. As the players freak out I describe, in fast breathless panic tones, the sound of the father drowning the daughter from behind the door.
At this point I ask the person who has the lowest current sanity score to make a POW x3 check. He/She does and I make a note of that. I tell that player that their character has wandered off. The other players have been distracted by the sounds and they believe that it's possible one of their number could have slipped off.
At this point, I have the others making a listen check. While they're doing so, I tell the POW x3 player that he/she is in the master bedroom and he's looking at the ghost of the artist who hung himself. The hanging didn't go well. I have him/her make a sanity check and, whatever the results, I ask him/her to freak out when the other players find them.
The players return with their listen results. They hear the sound of a rope creaking and realize their friend is missing. The sound leads them to the master bedroom, they see the ligature mark on the beam, and the other player is freaking out. In the bedroom is the clue they needed. That's it. No guns fired, no players wound up dead, but that scene works EVERY DAMN TIME.
Running horror games has been one of the most rewarding experiences of my life. It taught me a lot about creating and sustaining atmosphere, helped me perfect my public speaking skills, and given me the tools to tell a story. Over the last few years I've shifted to writing both prose and scriptwriting, but I've also come to miss the immediate thrill of running scary games. You have to think on your feet, your audience constantly challenges you, and you learn how to read your players and develop new ideas on the fly. All of these skills are essential for all types of storytelling and I am deeply grateful to the games that help me develop as a writer.