Stephen King's early work did this very well, especially Pet Semetary and the novellas in Five Past Midnight. HP Lovecraft's writings never had a happy ending, the majority of his characters went bat crap crazy.
One of my favorite endings in movies is John Carpenter's The Thing. The ending essentially is two characters left alive in the ruins of their research base in Antarctica. Both are exhausted and paranoid that the other is actually The Thing. They have no way out. There's no escape. There's no help coming around the corner. It's an ambiguous ending and its unsettling. My imagination tends wonder about the possibilities of what could happen to them. I'm not just a passive viewer in that ending, it engages my curiosity and imagination. I love that kind of ending, too. I don't need the movie to hold my hand and explain everything out to me every time.
In a lot of ways, endings are just as important as hooking the reader in the beginning. A well written ending can stick with me for a long time and encourage me to return to read/view more of that person's work.
Gunsmoke & Ghouls