YA bookshelves still groan with dystopian novels, and teens continue to guzzle paranormal fantasies, but the category is pushing the envelope in search of the next crowd-pleasing genre—whether a hybrid of styles or a subversion of tried-and-true formulas.

The mashup is the New Black (you can have your zombies, but only if they are living in, say, fin de siècle Philadelphia); clones better have a romantic interest. And enough of cute, sparkly vampires: these days repulsive and strange are in vogue.

“To my mind, the secret to making a splashy sale right now is that you have to have something that has multiple angles,” says literary agent Barry Goldblatt. “You can still sell a straight paranormal romance, but it’s a lot harder than it used to be.”

Speculative fiction is seeking new directions with more science fiction, more science-fiction/fantasy mixes, and some interesting subgenres. Here we look at three emerging trends.

Meet Me In the Lab

If, as Curtis Brown agent Ginger Clark puts it, “paranormal is in a trench” at the moment, science fiction stories are in ascendance, especially those that also satisfy the voracious teen appetite for romance. Transparent, a March 2013 HarperTeen release by newcomer Natalie Whipple, is set in a world where everyone is born with some kind of special power or mutation. Fiona’s father uses his special ability—mind control—for evil purposes; her signature trait is invisibility. “It’s a version of the future that you could call dystopic,” Clark says, but that’s not how she’d sell it. “This is a story about a girl who has spent her whole life not knowing what she looks like, who finally meets a boy whose special power is that he can see invisible things. The romance is an indispensable part of the story.”

That combination of elements—genetic mutation plus romantic tension—is a hallmark of many of this fall’s science fiction offerings. Call it the Twilight Effect. Eve and Adam (Feiwel and Friends) by Animorphs creators Katherine Applegate and Michael Grant follows a girl’s recovery after her leg is severed in an accident and then reattached with astonishing ease in the biopharmaceutical lab run by her mother. Eve (short for Evening) gets a project to keep her busy while she heals—create the perfect male—and an easy-on-the-eyes orderly as a foil. Jessica Khoury’s debut, Origin (Razorbill), mines a similar vein: a “perfect” girl, Pia, has been created in a secret compound in the Amazon protected by an electric fence. She’s largely content to do the bidding of the scientists, who used advanced genetic engineering to make her invulnerable to illness and disease—until a storm rips a hole in the fence and she steps through it to meet Eio, a (hunky) indigenous boy. Suddenly, Pia is second-guessing the curious circumstances under which she’s been raised.

“In the case of high-concept science fiction, the love story is often what grounds the novel in that universal territory that keeps the story ‘relatable,’ even when the fictional world looks so different from the one we live in,” says Khoury’s agent, Lucy Carson of the Friedrich Agency. “When you’re marketing a book to teenagers, and let’s face it, [it’s] almost entirely to female teenagers, a love story is always at the heart of it.”

And sometimes that heart belongs to a clone. Both The Lost Girl by Sangu Mandanna (HarperCollins/Balzer + Bray) and Beta by Rachel Cohn (Hyperion) introduce chilling ideas about what it means to be human, with plot lines complicated by forbidden love.

In Mandanna’s debut, Eva is an “echo,” created in modern-day England by the “Weavers” to be an exact replica of a girl who lives in India. Should something happen to Amarra, Eva will replace her. It’s the ultimate insurance policy for parents: a human backup. But what happens to Eva if Amarra does not die? Will she be “unstitched?” And what will happen to her relationship with Sean, if she must abandon life in England for life in India?

In Cohn’s Beta, Elysia is an experimental model of a teenage clone, replicated from another girl, who had to die in order for Elysia to exist. She is bought to act as a companion to a wealthy family living on an exclusive island on an Earth recovering from catastrophic natural and man-made disasters. She isn’t supposed to have preferences or feelings, so when she realizes she is drawn to chocolate—and to a boy named Tahir—she worries she is a Defect. Defects are destroyed.

Both books recall Mary E. Pearson’s Jenna Fox series, which began with The Adoration of Jenna Fox (Holt, 2008) and will conclude with the third volume in spring 2013, Fox Forever. Jenna survived a horrific car accident because her father, a biomedical pioneer, had the technological know-how to keep her brain alive while he rebuilt her body. Pearson says, in her experience, teens responded to Jenna’s story (a sequel, The Fox Inheritance, followed the genetically engineered revival of the two friends who were in the same accident as Jenna) because it sparked discussions about two subjects teens spend a lot of time thinking about anyway: who they are and the meaning of life. “When I get letters from teens, they don’t usually mention the bio­engineering or cloning aspect. It always goes back to identity,” Pearson says. “What they’re interested in is, ‘What makes me, me?’ and ‘Why am I here?’ ”

The Undead at the World's Fair

A decade ago, when agent Goldblatt was shopping his wife Libba Bray’s A Great and Terrible Beauty, the first book in the trilogy featuring London boarding school student Gemma Doyle, he was worried. “I thought it was going to be a struggle to sell a book that was historical and fantasy and paranormal,” he recalls. “There’s always been this tendency in publishing to want to put things in a box and say it’s X or it’s Y or it’s Z, but I definitely think things have changed, at least for the moment. Now, if it’s just one thing, I think it’s harder to sell than if you have something that will appeal to more than one kind of reader.”

The success of the Gemma Doyle trilogy, and later of Kristin Cashore’s Grace­ling books, demonstrated there was a substantial readership for books that were more than one thing. Bray’s latest, The Diviners, exemplifies this. Set in Prohibition-era New York City, it’s a substantial read, clocking in at 578 pages, and rich in period detail—the kind of book readers who love historical fiction can lose themselves in. But it also has a strong supernatural strain, with a heroine who has psychic powers, action centered on a museum of the occult, and a villain back from the dead to terrorize the living. Little, Brown announced a 200,000-copy first print run.

Meanwhile, just down the pike in Philadelphia (and a few decades earlier), the undead are also wreaking havoc at the 1876 World’s Fair, in Susan Dennard’s Something Strange and Deadly (HarperCollins). The heroine, Eleanor, ably fends them off with a whack of her parasol. Then there’s Innocent Darkness by Suzanne Lazear (Flux), a steampunk fairy tale set in a boarding school in an alternate California, circa 1900. Or how about mythological creatures in a steampunk-infused, alternate Victorian-era London? That’s the setting of The Unnaturalists by Tiffany Trent (Simon & Schuster).

“People who are real readers love a historical setting,” says agent Erin Murphy, who pitched Robin LaFevers’s Grave Mercy to Houghton Mifflin, the first in a trilogy, as a “medieval historical thriller featuring teenage assassin nuns.”

“Their jaws dropped and their eyes lit up,” Murphy says. “If you take a really rich period of history and combine it with something else, you up the odds that you’re going to strike a chord with somebody.” Especially if the something else is teenage assassin nuns. The second book in the trilogy, Dark Triumph, follows a different novitiate, trained at the same convent, and is scheduled for a spring 2013 release.

The sparkling Edward Cullen was bound to produce some conceptual backlash: a new crop of titles features paranormal beings with very normal abs in stories that are dark, edgy, and hard to categorize.

In Wuftoom by Mary G. Thompson (Clarion), a seventh grader named Evan has been sick for years with a mysterious illness. His limbs are fusing, his skin is covered in a slimy pink membrane, and he’s slowly transforming into the large grub of the title. If that’s not bad enough, a rival species, the Vitflies, has kidnapped his mother, using her as bait to convince him to spy on his Wuftoom brethren. In a starred review, PW wrote, “Dark and unsettling, Thompson’s adventure presents a break from the same-old-same-old by creating something utterly new and weird—sexy YA paranormal this is not.”

Something similar could be said of The Chaos by Nalo Hopkinson (S&S/McElderry). Hopkinson, a well-regarded author of fantasy and science fiction for adults, makes her YA debut with a surreal thriller about Scotch, a Toronto teen of mixed heritage who inexplicably develops sticky black spots on her skin, a precursor to the arrival of the title’s pandemonium, an unleashing of fantastical events—a volcano erupting in Lake Ontario, a new island off the coast of Jamaica made of gumdrops—that separates Scotch from her friends and family, and sends her on a journey during which she must come to terms with her own changing nature.

Then there’s Above by Leah Bobet (Scholastic), another YA debut, this one by a writer well known for her poetry and short stories. In an underground community, a collection of societal outcasts have created a haven for themselves they’ve named Safe. The residents are broken or sick, self-styled freaks. The narrator, Matthew, was born to a father who had lion’s feet and a mother with gills. Another denizen, Ariel, transforms into a bee when stressed. Their refuge is threatened when the only person they’ve ever forcibly exiled returns, bent on revenge.

All of these titles are full of fantastical characters in settings that mix science fiction with reality. If you shelve them with the paranormal books, will they find their audience?

“Some readers are so focused and locked in about what they want to read that getting them to look outside the category they love is hard,” says Gold­blatt, who has encountered frustrations with books marketed as “paranormal” that didn’t really fit the label. “Publishers are using a little sleight of hand to get readers to take notice, and I’m fine with that when it helps a book find an audience. But it can backfire if you try to blatantly sell something as something it’s not.”

So Railsea by China Miéville (Del Rey) is not precisely a retelling of Moby-Dick, set on a network of crisscrossing railroad tracks and swapping out whales for giant moles, except that it sort of is. The main character, Sham Yes ap Soorap, is aboard a moletrain, witnessing his first hunt with a captain who’s been chasing the ivory-colored mole that took her arm in a battle years before. The world of Railsea, with its scavengers, pirates, and menacing vermin—fanged meerkats, predatory chipmunk packs—feels both futuristic and a bit like the Old West. It’s “a grim and bloody invention,” says bookseller Chris Hsiang of Books Inc. in San Francisco, one that will suit “teens who cut their teeth on Thomas the Tank Engine, then Lemony Snicket and Scott Westerfeld’s Leviathan.”

As the variety of these subgenres shows, YA continues to mutate, looking for new ways to reach its robust audience of readers. The pendulum swing away from vampires and werewolves may even eventually lead back to more traditional genres, Clark says. “I’m always hoping straight historical will get hot,” she said. “Believe me, everybody’s looking for the YA Downton Abbey.”