Lori Devoti wrote in comments: How about knowing you are writing the right thing for you?
One popular method is to depend on others (writer friends, crit groups, teachers, agents, editors, etc.) to look at your work and tell you which is right for you. There is nothing wrong with this, and it can be helpful if you already have a high success rate with the feedback you get from these folks -- meaning, things they give you the green light on actually sell for you and do well on the market.
That method didn't work for me.
You can also test drive genres and different ideas in short story form. If you're a published author, you can submit them to magazines and anthos and see what sells. I put mine up on my old web site and gathered feedback from readers, who told me what they liked. I took the stories they responded most positively to and made them novel length, which resulted in three book sales.
If that doesn't appeal to you, do some self-analysis. Are you having fun, or are you fighting for every word? Do you run to the keyboard, or come up with excuses to avoid it? Are you faithful to your genre, or are you playing around with others on the side? What do you love? What do you want to do on the page? Is there anything you're dying to try out, and if so, what's stopping you?
Anyone else have ideas, methods or experiences that resulted in finding and writing what was right for you?