Even long-time, well respected, professional testers will tell you: It is an art more than a science.
My trick to designing new test cases starts with the various types of tests you mention, and it must include all those to be thorough, but I try to find a list of all the ways I can interact with the code/product.
For the vending machine example, there are tons of parts, inside and out.
Simple testing, as the product is designed to work, gives plenty of cases
- Does it give the correct change
- How fast can it process the request
- What if an item is out of stock
- What if it is overfilled
- What if the change drawer is full
- What if the items are too big, or badly racked
- What if the user puts in too little money
- What if it is out of change
Then there are the interesting cases, which normal users wouldn't think about.
- What if you try to tip it over
- Give it a fake coin
- Steal from it
- Put a coin in with a string
- Give it funny amounts of change
- Give it half-ripped bills
- Pry it open with a crow-bar
- Feed it bad power/brownout
- Turn it off in the middle of various operations
The way to think like a tester is figure out every possible way you can attack it, from all the "funny cases" in usual scenarios, to all the methods that are completely outside of how it should be used. Any point of input, including ones you might think the developers/owners have control over, are fair game.
You can also use many automated test tools, such as pairwise test selection, model-based test toolkits, or for software, various stress/load and security tools.