Well, you have to use the heap if you want to use objects. Objects are inherently pointers on the stack (or inside other objects) to memory chunks in the heap.
Unlike C++ where you can put objects on the stack or heap, Java does things differently.
Even in C++, it's a good idea to use the heap for objects that must outlive the function they were created in. You probably can avoid it but you may find yourself with a performance problem with all the copy constructors.
As to your edit:
Is there any other kind of memory which we can think of alternative to Heap and Stack?
Sure there is: Static data members, the ones where there's only one per class rather than one per instantiated object must go somewhere. These can't be on the stack since they may disappear when you exit from a function. And they can't belong to an particular object.
These (at least in the Sun/Oracle JVM) go into the Method area.
In addition, you should not think of there being one stack. Every thread gets its own stack on creation.
There's also the runtime constant pool and stacks for native (non-Java) calls.
There's lots of information on the JVM internals here regarding that aspect but keep in mind there may be a distinction between the logical machine and the implementation.