Most chess search programs can only search to depth 9 or 10. This is not enough to overflow the stack at all.
You probably have an error in your program. In any case, you do need to have a depth limit on your search as you will not be able to do full-width search (exploring all possibilities across all moves up to a given depth) without a depth limit, since a chess game is potentially of unlimited depth (except for the repeated positions or limit on moves rule).
After you search to about depth 9, you have to use a static board evaluation function to evaluate and score the position. You then use the mini-max algorithm to prune branches of the search tree. It's still an exponential search though.
There are also techniques such as quiescent search, where you continue searching in positions where the position is not stable (i.e. if a piece can be taken or the king is in check).