For software to meet the definition of "open source" (by almost anyone's definition of "open source"), it can't discriminate against commercial use. (See, for example, item 6 of the Open Source Definition.)
If you want to make the source code available and prohibit commercial uses, then you ought to be able to find an appropriate (non-open source) license. The Creative Commons have a few license variants that prohibit commercial use; although they discourage using their licenses for software, there's no reason you can't.
In practice, licensing your software under the GPL (or the AGPL, for web apps) will generally discourage (but not prohibit) commercial use, since commercial software is often unwilling or unable to comply with the GPL's requirements.
Edit in response to the Creative Commons comments: From what I can tell from sites like this, this, and this, CC is discouraged for the following reasons:
- It's not GPL compatible, which makes using it in open source projects difficult.
- License proliferation is a problem in the open source world; using CC for software would worsen this.
- The CC licenses aren't written for software, so applying them to software is a bit murky.
Edit in response to your clarification: I don't think that trying to enforce an anti-commercial clause will work. There are various techniques you can try - restrictions on who can download the code, "phone home" logic embedded within the code, obfuscation, etc. - but all of this can be circumvented. (E.g., download the code once and then post it to a public site, rip out the phone home logic, reverse engineer the obfuscation, etc.)
More fundamentally, I assume that you're making your source code available so that people can use it. This is already a bit of an uphill battle - you have to convince people to use your code out of all of the hundreds of millions of lines of source code available, and to use your code instead of just writing something themselves. Any anti-commercial logic that you add - such as obfuscation, or "phone home" logic, or restrictions on who can download the code - will make the code that much less valuable to use, and I suspect that for most code it wouldn't take much for people to decide it's just not worth it.
Given the problems with enforcing a no-commercial use clause, your best option might be to choose an open source license like the GPL or AGPL that permits commercial use but makes it an unattractive option for many companies. This way, you gain the benefits of being a part of the open source community, including the potential help of folks like the SFLC and gpl-violations.org in finding and dealing with anyone who does violate your license.