PowerShell is an interactive shell like KornShell, Bash and er CMD.exe. And like those shells it supports a scripting language (KSH, Bash, Batch). However PowerShell is built on top of .NET and exposes .NET types and allows you to create and manipulate many .NET types. So you can use PowerShell to build scripts that can do what a typical .NET console app could do.
One factor to consider when writing little utilty console applications is how much effort you spend writing parsing & usage code versus the code required to achieve the fundamental purpose of the exe. I have switched to writing a lot of utilities as PowerShell scripts because PowerShell provides a parameter parsing engine with lots of nice features: named/positional parameters, required/optional parameters, default values for parameters, partial specification of parameter names, etc.
PowerShell 2.0 adds even more features in this area (validation attributes, etc) with advanced functions. You can easily write "man pages" as comments for your scripts or advanced functions. Whereas I used to spend 50-80% of my time messing with flaky, non-standard (is it - or / or both?) parameter parsing code in a C# console app, I let PowerShell handle that for me. However I do believe Jacob is correct in saying that complex tasks requiring lower-level .NET code would be easier to get right (static compile time checks) and debug in C#/VB/Visual Studio.
I would love to see the PowerShell parameter parsing functionality exposed via a set of .NET types in the BCL such that you could write a console app and get PowerShell's parsing functionality. A long time ago I used an open source component called Genghis but I think it has been abadoned. At one point during the .NET 4.0 betas, a command line parser appeared in the framework but was removed before RTM. I don't think this command-line parser had any connection to PowerShell - and it should have IMO. So probably a good thing it was pulled.