I would not recommend doing this but in theory you could run the exe using a shell command within an expression shape:
System.Diagnostics.Process.Start(@"C:\yourPath\yourExecutable.exe")
The System.Diagnostics
namespace is available in BizTalk 2006, I don't think it is available in BizTalk 2004 (BizTalk 2004 had a very restricted subset of the System namespace available).
I'm not sure about getting return values back but you should certainly be able to provide parameters.
Some references on C# shell commands can be found here and here.
I personally think there are three better options available to you:
Don't use BizTalk.
As Campbell suggests, use a windows service instead.
Only use BizTalk for something like this if you want to leverage an already existing BizTalk framework (logging, reporting etc...) or if you have other tasks in a workflow that BizTalk is going to perform. (There are arguments for putting everything into one platform - if you use BizTalk for one thing, use if for everything, but that is a different conversation).
Refactor the logic of your shredder into a C# class library that both your Console application and BizTalk can call.
Calling a class library from BizTalk is much easier to do cleanly and robustly that calling an executable would be.
Simply reference your signed and GACed assembly from an orchestration (create it as an orchestration variable) and you can then call it directly from an expression shape.
here is an article on this that covers the basics. It doesn't go into a lot of the ugly detail or offer discussion on best practices. Professional BizTalk Server 2006 is a good book for that.
As Campbell said, most of this can probably be done with pure BizTalk functionality.
I think perhaps a mix of options 2 and 3 would be best for what you want. Put the binary shredding logic you already have into a C# class library and call this from within a BizTalk orchestration that takes care of your file monitoring, error notification, tracking and integration with other processes.