How would I send a command to a Mac (which has network sharing turned on) over the Wi-Fi network that both it and the iphone is connected to? In my case I want it to open an application.
By network sharing, I assume you are only sharing files. To launch an app, you'd either need to have Remote Login turned on (which would allow Richard's SSH suggestion) or have an application running on the Mac which is listening on a specific port. Your iPhone app would then communicate over this port. Some iPhone apps have a Mac app counterpart (cf. Reach My Files, Here File, File) for the purpose of communicating with the Mac.
How much work are you expecting the user to do when they install your app? Will you walk them through how to turn on Remote Login option in System Preferences under Sharing (in which case, the iPhone app will need to have the user credentials of a user allowed to login remotely), or have them just install an app? The latter is much more user-friendly, but more work on your part.
To actually launch the app, you can use the open
command. Type man open
for more details, but basically, to launch, for example, iTunes: open -a iTunes.app
.