If you are using an HTTPListener, you have to use httpconfig.exe or netsh to configure your HTTPListener, because:
A. AFAIK There is no API for doing this from C#, only the command line tools
B. By default only SYSTEM or the local Administrators group can listen to http prefixes
So... if your app is running under the user's account, you need to explicitly grant access (using httpcfg.exe or netsh) because your HTTPHandler uses the HTTP.sys to share ports with the running webserver (so you can host your own web app on port 80 while IIS is running on port 80 as well).
A better solution might be to use an actual embedded web server to listen on the port you need. This won't work if another application (e.g. IIS) is already using the port, but since this DOESN'T use the HTTP.sys for port-sharing, there are no security restrictions about what user account is opening /listening on the port. Because you want the user to specify their own port, it seems acceptable to run on another port besides port 80.
My company makes a commercial product called the Neokernel Web Server (http://www.neokernel.com) that runs ASP.NET and lets you programmatically start/stop the server and set configurations; you can embed and distribute this assembly with the application.
There is also the Cassini source code which you can modify and embed, it's free but has a couple of drawbacks like no logging or SSL support.