How do you debug a firewall issue with your application at a customer site? Currently we have an issue where Apache and Tomcat (part of the application) throw errors that sound like the firewall is blocking our attempt to accept or bind to ports on the machine.
The problem for us is we can't see what the firewall is doing, because it is managed at a corporate level. Is there even a way to figure out if we are getting blocked or if it is an application specific issue. This is a java application.
You'll probably get some grief for asking a question here that belongs on Superuser or Serverfault.
Simple answer: Log in to any server that's trying to communicate with another, and simulate the requests using telnet:
telnet OtherServer PortNumber
If you get "Connection refused" then the other service is not there or the firewall is making it invisible. If nothing happens, you're probably hitting the firewall. If you get a connection (never mind if anything interesting happens after that), you are getting through.
Edit: Yes, I know this is quite simplistic. It's possible for a cleverly sadistic or very incompetent firewall setup to let a request come in to your server and yet make it unable to answer that request. But I've never encountered such a scenario. Much more common is the simple go/no go case.
If you want more qualified advice, perhaps you can copy us some error messages and/or stack traces from your server.
If the firewall is managed at the "corporate level" it doesn't sound like it's a host-based firewall (host-based being "on the computer you're running on"). If it's not a host-based firewall there's no way it's blocking your attempts to bind, and you can look elsewhere.
If it is host-based... well that's an aggresive host-based firewall then. It might be more likely you don't have Administrator or root privileges to bind to the port.
If you can bind to the port, but can't communicate, you need to start doing standard firewall testing - "Can I hit a static html page from the internet? From the intranet? From localhost?"
This is just my experience, but the tell-tale sign of a firewall problem over HTTP(S) is that the request hangs for a while and then time out with a Connection Timed Out-flavored exception. Depending on the firewall, the connection could also be refused right away. So this is what we have had to resort to for debugging firewall issues. The resolution is always to go to the network admin (we have developed a good relationship with that team) and ask them to let a particular IP address in.
If you're having trouble binding to a port on the Apache/Tomcat server from your Apache or Tomcat running on that very server, then it's more likely one of two other problems:
lsof -i :port and netstat -ant | grep port.
The port number is below 1025 and the process running Apache/Tomcat is not root.
traceroute (tracert for Windows) is a good thing for seeing where things go.
a UPnP tool is also nice to see firewalls on the way.