Typically, most firewalls for Linux are a compilation of shell scripts that organize strings to be sent to the iptables
, or sometimes ebtables
command depending on if bridging is in play. Additionally, they make talk to the tc
utility if traffic shaping is in use.
Shorewall is an example of such a firewall, but it does a lot more. It lets you set up various types of NATs, security around VPNs and more.
It looks like you want to do this completely in C. I would recommend obtaining the source to iptables and becoming intimately familiar with the ioctl hooks provided by its kernel counterpart. At that point, you'll be able to begin the process of writing your own wrappers that accomplish what you need to do.
NB: Root needs to be the user running your application, make sure you design for (or around) that.