how to prevent opening serial port on linux by foreign application?

Question

In case of "bug" in port driver using locking is not acceptable.

Answer 1

So you need to lock out other processes from opening a device file, but file locking is unacceptable...

Well, you could rename the device file to something non-standard, so everything that tries to open /dev/ttyS0 won't step on your toes.

Answer 2

Assuming you are guarding against mistaken opens by "foreign" binaries: in your driver's open() method, match the name of the opening binary (current->comm). And allow opens only for your binary.

This can, of course be easily circumvented (by renaming a foreign binary.)

Answer 3

Or you can change the properties of the corresponding dev file, only granting access to a custom group you are a member of. If you want the serial port to be accessible to any user, this might not work.