If you control the other process (have source code), the best plan is to use advisory locks in both processes. This locking is defined in POSIX, and will be portable across operating systems.
In Linux, you can use the utility lsof to see what files are opened by other processes.
This is limited to what you have permissions for - you have to do the check as a privileged user, or you'll only get results for files opened by the same user as the one doing the check.
I only know of the command line utility, not of any system call you can use directly from C code.
In Linux, it's also possible to turn on mandatory locking for a given filesystem (mount -o mand), and set special flags on the file (chmod g-x,g+s). Then when your process attempts to acquire a write lock, it will fail if another process has the file open. This is hardly ever used, but if you completely control the system in question, it may be an option.