Is it allowed to mix different file handling functions in a one system e.g.
I have a large application with a lot of legacy code and it seems that all three methods are used within this code. What are potential risks and side effects ?
Yes, you can mix all of that together. It all boils down to the CreateFile call in any case.
Of course, you can't pass a file pointer to CloseHandle and expect it to work, nor can you expect a handle opened from CreateFile to work with fclose.
Think of it exactly the same way you think of malloc/free vs new/delete in C++. Perfectly okay to use concurrently so long as you don't mix them.
It is perfectly OK to use all of these file methods, as long as they don't need to interact. The minute you need to pass a file opened with one method into a function that assumes a different method, you'll find that they're incompatible.
As a matter of style I would recommend picking one and sticking to it, but if the code came from multiple sources that may not be possible. It would be a big refactoring effort to change the existing code, without much gain.
Your situation isn't that uncommon.
Code that is designed to be portable is usually written using standard file access routines (fopen, open, etc). Code that is OS-specific is commonly written using that OS's native API. Your large application is most likely a combination of these two types of code. You should have no problem mixing the file access styles in the same program as long as you remember to keep them straight (they are not interchangeable).
The biggest risk involved here is probably portability. If you have legacy code that has been around for a while, it probably uses the standard C/C++ file access methods, especially if it pre-dates the Win32 API. Using the Win32 API is acceptable, but you must realize that you are binding your code to the scope and lifetime of that API. You will have to do extra work to port that code to another platform. You will also have to re-work this code if, say, in the future Microsoft obsoletes the Win32 API in favor of something new. The standard C/C++ methods will always be there, constant and unchanging. If you want to help future-proof your code, stick to standard methods and functions as much as possible. At the same time, there are some things that require the Win32 API and can't be done using standard functions.
If you are working with a mix of C-style, C++-style, and Win32-style code, then I would suggest separating (as best as is reasonably possible) your OS-specific code and your portable code into separate modules with well-defined APIs. If you have to re-write your Win32 code in the future, this can make things easier.