Answer nicked out of "MetaWINDOW FAQ - OMF vs COFF Object File Formats.htm"
Since the dawn of PC civilization up until about the time Microsoft Win32 programming tools came along, almost all PC compilers produced object files using the Intel Object Module Format (OMF) standard. Later, Intel introduced 386 processors and 32-bit protected-mode at which point they also expanded the OMF specification for 32-bits, leading to "OMF-386" which became the standard for most PC protected-mode environments. Around this same time, the original Windows NT development team was also designing code, not only for Intel processors, but also to support processors from other vendors. The Microsoft NT team selected a more portable object module format known as Common Object File Format (COFF) derived from the official object-code format for UNIX System V. COFF object modules later became the defacto standard for all Microsoft Win32 development tools, and gained an advantage in being much closer in format to Portable Executable files - the native executable format for Win32 (a COFF-format linker has much less work to create a 32-bit EXE or DLL from a COFF file than from an OMF format file).
Just as there are OMF- and COFF-format object files (.obj's), there are also OMF and COFF format library files (.lib's). The libraries, fortunately, are basically just a collection of the object files, along with some header information that lets the linker determine which object files to use from the library. To make things difficult however, both OMF and COFF use the same file name extensions, .obj and .lib, to reference the two different types of object and library file formats (because of this you can't just look at the file name extension to tell if the object module or library file is OMF or COFF).
The problem with mixing object files and library files from different compiler vendors is that some vendors support COFF, other vendors use OMF, and a few can handle both. Borland, for example, still uses OMF object files and libraries, while Microsoft's 32-bit compilers produce COFF format files. Watcom C/C++ v11.0 seems to prefer COFF when compiling and linking Windows applications, but generates OMF object files for use with their DOS4GW 32-bit protected-mode DOS-extender. Along with this, Microsoft MASM 6.13 produces OMF files by default, but with the /coff switch can emit COFF object files instead.
When it comes time to link files with different formats, different linkers do differnt things. For example, the Microsoft Visual C/C++ linker is designed for COFF format object files and libraries, but will try to convert OMF object files into COFF files if necessary. This works in some cases, but unfortunately Microsoft LINK does not support all OMF record types, so in many situations the linker may still fail when given OMF format object files. Also while Microsoft LINK attempts some support for OMF object files, it will refuse to process any OMF format libraries. Other linkers, such as Borland's TLINK, are designed for OMF object files and will similarly refuse to work with COFF-format object or library files. Some DOS extender and embedded system vendors, such as Phar Lap, provide their own linkers which support both OMF and COFF, giving you a choice.
The bottom line is that mixing OMF and COFF object and library file types can be a mess (plus the cryptic error messages from the linkers don't help). Unless your linker specifically supports it, you should stick with recommended object and library format for your compiler/linker/platform, and avoid mixing OMF and COFF files.