I never really understood: what is Application Binary Interface (ABI)?

Question

I never clearly understood what is an ABI. I'm sorry for such a lengthy question. I just want to clearly understand things. Please don't point me to wiki article, If could understand it, I wouldn't be here posting such a lengthy post.

This is my mindset about different interfaces:

TV remote is an interface between user and TV. It is an existing entity but useless (doesn't provide any functionality) by itself. All the functionality for each of those buttons on the remote is implemented in the Television set.

Interface: It is a "existing entity" layer between the functionality and consumer of that functionality. An, interface by itself is doesn't do anything. It just invokes the functionality lying behind.

Now depending on who the user is there are different type of interfaces.

Command Line Interface(CLI) commands are the existing entities, consumer is the user and functionality lies behind.

functionality: my software functionality which solves some purpose to which we are describing this interface.

existing entities: commands

consumer: user

Graphical User Interface(GUI) window,buttons etc.. are the existing entities, again consumer is the user and functionality lies behind.

functionality: my software functionality which solves some purpose to which we are describing this interface.

existing entities: window,buttons etc..

consumer: user

Application Programming Interface(API) functions or to be more correct, interfaces (in interfaced based programming) are the existing entities, consumer here is another program not a user. and again functionality lies behind this layer.

functionality: my software functionality which solves some purpose to which we are describing this interface.

existing entities: functions, Interfaces(array of functions).

consumer: another program/application.

Application Binary Interface (ABI) Here is my problem starts.

functionality: ???

existing entities: ???

consumer: ???

• I've wrote few softwares in different languages and provided different kind of interfaces (CLI, GUI, API) but I'm not sure, if I ever, provided any ABI.

http://en.wikipedia.org/wiki/Application_binary_interface says:

ABIs cover details such as

• data type, size, and alignment;
• the calling convention, which controls how functions' arguments are passed and return values retrieved;
• the system call numbers and how an application should make system calls to the operating system;

Other ABIs standardize details such as

• the C++ name mangling,[2] .
• exception propagation,[3] and
• calling convention between compilers on the same platform, but do not require cross-platform compatibility.

• Who needs these details? Please don't say, OS. I know assembly programming. I know how linking & loading works. I know what exactly happens inside.

• Where did C++ name mangling come in between? I thought we are talking at the binary level. Where did languages come in between?

anyway, I've downloaded the [PDF] System V Application Binary Interface Edition 4.1 (1997-03-18) to see what exactly it contains. Well, most of it didn't make any sense.

• Why does it contain 2 chapters (4th & 5th) which describe the ELF file format.Infact, these are the only 2 significant chapters that specification. Rest of all the chapters "Processor Specific". Anyway, I thought that it is completely different topic. Please don't say that ELF file format specs are the ABI. It doesn't qualify to be Interface according to the definition.

• I know, since we are talking at such low level it must be very specific. But I'm not sure how is it "Instruction Set Architecture(ISA)" specific?

• Where can I find MS Window's ABI?

So, these are the major queries that are bugging me.

Answer 1

The ABI needs to be consistent between caller and callee to be certain that the call succeeds. Stack use, register use, end-of-routine stack pop. All these are the most important parts of the ABI.

Answer 2

Let me at least answer a part of your question. With an example of how the Linux ABI affects the systemcalls, and why that is usefull.

A systemcall is a way for a userspace program to ask the kernelspace for something. It works by putting the numeric code for the call and the argument in a certain register and triggering an interrupt. Than a switch occurs to kernelspace and the kernel looks up the numeric code and the argument, handles the request, puts the result back into a register and triggers a switch back to userspace. This is needed for example when the application wants to allocate memory or open a file (syscalls "brk" and "open").

Now the syscalls have short names "brk", etc. and corresponding opcodes, these are defined in a system specific header file. As long as these opcodes stay the same you can run the same compiled userland programs with different updated kernels without having to recompile. So you have an interface used by precompiled binarys, hence ABI.

Answer 3

If you know assembly and how things work at the OS-level, you are conforming to a certain ABI. The ABI govern things like how parameters are passed, where return values are placed. For many platforms there is only one ABI to choose from, and in those cases the ABI is just "how things work".

However, the ABI also govern things like how classes/objects are laid out in C++. This is necessary if you want to be able to pass object references across module boundaries or if you want to mix code compiled with different compilers.

Also, if you have an 64-bit OS which can execute 32-bit binaries, you will have different ABIs for 32- and 64-bit code.

In general, any code you link into the same executable must conform to the same ABI. If you want to communicate between code using different ABIs, you must use some form of RPC or serialization protocols.

I think you are trying too hard to squeeze in different types of interfaces into a fixed set of characteristics. For example, an interface doesn't necessarily have to be split into consumers and producers. An interface is just a convention by which two entities interact.

ABIs can be (partially) ISA-agnostic. Some aspects (such as calling conventions) depend on the ISA, while other aspects (such as C++ class layout) do not.

A well defined ABI is very important for people writing compilers. Without a well defined ABI, it would be impossible to generate interoperable code.

EDIT: Some notes to clarify:

• "Binary" in ABI does not exclude the use of strings or text. If you want to link a DLL exporting a C++ class, somewhere in it the methods and type signatures must be encoded. That's where C++ name-mangling comes in.
• The reason why you never provided an ABI is that the vast majority of programmers will never do it. ABIs are provided by the same people designing the platform (i.e. operating system), and very few programmers will ever have the privilege to design a widely-used ABI.

Answer 4

In order to call code in shared libraries, or call code between compilation units, the object file needs to contain labels for the calls. C++ mangles the names of method labels in order to enforce data hiding and allow for overloaded methods. That is why you cannot mix files from different C++ compilers unless they explicitly support the same ABI.

Answer 5

In short and in philosophy, only things of a kind can get along well, and the ABI could be seen as the kind of which software stuff work together.

Answer 6

One easy way to understand "ABI" is to compare it to "API".

You are already familiar with the concept of an API. If you want to use the features of, say, some library or your OS, you will use an API. The API consists of data types/structures, constants, functions, etc that you can use in your code to access the functionality of that external component.

An ABI is very similar. Think of it as the compiled version of an API (or as an API on the machine-language level). When you write source code, you access the library though an API. Once the code is compiled, your application accesses the binary data in the library through the ABI. The ABI defines the structures and methods that your compiled application will use to access the external library (just like the API did), only on a lower level.

ABIs are important when it comes to applications that use external libraries. If a program is built to use a particular library and that library is later updated, you don't want to have to re-compile that application (and from the end-user's standpoint, you may not have the source). If the updated library uses the same ABI, then your program will not need to change. The interface to the library (which is all your program really cares about) is the same even though the internal workings may have changed. Two versions of a library that have the same ABI are sometimes called "binary-compatible" since they have the same low-level interface (you should be able to replace the old version with the new one and not have any major problems).

Sometimes, ABI changes are unavoidable. When this happens, any programs that use that library will not work unless they are re-compiled to use the new version of the library. If the ABI changes but the API does not, then the old and new library versions are sometimes called "source compatible". This implies that while a program compiled for one library version will not work with the other, source code written for one will work for the other if re-compiled.

For this reason, library writers tend to try to keep their ABI stable (to minimize disruption). Keeping an ABI stable means not changing function interfaces (return type and number, types, and order of arguments), definitions of data types or data structures, defined constants, etc. New functions and data types can be added, but existing ones must stay the same. If you expand, say, a 16-bit data structure field into a 32-bit field, then already-compiled code that uses that data structure will not be accessing that field (or any following it) correctly. Accessing data structure members gets converted into memory addresses and offsets during compilation and if the data structure changes, then these offsets will not point to what the code is expecting them to point to and the results are unpredictable at best.

An ABI isn't necessarily something you will explicitly provide unless you are expecting people to interface with your code using assembly. It isn't language-specific either, since (for example) a C application and a Pascal application will use the same ABI after they are compiled.

Edit: Regarding your question about the chapters regarding the ELF file format in the SysV ABI docs: The reason this information is included is because the ELF format defines the interface between operating system and application. When you tell the OS to run a program, it expects the program to be formatted in a certain way and (for example) expects the first section of the binary to be an ELF header containing certain information at specific memory offsets. This is how the application communicates important information about itself to the operating system. If you build a program in a non-ELF binary format (such as a.out or PE), then an OS that expects ELF-formatted applications will not be able to interpret the binary file or run the application. This is one big reason why Windows apps cannot be run directly on a Linux machine (or vice versa) without being either re-compiled or run inside some type of emulation layer that can translate from one binary format to another.

IIRC, Windows currently uses the Portable Executable (or, PE) format. There are links in the "external links" section of that Wikipedia page with more information about the PE format.

Also, regarding your note about C++ name mangling: The ABI can define a "standardized" way for a C++ compiler to do name mangling for the purpose of compatibility. That is, if I create a library and you develop a program that uses the library, you should be able to use a different compiler than I did and not have to worry about the resulting binaries being incompatible due to different name mangling schemes. This is really only of use if you are defining a new binary file format or writing a compiler or linker.

Answer 7

Application Binary Interface (ABI) similar to API, but the function is not accessible to the caller at source code level. only a binary representation is accessible/available.

ABIs may be defined at the processor-architecture level or at the OS level. the ABIs are standards to be followed by the code-generator phase of the compiler. the standard is fixed either by the OS or by the processor.

functionality: define the mechanism/standard to make function calls independent of the implementation language or a specific compiler/linker/toolchain. provide the mechanism which allows JNI, or python-c interface etc

existing entities: functions in machine code form.

consumer: another function (including one in another language, compiled by another compiler, or linked by another linker)

Answer 8

Functionality: A set of contracts which affect the compiler, assembly writers, the linker, and the operating system. The contracts specify how functions are laid out, where parameters are passed, how parameters are passed, how function returns work. These are generally specific to a (processor architecture, operating system) tuple.

Existing entities: parameter layout, function semantics, register allocation. For instance, the ARM architectures has numerous ABIs (APCS, EABI, GNU-EABI, never mind a bunch of historical cases) - using the a mixed ABI will result in your code simply not working when calling across boundaries.

Consumer: The compiler, assembly writers, operating system, CPU specific architecture.

Who needs these details? The compiler, assembly writers, linkers which do code generation (or alignment requirements), operating system (interrupt handling, syscall interface). If you did assembly programming, you were conforming to an ABI!

C++ name mangling is a special case - its a linker and dynamic linker centered issue - if name mangling is not standardized, then dynamic linking will not work. Henceforth, the C++ ABI is called just that, the C++ ABI. It is not a linker level issue, but instead a code generation issue. Once you have a C++ binary, it is not possible to make it compatible with another C++ ABI (name mangling, exception handling) without recompiling from source.

ELF is a file format for the use of a loader and dynamic linker. ELF is a container format for binary code and data, and as such specifies the ABI of a piece of code. I would not consider ELF to be an ABI in the strict sense, as PE executables are not an ABI.

All ABIs are instruction set specific. An ARM ABI will not make sense on an MSP430 or x86_64 processor.

Windows has several ABIs - for instance, fastcall and stdcall are two common use ABIs. The syscall ABI is different again.