There are multiple sections in the manpages. Two of them are:
2 Unix and C system calls 3 C Library routines for C programs
For example there is getmntinfo(3) and getfsstat(2), both look like they do the same thing. When should one use which and what is the difference?
System calls are operating system functions, like on UNIX, the malloc() function is built on top of the sbrk() system call (for resizing process memory space).
Libraries are just application code that's not part of the operating system and will often be available on more than one OS. They're basically the same as function calls within your own program.
The line can be a little blurry but just view system calls as kernel-level functionality.
System calls are the interface between user-level code and the kernel. C Library routines are library calls like any other, they just happen to be really commonly provided (pretty much universally). A lot of standard library routines are wrappers (thin or otherwise) around system calls, which does tend to blur the line a bit.
As to which one to use, as a general rule, use the one that best suits your needs.
System calls annoy the kernel, library calls frequently do not (unless they use syscalls).
Linux kernel developers despise additional ioctl() interfaces, sysconf() calls and more, they would rather that most user space programs just read from /proc (which 80% of the time gets you what you want). That's why Linus introduced USER_HZ (units exported in procfs in units of 100'ths of a second intervals).
In short, if you can avoid pestering the kernel to get some value, do so. If your talking to the kernel, you should be telling it to do something, not asking it for info.
Yes, we need ioctl, we need syscalls, but they need to be limited. Bugging the kernel to allocate memory is fine, bugging the kernel to fill a sysinfo structure is fine. Bugging the kernel to find out where you put your house keys, bad.
99.9 % of the time that a process is put into perennial disk sleep (D) , its because its waiting for the answer to an impossible question. Usually, that question is made impossible because you forgot to handle the previous 10 syscalls.
Microkernels are a little better at telling you to go away, but not always. :)
The calls described in section 2 of the manual are all relatively thin wrappers around actual calls to system services that trap to the kernel. The C standard library routines described in section 3 of the manual are client-side library functions that may or may not actually use system calls.
This posting has a description of system calls and trapping to the kernel (in a slightly different context) and explains the underlying mechanism behind system calls with some references.