What is kernel's KMS(kernel mode setting) API?
Mode setting is usually related to Graphics setup.
A reference article dated April 19, 2008 notes,
kernel mode-setting involves moving the mode-setting code for video adapters from the user-space X server drivers into the Linux kernel. This may seem like an uninteresting topic for end-users, but having the mode-setting done in the kernel allows for a cleaner and richer boot process, improved suspend and resume support, and more reliable VT switching (along with other advantages). Kernel mode-setting isn't yet in the mainline Linux kernel nor is the API for it frozen, but Fedora 9 shipping next month will be the first major distribution carrying this initial support. In this article we're looking more closely at kernel mode-setting with the Intel X.Org driver as well as showing videos of kernel-based mode-setting in action.
Here is a Fedora wiki KernelModesetting page.