Are you sure you have a good serial connection?
The cable must be a null modem cable - you should verify this using a terminal program (without kernel debugging enabled on the target). Run the program (HyperTerminal or whatever) on both the target and the host machines (remember, like I said before kernel debugging over that port must not be enabled on the target or the port can't be opened).
You can find some links to free terminal programs (I don't think Microsoft provides HyperTerminal anymore since Vista) here: http://www.lvr.com/serport.htm#software
Type some characters on one machine, and make sure they show up on the other side. Do the same on the other machine.
If you can't do this, then there's no serial connection and you'll have to get a null modem cable or adapter. There's no point in trying to get KD to work over that connection until you can do this.
Also, remember that the COM port used by kernel debugging on the target must be a standard 8250 family UART (usually a 16550 or better nowadays, and usually baked into the motherboard chipset). It can't be a USB serial port connector (although that kind will work fine on the host side, since on the host kd.exe is a regular Win32 program).
Edit:
If it's not a serial connection problem, the problem might be that your target doesn't support configuration using boot.ini. Since Vista, boot parameters have been specified using the "Boot Configuration Database" (BCD) which is manipulated with a program like bcdedit.exe. Unfortunately, configuring a system using BCDedit is much more complicated than just editing a simple text file like boot.ini.
You should read the debugger helpfile section "Configuring Software on the Target Computer"; in particular the "Using Boot Parameters" part. there are details there on how bcdedit can be used to enable debugging on Vista and later systems.