The instructions you are following are simply to over-ride the o/s limits that may prevent you from getting a core dump.
The generation of a core dump is a trivial process, you send a signal to the process as follows
kill -ABRT pid_of_process
There are many things however that may prevent this from happening, however you should try this first and see if it produces a core dump in your current directory. If the program is interactive and doesn't trap the quit signnal then you may be able to cause core to dump by sending SIGQUIT to the process, this is usually bound to CTRL-\
The area you are having problems with in the referenced document refers to process that run setuid/setguid if your process is not running in either of those modes then you can safely ignore that step. (You can tell if the process is running setuid/setguid by looking at the file permissions of the program and examining the setuid and setguid bits this can be done by issuing an ls -l command and looking for s in the 4th position (setuid) or 7th position (setgid) (example of setuid below)
-r-sr-xr-x 1 root wheel 57616 28 Oct 03:28 /usr/bin/login
Have you tried to generate a core without using the step that is not working and did it work?
You will need to be able to write in the directory that the process is running in, or the directory defined for core dumps if that is not the current directory. Running as root may solve the permissions issues.