Treur got it right, but I still want to clarify a few things and provide you with some sources for reference material. As Treur said, NEVER ever trust user input data, that includes all headers sent by the browser.
What you are describing, is a typical Cross-Site Request Forgery attack. Checking the referrer header is not a valid protection against CSRF attacks, since according to the RFC2616 (Hyper Text Transfer Protocol 1.1), the referer header is optional and thus may be omitted by the browser at any time. If you are using SSL, then the referer header is always omitted by browsers. Secondly, it is a user defined value, and thus should not be trusted.
The recommended protection against CSRF attacks is to use the synchronized token pattern. This means that you should create a secret token which is embedded as a hidden field in your form. When the form is posted, you verify that the secret token is present and that it is valid. There are multiple strategies for creating security tokens. I'll describe one way for creating the tokens:
For each action in your application, create a unique action name for them. For example, "delete_user", "add_user" or "save_user_profile". Let's say that the form you described has the action name "foobar". Concatenate the action name with the user's session id and a secret value.
$stringValue = "foobar" . "secret value" . session_id();
To create the security token, create a hash of the concatenated string, you can use sha1 to create the hash. To decrease the risk of brute force attacks, use a larger key in the hash, for example, sha 512.
$secretToken = hash("sha5125", $stringValue);
Set this token in your form's hidden field. When the form is submitted, recreate the token and verify that it matches the one submitted in the form. This token is valid for one user session. One may argue, that there is a window of opportunity where an attacker can reuse the token as it is not regenerated at every request. However, with proper session management strategies, this shouldn't really be a concern.
Like I said, proper session management is necessary. This means that you shouldn't keep the sessions alive to long. Especially session fixation vulnerabilities will undo any CSRF protection measures, as the attacker is then in control of the user session and hence can "predict" the secret tokens.
Here are a couple of links that I recommend you read through: