I need the best way to prevent any access to doc files when it is not my web application, for example, I need some files to be hidden from search engines or public users and only private users may reach and download them.
I would like to save the files in the file system and not in the DB in order not to increase the DB usage.
You can store your files in the file system, in a directory that cannot be accessed through an URL (such as a sibling of your root web directory). This will prevent direct access.
Then, write a PHP script which, when queried for a given file, will check whether the user can access that file, and send it with readfile (along with header for the content type and content disposition).
Robots.txt is a way to block/allow certain HTTP user agents. It's not the most secure way though. You could use basic authentication to let your web app login.
When you say "private users" only I'm assuming that means these users are authenticated somehow. You could store these files outside the web directory, and then serve them up via PHP/Perl/Your_Favorite_Programming_language.
A link could lead to a script that checks if that user is authenticated, if so, serve the file up via the script.
In php, after authenticating the user, you could use:
header('Content-disposition: attachment; filename=movie.mpg');
header('Content-type: video/mpeg');
readfile('/full/path/to/your/files/movie.mpg');
Just remember to make that directory outside your webroot, and to set the permissions to the same user that your scripts run as (nobody, generally under apache).