I want to execute an untrusted .lua file in its own environment by calling lua_setfenv() so that it cannot affect any of my code.
The documentation for that function though only explains how to call a function, not how to execute a file.
Currently to run the file I use:
int error = luaL_loadfile(mState, path.c_str()) || lua_pcall(mState, 0, 0, 0);
Do I have to call the "dofile" lua function from the C API with lua_setfenv, or is there a more elegant way to do it?
luaL_loadfile() will load the chunk, then call lua_setfenv() to set the environment table, then call lua_pcall() to execute the chunk. See recent answer given by Judge Maygarden at http://stackoverflow.com/questions/3432231/calling-lua-functions-from-luas-using-handles/3441124#3441124
See the discussion at the Lua User's Wiki of sandboxing, and the more general topic of script security. There are a number of subtle and not so subtle issues with this kind of thing. It can be done, but protecting against code such as for i=1,1e39 do end requires more than just restricting what functions are available to a sandbox.
The general technique is to create a function environment for the sandbox that has a whitelist of permitted functions in it. In some cases, that list might even be empty, but letting the user have access to pairs(), for example, is almost certainly harmless. The sandbox page has a list of the system functions broken down by their safety as a handy reference for constructing such a whitelist.
You then use lua_setfenv() to apply the function environment to the user's script which you loaded (but haven't yet executed) with lua_loadfile() or lua_loadstring() as appropriate. With the environment attached, you could execute it with lua_pcall() and friends. Before execution, some people have actually scanned the loaded bytecode for operations that they don't want to permit. That can be used to absolutely forbid loops or writing to global variables.
One other note is that the load functions will generally load either precompiled bytecode or Lua text. It turns out to be a lot safer if you never permit precompiled bytecode, as a number of ways to make the VM misbehave have been identified that all depend on handcrafting invalid bytecode. Since bytecode files begin with a well-defined byte sequence that is not plain ASCII text, all you need to do is read the script into a string buffer, test for the absense of the marker, and only pass it to lua_loadstring() if it is not bytecode.
There has been a fair amount of discussion at the Lua-L mailing list over the years of this kind of thing, so searching there is also likely to be helpful.