Yesterday I committed a file and then wanted to hg backout, but Mercurial says cannot backout because I have other modified files in the project…
That's a little bit strange… it is not atomic level on each file? Commit 1 file and then backout 1 file?
Second, I can save a copy of those modified files A, B, C, to tmp files, hg revert them, and then backout, and then copy those tmp files back to A, B, C, and isn't that the same as just hg backout that last commit but just more work?
Nope. Mercurial is not a file-revision tool, it's changeset-revision. A changeset includes all the files affected by particular change, not individual files.
Yes, theoretically doing revert on all open files, backout and then editing those files again will achieve the same effect as doing a backout of the individual file. However, in practice the history for the repo does not contain separate entries for each file, so you can't roll back only one of the files.
In general, doing a backout implies doing a merge. When merging, the working copy is used as a scratch space: merge conflicts show up as changes to the files in your working coyp, and you resolve them by editing the files in your working copy. This is the reason why Mercurial insists on you having a clean working copy before you merge or backout: there is really no other good place for Mercurial to store the files when it needs you to resolve conflicts.
Personally, I've never run into this problem. I never have modified files lying around -- I either commit them or I stash them in a MQ patch. If you don't want to use MQ, then a simple
hg diff > tmp
hg revert --all
# working directory is now clean, do your merge/backout/...
hg import --no-commit tmp
is enough to deal with your case. This has of course been wrapped into easier commands by the nice people who write extensions: see the attic extension for an example.