I recommend git.
Either way you're going to want the canonical repository on the USB key. In git you might do this:
Make a "bare" repo on the USB key:
$ mkdir /path/to/usbkey/myapp.git
$ cd /path/to/usbkey/myapp.git/
$ git init --bare
Initialized empty Git repository in /path/to/usbkey/myapp.git/
Bare repository directories are usually named "something.git" - you can name them whatever you want, but the ".git" convention is very widely used.
Now you can clone the repo:
$ cd /my/source/dir/
$ git clone /path/to/usbkey/myapp.git
Initialized empty Git repository in /my/source/dir/myapp/.git/
warning: You appear to have cloned an empty repository.
It will warn you that the repo is empty. Let's put something in it:
$ cd myapp
$ echo "some stuff." > README
$ git add README
$ git commit -m 'added a README'
[master (root-commit) 155b8ea] added a README
1 files changed, 1 insertions(+), 0 deletions(-)
create mode 100644 README
And then push it to the USB key:
$ git push origin master
Counting objects: 3, done.
Writing objects: 100% (3/3), 231 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
To /path/to/usbkey/myapp.git
* [new branch] master -> master
When you get to your other computer, just clone the repo from your USB key again. You'll have to make sure you remember to push your changes, but you know you'll always have a backup because you'll have three full copies of the repo whenever you're synced up.
An alternate way to do it with git is to only have one repo - the one on the USB key. You wouldn't ever have to remember to push to it, but your code would only be on the key unless you used some other explicit backup system. That would be bad.
If you were to use SVN on the USB key you would still have to remember to commit and pull your changes in the same way has having a bare git repo, but you wouldn't get the free automatic backups that doing so with git gives you. Also you would miss out on all the other niceties of git, but that's a whole other discussion. See Why Git is Better Than X.