There's no point in trying to find a free display number. As you have guessed, between the time you find a free one and the time Xvfb starts, another X server might have taken the port you thought was free. So, better to just try to launch Xvfb, handle the failure if the port is taken, and then retry at the next port until you succeed or run out of ports to try.
#!/bin/bash
DISPLAY_NUM=0
unset TEST_HAS_RUN
until [ $TEST_HAS_RUN ] || (( $DISPLAY_NUM > 10 ))
do
Xvfb :$DISPLAY_NUM &
jobs
sleep 2 # assumption here is that Xvfb will exit quickly if it can't launch
if jobs | grep Xvfb
then
echo launching test on :$DISPLAY_NUM
xterm -display :$DISPLAY_NUM
TEST_HAS_RUN=1
kill %-
else
let DISPLAY_NUM=$DISPLAY_NUM+1
fi
done