Your original pdf is smaller than the thumbnail you're creating. Imagemagick scales the image to match the requested dimensions. Use the following parameters:
convert -scale '800x800+0+0>' -colorspace rgb -strip in.pdf[0] out.png
The trailing >
in the scale parameter tells Imagemagick to not scale the image to larger than the original.
Edit: Imagemagick uses Ghostscript to render PDF files. You can use Ghostscript directly if you need to set some parameters, like resolution to get a better image. Default resolution is 72 DPI which means that an A4 paper has size of 595 x 841 pixels. With 150 DPI you'll get twice the number of pixels. E.g.
gs -q -dBATCH -dNOPAUSE -sDEVICE=pngalpha -dMAxBitmap=500000000 -dAlignToPixles=0 -dGridFitTT=0 -r150x150 -sOutputFile=out.png in.pdf
The above command is almost identical to the one Imagemagick uses. Note the -r parameter which sets 150 DPI resolution. You can use ImageMagick to scale the resulting image to smaller size.
Using a higher resolution will reduce fuzziness when you resize the image.