I have been playing with LLVM on and off for many months now. I wrote two OCaml Journal articles covering the use of LLVM from the OCaml programming language. That is particularly interesting because the OCaml language is ideal for writing compilers and has a wealth of powerful and mature tools and libraries for parsing and so on.
Overall, my experience has been extremely positive. LLVM does what it says on the tin and is very easy to use. The performance of the generated code is superb. One of the programs I wrote was a simple little Brainf*ck compiler that generates some of the fastest executables of any compiler I tested (including GCC).
I have only two gripes with LLVM. Firstly, it uses abort() whenever anything goes wrong instead of raising an exception. This was a deliberate design decision by its authors who are striving to remove all uses of exceptions from LLVM but it makes it impossible to get backtraces from OCaml when trying to debug a compiler that uses LLVM: your program just dies with a textual explanation from LLVM but no clue as to where the error occurred in your source. Secondly, LLVM's compiled library is monstrously big (20Mb). I assume this is due to the bloat incurred by C++ but it makes compilation painfully slow.
EDIT: My work on LLVM culminated in the creation of a high-performance high-level garbage-collected virtual machine. Free download here and check out the corresponding benchmarks (wow!). @Alex: I'll get that BF compiler up for you somewhere ASAP.