+1  A: 

Okay, this is a bit tricky.

The Easy Way

You could simply type in IRSSI:

/SET term_charset utf-8

The Hard Way

If things doesn't go the easy way, take a look at this.

Install FriBiDi and HebXFonts. Now, the next part is a kind of an academic guess: It might involve some path altering in the Makefile of HebXFonts.

As for me, I did all of the above until it just worked.

Good luck.

konzepz