How can I know if a character shows on the shell?

Question

I am writing a program with special characters in it. Characters like װאבדג (hebrew).

Using some Ubuntu I had handy here I could get them to work inside the X environment (inside gnome-terminal). In rxvt, I get strange characters instead of what I have in the file; and in bare xterm I get some of them.

The file itself may be just as simple as

letters="⅄ႥႣႬזלבגװאבדגהוזחטענסףמלךלכפץצקႠႣႤႥႬႫႹჄႾႨ"
letters=$(echo $letters | sed -e 's/./\0\n/g')

letters=$(for i in $letters; do echo "$RANDOM$i" done | sort -rn | sed -e 's/[0-9]*//g')
echo $letters

In OS X it just shows "nnnnnnnnnnnnnnnnnnnn".

Within the tty without X.Org started, it just shows a diamond.

In all the terms, I have

LANG=es_ES.UTF-8

Is there any way to know within the script if the chars will be shown correctly (I could implement some fallback if so), or if we can set the terminal to show it.

Answer 1

You have a bug here:

echo $letters | sed -e 's/./\0\n/g'

EDIT (Since you mention you are on OS X I removed the part talking about GNU Sed)

With the version of set built in to OS X, \0\n means "0n" (the character zero and the character n).

You are replacing every character in your input, so you should not be surprised that you are not seing them in the output.

Answer 2

On Mac OS X you can check Terminal.app for UTF-8 readiness:

defaults read com.apple.Terminal StringEncoding  # 4
defaults read com.apple.Terminal DoubleWideChars  # YES

Furthermore, Mac OS X uses FreeBSD sed which does not accept \0.

printf "%s" "$letters" | sed $'s/./&\\\n/g'
printf "%s" "$letters" | gsed $'s/./&\\\n/g'
printf "%s" "$letters" | awk -vFS="" '{for(i=1;i<=NF;i++) print $i}'

# randomize letters
letters=$(echo $letters | sed $'s/./&\\\n/g')
# note the additional ";" after "${RANDOM}${i}"
letters=$(for i in $letters; do echo "${RANDOM}${i}"; done | sort -rn | sed -e 's/[0-9]*//g')
echo $letters