ZSH: why error using array: "illegal character: ^M"?

Question

$ values=( 300 400 )
$ echo "scale=20; $values[1]-$values[2]" | bc
(standard_in) 1: illegal character: ^M         // Why does it not print -100?
$ echo $values                                 // no ^M sign found!
300 400

[Update]

1. Why is 5E 4d 0a, ie ^M, 13th char in ASCII?
2. Why is the ending sign 0a shown as .? . is 2E in DEC. Hex number 5E is 94 in DEC, 4d is 77 in DEC.

ASCII and ^M

$ echo "^M" | hexdump -C
00000000  5e 4d 0a                                          |^M.|
00000003
$ ascii -x              
    0 NUL    10 DLE    20      30 0    40 @    50 P    60 `    70 p 
    1 SOH    11 DC1    21 !    31 1    41 A    51 Q    61 a    71 q 
    2 STX    12 DC2    22 "    32 2    42 B    52 R    62 b    72 r 
    3 ETX    13 DC3    23 #    33 3    43 C    53 S    63 c    73 s 
    4 EOT    14 DC4    24 $    34 4    44 D    54 T    64 d    74 t 
    5 ENQ    15 NAK    25 %    35 5    45 E    55 U    65 e    75 u 
    6 ACK    16 SYN    26 &    36 6    46 F    56 V    66 f    76 v 
    7 BEL    17 ETB    27 '    37 7    47 G    57 W    67 g    77 w 
    8 BS     18 CAN    28 (    38 8    48 H    58 X    68 h    78 x 
    9 HT     19 EM     29 )    39 9    49 I    59 Y    69 i    79 y 
    A LF     1A SUB    2A *    3A :    4A J    5A Z    6A j    7A z 
    B VT     1B ESC    2B +    3B ;    4B K    5B [    6B k    7B { 
    C FF     1C FS     2C ,    3C <    4C L    5C \    6C l    7C | 
    D CR     1D GS     2D -    3D =    4D M    5D ]    6D m    7D } 
    E SO     1E RS     2E .    3E >    4E N    5E ^    6E n    7E ~ 
    F SI     1F US     2F /    3F ?    4F O    5F _    6F o    7F DEL 

Answer 1

Unix and Windows have different line ending formats. In the Unix world, lines end with the linefeed character (LF, ascii char 10). Windows ends lines with a carriage return (CR, ascii char 13) followed by a linefeed.

Files with Windows line endings must be converted to Unix format before they can work with Unix tools. Otherwise programs like bc see the CR characters as junk and complain, as in your case.

To convert files to Unix format, you can use dos2unix(1) if you have it installed, or alternately pass it through sed 's/^M//g' (but don't type a literal ^M - press Ctrl+V, followed by Ctrl+M).

So why ^M? Well, the carriage return is a nonprintable character. It has no printable representation. For convenience, your terminal will display it as ^M*. So why didn't it appear when you did echo $values? Unfortunately, the command line argument processing strips it out, so you don't see it.

Also for your convenience, your terminal allows you to type nonprintable characters via Ctrl+V and Ctrl + some letter. Ctrl+V and Ctrl+M will produce a ^M character, but move your cursor left and right and you'll see it skips over the whole thing as a single character - not the same as typing ^ followed by M. While you see ^M, command line programs see only the raw data, and will see an actual carriage return character.

Why is 5E 4d 0a, ie ^M, 13th char in ASCII?

You ran hexdump on the output of echo "^M", which produces three characters: a ^, a M, and a linefeed character (LF). See above, that's not the same as a carriage return!

Why is the ending sign 0a shown as .? . is 2E in DEC. Hex number 5E is 94 in DEC, 4d is 77 in DEC.

Hexdump displays all non-printable characters as . characters, including carriage return and line feed characters.

---

*Why M in particular? The convention is to add 64 to the ASCII code. A carriage return is ASCII code 13 (0x0D). Add 64 and you get 77 (0x4D) which is an uppercase M. See this page for a full listing.

Answer 2

^M in one character - the Carriage Return is dos/windows, or ASCII 13 character. It seems that somehow you compromised your input, maybe by pasting something from a windows file or something like that...

Answer 3

You might have better luck if you run your file through dos2unix, if that program is available on your system.

Among other things, dos2unix will convert the Windows \r\n line endings to Unix-style \n line endings.