Foreign Characters Appearing In Git-Managed Files

Question

I am using git 1.7.2.3 via cygwin on Windows 7 and seeing strange artifacts appearing in some of my source files when switching branches. git status reports everything as unchanged yet they crazy characters are present. I've confirmed on GitHub that the files are as they should be in the repo.

My Copy:

਍        ⼀⼀⼀ 㰀猀甀洀洀愀爀礀㸀ഀഀ
        /// Set up method.
਍        ⼀⼀⼀ 㰀⼀猀甀洀洀愀爀礀㸀ഀഀ
        [SetUp]
਍        瀀甀戀氀椀挀 漀瘀攀爀爀椀搀攀 瘀漀椀搀 匀攀琀甀瀀⠀⤀ഀഀ
        {
਍            琀栀椀猀⸀匀挀漀瀀攀 㴀 渀攀眀 吀爀愀渀猀愀挀琀椀漀渀匀挀漀瀀攀⠀⤀㬀ഀഀ
਍            琀栀椀猀⸀琀攀猀琀䤀琀攀洀 㴀 渀攀眀 嘀椀攀眀䐀漀挀甀洀攀渀琀䠀椀猀琀漀爀礀⠀ ഀഀ
                625016, 
਍                㔀㜀㤀㤀㘀Ⰰ ഀഀ
                'T', 
਍                ㌀㐀㠀㌀㔀㈀㤀Ⰰ ഀഀ
                DateTime.Parse("2003-01-08 09:57:04.957"),
਍                ㌀Ⰰ ഀഀ
                "Invoice (PG-PS) - SUPP(11/16/2008)", 
਍                ∀䘀䤀一䄀一䌀䔀∀Ⰰ ഀഀ
                DateTime.Parse("2008-04-11 11:15:07.770"), 
਍                䀀∀尀尀䐀伀匀䬀尀䌀䜀䐀伀䌀匀尀㌀㜀㐀㤀㄀㐀尀㐀㘀　㐀㘀尀戀椀氀猀氀椀瀀开㄀　㠀㄀㘀㐀㠀⸀搀漀挀∀⤀㬀ഀഀ
        }

Repo Copy:

        /// <summary>
        /// Set up method.
        /// </summary>
        [SetUp]
        public override void Setup()
        {
            this.Scope = new TransactionScope();

            this.testItem = new ViewDocumentHistory( 
                625016, 
                57996, 
                'T', 
                3483529, 
                DateTime.Parse("2003-01-08 09:57:04.957"),
                3, 
                "Invoice (PG-PS) - SUPP(11/16/2008)", 
                "FINANCE", 
                DateTime.Parse("2008-04-11 11:15:07.770"), 
                @"\\DOSK\CGDOCS\374914\46046\bilslip_1081648.doc");
        }

I'm also using a .gitattributes file to ensure line endings are correct since we are developing on Windows.

*.cs       eol=crlf text
*.csproj   eol=crlf text
*.sln      eol=crlf text
*.xml      eol=crlf text

The text is an addition by me to attempt to fix the problem as git diff was interpreting the file as binary when I modified it. Didn't have any effect.

This also occurs on fresh checkouts in 1.7.2.3 but not in 1.6.5.1 (mysysgit) as far as I can tell. The caveat is that 1.6 doesn't support .gitattributes which I need for working on Windows. This seems to be a fairly new bug and I haven't changed any configuration.

Does anyone have any idea what could be causing this?

---

edit:

hexdump -C ViewDocumentHistoryTests.cs | sed -n "130,212p"

000008d0  00 20 00 20 00 2f 00 2f  00 2f 00 20 00 3c 00 73  |. . ./././. .<.s|
000008e0  00 75 00 6d 00 6d 00 61  00 72 00 79 00 3e 00 0d  |.u.m.m.a.r.y.>..|
000008f0  00 0d 0a 00 20 00 20 00  20 00 20 00 20 00 20 00  |.... . . . . . .|
00000900  20 00 20 00 2f 00 2f 00  2f 00 20 00 53 00 65 00  | . ./././. .S.e.|
00000910  74 00 20 00 75 00 70 00  20 00 6d 00 65 00 74 00  |t. .u.p. .m.e.t.|
00000920  68 00 6f 00 64 00 2e 00  0d 00 0d 0a 00 20 00 20  |h.o.d........ . |
00000930  00 20 00 20 00 20 00 20  00 20 00 20 00 2f 00 2f  |. . . . . . ././|
00000940  00 2f 00 20 00 3c 00 2f  00 73 00 75 00 6d 00 6d  |./. .<./.s.u.m.m|
00000950  00 61 00 72 00 79 00 3e  00 0d 00 0d 0a 00 20 00  |.a.r.y.>...... .|
00000960  20 00 20 00 20 00 20 00  20 00 20 00 20 00 5b 00  | . . . . . . .[.|
00000970  53 00 65 00 74 00 55 00  70 00 5d 00 0d 00 0d 0a  |S.e.t.U.p.].....|
00000980  00 20 00 20 00 20 00 20  00 20 00 20 00 20 00 20  |. . . . . . . . |
00000990  00 70 00 75 00 62 00 6c  00 69 00 63 00 20 00 6f  |.p.u.b.l.i.c. .o|
000009a0  00 76 00 65 00 72 00 72  00 69 00 64 00 65 00 20  |.v.e.r.r.i.d.e. |
000009b0  00 76 00 6f 00 69 00 64  00 20 00 53 00 65 00 74  |.v.o.i.d. .S.e.t|
000009c0  00 75 00 70 00 28 00 29  00 0d 00 0d 0a 00 20 00  |.u.p.(.)...... .|
000009d0  20 00 20 00 20 00 20 00  20 00 20 00 20 00 7b 00  | . . . . . . .{.|
000009e0  0d 00 0d 0a 00 20 00 20  00 20 00 20 00 20 00 20  |..... . . . . . |
000009f0  00 20 00 20 00 20 00 20  00 20 00 20 00 74 00 68  |. . . . . . .t.h|
00000a00  00 69 00 73 00 2e 00 53  00 63 00 6f 00 70 00 65  |.i.s...S.c.o.p.e|
00000a10  00 20 00 3d 00 20 00 6e  00 65 00 77 00 20 00 54  |. .=. .n.e.w. .T|
00000a20  00 72 00 61 00 6e 00 73  00 61 00 63 00 74 00 69  |.r.a.n.s.a.c.t.i|
00000a30  00 6f 00 6e 00 53 00 63  00 6f 00 70 00 65 00 28  |.o.n.S.c.o.p.e.(|
00000a40  00 29 00 3b 00 0d 00 0d  0a 00 0d 00 0d 0a 00 20  |.).;........... |
00000a50  00 20 00 20 00 20 00 20  00 20 00 20 00 20 00 20  |. . . . . . . . |
00000a60  00 20 00 20 00 20 00 74  00 68 00 69 00 73 00 2e  |. . . .t.h.i.s..|
00000a70  00 74 00 65 00 73 00 74  00 49 00 74 00 65 00 6d  |.t.e.s.t.I.t.e.m|
00000a80  00 20 00 3d 00 20 00 6e  00 65 00 77 00 20 00 56  |. .=. .n.e.w. .V|
00000a90  00 69 00 65 00 77 00 44  00 6f 00 63 00 75 00 6d  |.i.e.w.D.o.c.u.m|
00000aa0  00 65 00 6e 00 74 00 48  00 69 00 73 00 74 00 6f  |.e.n.t.H.i.s.t.o|
00000ab0  00 72 00 79 00 28 00 20  00 0d 00 0d 0a 00 20 00  |.r.y.(. ...... .|
00000ac0  20 00 20 00 20 00 20 00  20 00 20 00 20 00 20 00  | . . . . . . . .|
00000ad0  20 00 20 00 20 00 20 00  20 00 20 00 20 00 36 00  | . . . . . . .6.|
00000ae0  32 00 35 00 30 00 31 00  36 00 2c 00 20 00 0d 00  |2.5.0.1.6.,. ...|
00000af0  0d 0a 00 20 00 20 00 20  00 20 00 20 00 20 00 20  |... . . . . . . |
00000b00  00 20 00 20 00 20 00 20  00 20 00 20 00 20 00 20  |. . . . . . . . |
00000b10  00 20 00 35 00 37 00 39  00 39 00 36 00 2c 00 20  |. .5.7.9.9.6.,. |
00000b20  00 0d 00 0d 0a 00 20 00  20 00 20 00 20 00 20 00  |...... . . . . .|
00000b30  20 00 20 00 20 00 20 00  20 00 20 00 20 00 20 00  | . . . . . . . .|
00000b40  20 00 20 00 20 00 27 00  54 00 27 00 2c 00 20 00  | . . .'.T.'.,. .|
00000b50  0d 00 0d 0a 00 20 00 20  00 20 00 20 00 20 00 20  |..... . . . . . |
00000b60  00 20 00 20 00 20 00 20  00 20 00 20 00 20 00 20  |. . . . . . . . |
00000b70  00 20 00 20 00 33 00 34  00 38 00 33 00 35 00 32  |. . .3.4.8.3.5.2|
00000b80  00 39 00 2c 00 20 00 0d  00 0d 0a 00 20 00 20 00  |.9.,. ...... . .|
00000b90  20 00 20 00 20 00 20 00  20 00 20 00 20 00 20 00  | . . . . . . . .|
00000ba0  20 00 20 00 20 00 20 00  20 00 20 00 44 00 61 00  | . . . . . .D.a.|
00000bb0  74 00 65 00 54 00 69 00  6d 00 65 00 2e 00 50 00  |t.e.T.i.m.e...P.|
00000bc0  61 00 72 00 73 00 65 00  28 00 22 00 32 00 30 00  |a.r.s.e.(.".2.0.|
00000bd0  30 00 33 00 2d 00 30 00  31 00 2d 00 30 00 38 00  |0.3.-.0.1.-.0.8.|
00000be0  20 00 30 00 39 00 3a 00  35 00 37 00 3a 00 30 00  | .0.9.:.5.7.:.0.|
00000bf0  34 00 2e 00 39 00 35 00  37 00 22 00 29 00 2c 00  |4...9.5.7.".).,.|
00000c00  0d 00 0d 0a 00 20 00 20  00 20 00 20 00 20 00 20  |..... . . . . . |
00000c10  00 20 00 20 00 20 00 20  00 20 00 20 00 20 00 20  |. . . . . . . . |
00000c20  00 20 00 20 00 33 00 2c  00 20 00 0d 00 0d 0a 00  |. . .3.,. ......|
00000c30  20 00 20 00 20 00 20 00  20 00 20 00 20 00 20 00  | . . . . . . . .|
*
00000c50  22 00 49 00 6e 00 76 00  6f 00 69 00 63 00 65 00  |".I.n.v.o.i.c.e.|
00000c60  20 00 28 00 50 00 47 00  2d 00 50 00 53 00 29 00  | .(.P.G.-.P.S.).|
00000c70  20 00 2d 00 20 00 53 00  55 00 50 00 50 00 28 00  | .-. .S.U.P.P.(.|
00000c80  31 00 31 00 2f 00 31 00  36 00 2f 00 32 00 30 00  |1.1./.1.6./.2.0.|
00000c90  30 00 38 00 29 00 22 00  2c 00 20 00 0d 00 0d 0a  |0.8.).".,. .....|
00000ca0  00 20 00 20 00 20 00 20  00 20 00 20 00 20 00 20  |. . . . . . . . |
*
00000cc0  00 22 00 46 00 49 00 4e  00 41 00 4e 00 43 00 45  |.".F.I.N.A.N.C.E|
00000cd0  00 22 00 2c 00 20 00 0d  00 0d 0a 00 20 00 20 00  |.".,. ...... . .|
00000ce0  20 00 20 00 20 00 20 00  20 00 20 00 20 00 20 00  | . . . . . . . .|
00000cf0  20 00 20 00 20 00 20 00  20 00 20 00 44 00 61 00  | . . . . . .D.a.|
00000d00  74 00 65 00 54 00 69 00  6d 00 65 00 2e 00 50 00  |t.e.T.i.m.e...P.|
00000d10  61 00 72 00 73 00 65 00  28 00 22 00 32 00 30 00  |a.r.s.e.(.".2.0.|
00000d20  30 00 38 00 2d 00 30 00  34 00 2d 00 31 00 31 00  |0.8.-.0.4.-.1.1.|
00000d30  20 00 31 00 31 00 3a 00  31 00 35 00 3a 00 30 00  | .1.1.:.1.5.:.0.|
00000d40  37 00 2e 00 37 00 37 00  30 00 22 00 29 00 2c 00  |7...7.7.0.".).,.|
00000d50  20 00 0d 00 0d 0a 00 20  00 20 00 20 00 20 00 20  | ...... . . . . |
00000d60  00 20 00 20 00 20 00 20  00 20 00 20 00 20 00 20  |. . . . . . . . |
00000d70  00 20 00 20 00 20 00 40  00 22 00 5c 00 5c 00 44  |. . . .@.".\.\.D|
00000d80  00 4f 00 53 00 4b 00 5c  00 43 00 47 00 44 00 4f  |.O.S.K.\.C.G.D.O|
00000d90  00 43 00 53 00 5c 00 33  00 37 00 34 00 39 00 31  |.C.S.\.3.7.4.9.1|
00000da0  00 34 00 5c 00 34 00 36  00 30 00 34 00 36 00 5c  |.4.\.4.6.0.4.6.\|
00000db0  00 62 00 69 00 6c 00 73  00 6c 00 69 00 70 00 5f  |.b.i.l.s.l.i.p._|
00000dc0  00 31 00 30 00 38 00 31  00 36 00 34 00 38 00 2e  |.1.0.8.1.6.4.8..|
00000dd0  00 64 00 6f 00 63 00 22  00 29 00 3b 00 0d 00 0d  |.d.o.c.".).;....|
00000de0  0a 00 20 00 20 00 20 00  20 00 20 00 20 00 20 00  |.. . . . . . . .|
00000df0  20 00 7d 00 0d 00 0d 0a  00 0d 00 0d 0a 00 20 00  | .}........... .|

It appears this is some sort of encoding problem.

Answer 1

You're saving your files as UTF-16, the encoding that Windows text editors misleadingly call “Unicode”.

UTF-16 is not ASCII-compatible and so won't work properly with the diff tool used by git. What you're getting is a single byte change to the input on every newline (presumably due to conversion between LF and Windows CRLF line endings) causing the two-byte alignment of UTF-16 code units to be out by one, causing the low byte and high byte to be swapped:

original text:               <     s     u     m     m     a     r     y     >
representation in UTF-16LE:  3C 00 73 00 75 00 6D 00 6D 00 61 00 72 00 79 00 3E 00
accidentally misaligned:     00 3C 00 73 00 75 00 6D 00 6D 00 61 00 72 00 79 00 3E
decoded from misaligned:     㰀    猀    甀    洀    洀    愀    爀    礀    㸀

Save your files in an ASCII-compatible encoding and you'll not have this trouble. Preferably: UTF-8-without-BOM.