C# string duplication issue

Question

Hello everyone,

I am using VSTS 2008 + C# + .Net 3.0. I have two input strings, I think they are different. But the following C# code thinks they are the same, and throws System.Data.ConstraintException, says Column Name is contrained to be unique, but value already exists. Any ideas what is wrong?

Here is my code and my input strings,

Hex view of my input strings,

http://i30.tinypic.com/2anx2b.jpg

Notepad view of my input strings,

http://i30.tinypic.com/2q03hn4.jpg

My code,

    static void Main(string[] args)
    {
        string[] buf = new string[] { "2ch", "２ｃｈ" };

        DataTable bulkInserTable = new DataTable("BulkTable");
        DataColumn column = null;
        DataRow row = null;

        column = new DataColumn();
        column.DataType = System.Type.GetType("System.String");
        column.ColumnName = "Name";
        column.ReadOnly = true;
        column.Unique = true;
        bulkInserTable.Columns.Add(column);

        foreach (string item in buf)
        {
            row = bulkInserTable.NewRow();
            row["Name"] = item;
            bulkInserTable.Rows.Add(row);
        }
    }

EDIT 1:

My confusion is, why C# Dictionary thinks they are different, but DataSet thinks they are of the same. Any solution to make the behavior consistent? Here is my code to prove C# Dictionary thinks they are different, the return buf array is of two elements.

            Dictionary<string, bool> dic = new Dictionary<string, bool>();
            foreach (string s in buf)
            {
                dic[s] = true;
            }
            buf = new List<string>(dic.Keys).ToArray(); // we got two strings here, other than one, which proves Dictionary thinks the two strings are different.

thanks in advance, George

Answer 1

Where are you putting the string into the row? It looks to me like you are creating blank rows and inserting 2 of them?

Something like this?

        foreach (string item in buf)
        {
            row = bulkInserTable.NewRow();
            row["Name"] = item;//Set the data<------------
            bulkInserTable.Rows.Add(row);
        }

Answer 2

well for a start you need you sample code to be:

foreach (string item in buf)
{
    row = bulkInserTable.NewRow();
    row["Name"] = item;
    bulkInserTable.Rows.Add(row);
}

Though that still exhibits the issue at least it's for the real reason

The reason for this is that, when creating a data table the default compare options in effect are:

this._compareFlags = CompareOptions.IgnoreWidth 
                     CompareOptions.IgnoreKanaType | 
                     CompareOptions.IgnoreCase;

From the docs Ignore Width:

Indicates that the string comparison must ignore the character width. For example, Japanese katakana characters can be written as full-width or half-width. If this value is selected, the katakana characters written as full-width are considered equal to the same characters written as half-width.

System.Globalization.CultureInfo.CurrentCulture.CompareInfo.Compare(
    "2ch", "２ｃｈ", System.Globalization.CompareOptions.IgnoreWidth);

returns 0, i.e. identical

I strongly suggest you do consider such values identical or cause further confusion down the line however if you really want to change it:

//CaseSensitive property uses this under the hood
internal bool SetCaseSensitiveValue(
    bool isCaseSensitive, bool userSet, bool resetIndexes)
{
    if (!userSet && (
        this._caseSensitiveUserSet || (this._caseSensitive == isCaseSensitive)))
    {
        return false;
    }
    this._caseSensitive = isCaseSensitive;
    if (isCaseSensitive)
    {
        this._compareFlags = CompareOptions.None;
    }
    else
    {
        this._compareFlags = CompareOptions.IgnoreWidth | 
                             CompareOptions.IgnoreKanaType | 
                             CompareOptions.IgnoreCase;
    }
    if (resetIndexes)
    {
        this.ResetIndexes();
        foreach (Constraint constraint in this.Constraints)
        {
            constraint.CheckConstraint();
        }
    }
    return true;
}

Thus you can ignore case and totally disable the complex comparison options.

If you want to make a Dictionary with the same behaviour use the following comparer:

public class DataTableIgnoreCaseComparer : IEqualityComparer<string>
{
    private readonly System.Globalization.CompareInfo ci =
        System.Globalization.CultureInfo.CurrentCulture.CompareInfo; 
    private const System.Globalization.CompareOptions options = 
        CompareOptions.IgnoreCase | 
        CompareOptions.IgnoreKanaType | 
        CompareOptions.IgnoreWidth;

    public DataTableIgnoreCaseComparer() {}

    public bool Equals(string a, string b)
    {
        return ci.Compare(a, b, options) == 0;
    }

    public int GetHashCode(string s)
    {
        return ci.GetSortKey(s, options).GetHashCode();
    }
}

Answer 3

It looks like the encoding is different on the second string. When debugging, the second string comes back as garbage. If I delete the second string and enter "2 c h" in Visual Studio, it works correctly.

Answer 4

It depends on what you mean by "the same".

The two strings have different Unicode values, but I suspect under some normalization rules they would be the same. Just so that others can reproduce it easily without cut and paste issues, the second string is:

"\uff12\uff43\uff48"

These are the "full width" versions of "2ch".

EDIT: To respond to your edit, clearly the DataSet uses a different idea of equality, whereas unless you provide anything specific, Dictionary will use ordinal comparisons (as provided by string itself).

EDIT: I'm pretty sure the problem is that the DataTable is using CompareOptions.IgnoreWidth:

using System;
using System.Data;
using System.Globalization;

class Test
{
    static void Main()
    { 
        string a = "2ch";
        string b = "\uff12\uff43\uff48";

        DataTable table = new DataTable();            
        CompareInfo ci = table.Locale.CompareInfo;

        // Prints 0, i.e. equal
        Console.WriteLine(ci.Compare(a, b, CompareOptions.IgnoreWidth));
    }
}

EDIT: If you set the DataTable's CaseSensitive property to true, I suspect it will behave the same as Dictionary.