special characters parsed from html

Question

I have an app that is parsing the html page and extracts some text with foreign characters for example 'Felvidék Ma'. Now I want to enter this into my database but not in this format but the original format. So can I convert it into utf 8 before writing to a sql server database or even writing to a textfile. here is the original term 'Felvidék Ma'. I use regex expressions to parse the html so Im not sure if there is an option to assist with this. Here is my code:

 If Not String.IsNullOrEmpty(_html) Then
            'get all href tags in the html page
            Dim regex As Regex = New Regex( _
                        "<TotalFound>(?<link>.*?)</TotalFound>", _
                            RegexOptions.IgnoreCase _
                            Or RegexOptions.CultureInvariant _
                            Or RegexOptions.IgnorePatternWhitespace _
                            Or RegexOptions.Compiled _
                            )

            Dim ms As MatchCollection = regex.Matches(_html)
            Dim url As String = String.Empty
            For Each m As Match In ms
                url = m.Groups("link").Value
                If Not String.IsNullOrEmpty(url) Then

I found the source of my problem. it was when fetching the html page and reading the stream. I changed default encoding to UTF 8 and all is well now. Thanks again.

  Dim reader As StreamReader = New StreamReader(responseStream, Encoding.Default)
            returnContent = reader.ReadToEnd()

Answer 1

My suggestion would be to convert the data into UTF8 before or when it passes through your application, if possible.

Answer 2

with foreign characters for example 'Felvidék Ma'

That's where your real trouble starts, there's little you can do afterwards to fix this problem. It isn't clear how you got the string, but it was created from the http stream without paying attention to the web page encoding. HttpResponse.ContentEncoding for example.

Once you get this right, everything else is simple. No need to convert anything, what you write to the dbase is the actual text. If you cannot figure this out, be sure to update your question with details that describe how you got the _html string value.