The reason for this is that with a char data type, you are sorting the rows as a string.
The idea to ORER BY CAST() is correct, however performance of this will go down as the number of returned results increases.
If it's only numerical data in this column, the best practice would be to find a suitable numerical data type and change it.
If you really can't change the column and you find yourself having performance issues, I suggest having a sort order column which contains the value cast to an integer (will nulls converted to an appropriate value).
Index the sort order column and ideally, add a trigger to the char column so that inserts or updates to the char value trigger an update to the integer value.