When you say the DB field has a length of 4, that means 4 bytes, which is equivalent to an Int32 (4 bytes = 32 bits). That's why your column is being returned as an int32.
There are different integer datatypes in SQL Server -- if you are sure the number won't get higher than 6, you should declare the column in the database as a "tinyint", which uses a single byte and can hold values from 0 to 255. Then the SQL data source should convert it to a "byte" datatype, which will be fine for your purposes.
CLR "byte" == SQL "tinyint" (1 byte)
CLR "Short" (or int16) == SQL "smallint" (2 bytes)
CLR "int32" == SQL "int"
EDIT: just because you can do something, doesn't mean you should -- I agree with Michael Haren, the development headache of managing these less common datatypes outweighs the small performance gain you would get, unless you are dealing with very high-performance software (in which case, why would you be using ASP.NET?)