If you don't want to use a DateTime or TimeSpan, and just want to store the time of day, you could just store the seconds since midnight in an Int32, or (if you don't even want seconds) the minutes since midnight would fit into an Int16. It would be trivial to write the few methods required to access the Hour, Minute and Second from such a value.
The only reason I can think of to avoid DateTime/TimeSpan would be if the size of the structure is critical.
(Of course, if you use a simple scheme like the above wrapped in a class, then it would also be trivial to replace the storage with a TimeSpan in future if you suddenly realise that would give you an advantage)