You can't control which files can be selected, but you can read the filename with javascript after the file is chosen. You could then display a warning and/or disable the submit button if the file is the wrong type. However, remember that you can't rely on the extension to tell you whether or not the file is really of the right type. It should only be treated as a way to help users who might otherwise waste time uploading a huge file before discovering that you don't support that file type.
Here's some example code to do that. It uses jQuery, but you could do the same in plain javascript too.
$(function() {
$('#inputId').change( function() {
var filename = $(this).val();
if ( ! /\.txt$/.test(filename)) {
alert('Please select a text file');
}
});
});