The other answers indicating using IsNumeric in the where clause are correct, as far as they go, but it's important to remember that it returns 1 if the value can be converted to any numeric type. As such, oddities such as "1d3" will make it through the filter.
If you need only values composed of digits, search for that explicitly:
SELECT column1 FROM table WHERE column1 not like '%[^0-9]%'
The above is filtering to reject any column which contains a non-digit character
Note that in any case, you're going to incur a table scan, indexes are useless for this sort of query.