Well what does list2 actually contain? If you can express your query precisely, we can probably express it in LINQ. Without knowing what list1, list2 and column1 are it's hard to help.
What I would say is that List<T>.Contains is going to be O(n) for each item you check. If list2 is potentially non-small, you may well want to create a HashSet<T> - then each Contains call will be a lot faster.
But then again, when we know more about the situation we may well suggest a completely different solution. Please make the question as concrete as possible to get the best answer.
EDIT: If tvanfosson's solution works for you and if you're using LINQ to Objects, you've got a potential performance pit there. It would be better (IMO) to do the projection on list2 once and build a set:
Dim invalid = New HashSet(Of Integer)(list2.Select(Function(x) x.Id))
list1 = From t in list1 Where Not invalid.Contains(t.column1)