The problem is that you aren't really checking if StudentId of ListA doesn't exist in dtListB. What your code actually does is it compares EVERY element in dtListB with the current element from ListA and if in case they are not the same, it adds a new row to dtListB. So assume you have the following lists:
ListA = { 3, 4 };
dtListB = { 1, 2, 3 };
The first iteration for ListA, value is 3.
Check: (3 <> 1) and add '3' to dtListB (dtListB = { 1, 2, 3, 3 })
Check: (3 <> 2) and add '3' to dtListB (dtListB = { 1, 2, 3, 3, 3 })
Check: (3 <> 3) and don't do anything.
(Since the list has grown, you have two more iterations)
For the second iteration for ListA with value 4 you will insert it 5 times for every element of the existing array you compare with, and get the resulting list: { 1, 2, 3, 3, 3, 4, 4, 4, 4, 4 }. Obviously that's not the result you are expecting, and you probably have larger lists and run out of memory.
I would suggest keeping a Boolean flag itemFound when looping through dtListB. Set it to False before the loop and if a matching item is encountered, set it to True. After the loop, check if itemFound is False and if so, add the item to the list. This probably isn't valid VB syntax, but you get the idea:
For Each la In listA
stidListA = la.StudentId
Dim itemFound as Boolean = False
For n = 0 To dtListB.Rows.Count - 1 Step 1
stIdListB = dtListB.Rows(n)("StudentId")
If stIdListB <> stidListA Then
itemFound = True
Break //(is there Break in VB?)
End If
Next
If Not itemFound Then
dtListB.Rows.Add(New Object() {la.StudentId, la.AssignedId, la.TimeSheetId, 0})
End If
Next