SYN is only at the beginning.
ACK is on every subsequent packet in either direction.
Both are actually fields in the TCP header and can be sent with data, though the SYN and the first ACK typically are data-less.
So neither of the scenarios you describe is quite correct. The first is actually closer to reality, but all the data packets after the SYN do have to include an ACK, and also an acknowledgement number field which identifies the number of the next packet expected.
The end of a session also involves handshakes with FIN flagged packets and ACKs relating to them.
The exchanged sequence numbers are used to identify lost packets and enable a retry mechanism, and also to reassemble the entire stream of packets in the correct order.
Also, if it's the first case, are there any benefits of UDP over TCP if you just keep the connection open over a long period of time?
With UDP you can't just keep the connection open over a long period of time. There is no connection.
This sequence of SYN/ACK/FIN flags is what makes a connection.
With UDP, there are no SYNs or ACKs, so communication is one-way, delivery is not guaranteed and order is not preserved. But it has less overhead, so it's useful when speed is more important than reliability, as for example in streaming media.
This is a bit simplified yet, but it's the best I can do at the moment.
There's much more on this in the wikipedia entry on TCP and of course in the RFCs.