So this might be a fairly specific issue but I figured I'd post it since I spent hours struggling with it before I was able to determine the cause.
<asp:GridView ID="gvAttachments" DataKeyNames="UploadedID" AutoGenerateColumns="false" OnSelectedIndexChanged="gvAttachments_SelectedIndexChanged" runat="server">
<EmptyDataTemplate>There are no attachments associated to this email template.</EmptyDataTemplate>
<Columns>
<asp:TemplateField ItemStyle-Width="100%">
<ItemTemplate>
<asp:LinkButton CommandName="Select" runat="server"><img src="/images/icons/trashcan.png" style="border: none;" /></asp:LinkButton>
</ItemTemplate>
</asp:TemplateField>
</Columns>
</asp:GridView>
In the ItemTemplate of the TemplateField of the GridView I have a LinkButton with an image inside of it. Normally I do this when I have an image with some text next to it but this time, for whatever reason, I just have the image. This causes the UpdatePanel to always do a full postback.