Unused formatting is the commonest cause of 'file bloat' in Excel: a close second is that the 'used range' on a sheet is larger than you think - Excel maintains metadata about every cell on a sheet from A1 to the lowest, furthest cell you've ever edited and that takes a lot of memory.
But the 'used range' in a new worksheet is saved as cell A1 to the last cell with data in it.
A hint: take a look at the sliders in the scroll bars for your worksheet, vertical and horizontal: are they chunky grey blocks that look as if they'll only scroll fifty rows or columns? Or are they little grey dashes that scroll hundreds of rows if the mouse goes near them? When a slider bar moves a distance equal it's own length, it completes one 'page down' (or page sideways) in the available page space; the larger the space or page set (or used range in Excel), the smaller they will be.
I'd suggest de-bloating (There's got to be a better word! Deflating? Detumescing? Deforming?) your workbook. There used to be lots of third-party tools for this but Excel 2003 onwards has a built-in workbook rebuilder: just click 'Save' and 'Save As...' a web page. Then exit Excel, reopen Excel, and use 'Open' to open the html workbook (despite being an HTML file, it'll have an Excel icon), and 'Save As...' an xls file again. Close Excel again, and have a look at the file size.