HTML is the only major factor in your site and SEO. CSS does play a greater role than most people would realize.
Many designers may end up writing invalid or very hacky non-semantic markup to get a design working on their site. While this may be quite nice and find on the user (who doesn't typically give a ratte's arsch about semantic markup) it often causes a less than spectacular response in terms of your search engine rankings.
A secondary, albeit less realized way non-semantic or invalid markup can affect the end-user comes in the case of a user who uses a screen-reader. Basically the way you organize your page's code can affect the way the reader interprets things which under worst case circumstances may make your site unusable to anyone using a screen reader.
Remember:
- Keep your markup semantic and valid
- Don't use hacky code if and when possible
- CSS doesn't directly affect SEO