It depends on how complex your layout is.
On the whole, there are a set of "problem features" with some browsers, such as the browser not following the border-box model and assigning margin, border and padding values incorrectly.
Many designs aren't affected by the problems and look very similar in all browsers, but if you have a part of your design that has a touch point on one of the issues, it is probably better to extract the "fixes" into a separate stylesheet, rather than pollute your standard CSS with hacks.
You should find that you have
1 Stylesheet for "standards compliant browsers"
and
1 Stylesheet to cater for any issues with the others
If you have more than this, you're creating trouble for yourself and are juat adding to the number of http requests for the page.