geographika's answer is superb. I'll only add that I use gdal/ogr when simplicity and speed is wanted in conversions. Examples include one format to another, adding image pyramids, simple merging and coordinate transformation (raster & vector). When all you want to do is jam a bunch of shapefiles together or convert a large series of rasters to a common projection gdal/ogr is faster to startup and run as it doesn't need to load the whole ArcObjects COM library to access one or two functions. (I've been told ESRI has the largest MS COM system on earth, by a large margin, including all of Microsoft's offerings.) Or check for a valid license. In one test I ran comparing ESRI arcgisscripting to GDAL ogr2ogr processing time went from 6 minutes to 10 seconds.
The only significant thing, in the realm of conversion and projection, that I feel is missing from gdal is the ability to log geoprocessing activity in the Lineage section of the metadata. ESRI tools do this automatically (most of the time).
When moving beyond conversion and getting into proper analysis though there really is no place outside of ESRI. In my experience that is. Before you dig into ArcObjects directly I'd spend some time learning what can be accomplished via arcgisscripting (arcpy in v10), which is python, and dot net.