You would have to override the Expander's ControlTemplate, and change the ContentControl with the TemplateBinding of Header (this is usually a ToggleButton) with your header content.
In your case this is the +/- image/geometry and text.
However, I would go a different route than using an Expander. Since you're using a very specific orientation, I would just borrow the ToggleButton+ContentControl interaction that the default Expander ContentControl combines, and create a new Style/ControlTemplate for the ToggleButton instead.
The primary reason why I recommend this instead of overriding the ControlTemplate of the Expander is that Expander has a very large default ControlTemplate definition, and you won't have to deal with overriding all of the ExpandDirection options.
Alternately, you could just ignore the ExpandDirection property of the Expander and remove the ControlTemplate trigger code and associated Styles which deal with that.