It looks like you're confused about the semantics of XAML. Until you get more used to XAML it might help to think about it as the C# equivalent. This is essentially what you're doing right now:
Style BorderStyle = new Style();
Border inlineStyle = new Border { Style = BorderStyle };
Style listBoxItemDefaultStyle = new Style();
listBoxItemDefaultStyle.Setters.Add(new Setter(StyleProperty, inlineStyle));
ListBoxItem item = new ListBoxItem { Style = listBoxItemDefaultStyle };
One issue is that you're setting a Style for the ListBoxItem from a Setter inside the Style for your ListBoxItem, which is of course going to cause some sort of problem with recursion. So removing that extra Style from the code we get:
Style BorderStyle = new Style();
Border inlineStyle = new Border { Style = BorderStyle };
ListBoxItem item = new ListBoxItem { Style = inlineStyle };
This is invalid because it's trying to set a Style property (of Type Style) to a Border object. This is essentially what's at the core of the error you're seeing.
What you really want in this case is to change the ListBoxItem ControlTemplate to incorporate your Border Style. This is the default Style modified to use your Border instead of the standard one using TemplateBindings to set its properties:
<Style TargetType="{x:Type ListBoxItem}">
<Setter Property="Background" Value="Transparent"/>
<Setter Property="HorizontalContentAlignment" Value="{Binding HorizontalContentAlignment, RelativeSource={RelativeSource AncestorType={x:Type ItemsControl}}}"/>
<Setter Property="VerticalContentAlignment" Value="{Binding VerticalContentAlignment, RelativeSource={RelativeSource AncestorType={x:Type ItemsControl}}}"/>
<Setter Property="Padding" Value="2,0,0,0"/>
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="{x:Type ListBoxItem}">
<Border x:Name="Bd" Style="{StaticResource BorderStyle}" Width="200">
<ContentPresenter HorizontalAlignment="{TemplateBinding HorizontalContentAlignment}" VerticalAlignment="{TemplateBinding VerticalContentAlignment}" SnapsToDevicePixels="{TemplateBinding SnapsToDevicePixels}"/>
</Border>
<ControlTemplate.Triggers>
<Trigger Property="IsSelected" Value="true">
<Setter Property="Background" TargetName="Bd" Value="{DynamicResource {x:Static SystemColors.HighlightBrushKey}}"/>
<Setter Property="Foreground" Value="{DynamicResource {x:Static SystemColors.HighlightTextBrushKey}}"/>
</Trigger>
<MultiTrigger>
<MultiTrigger.Conditions>
<Condition Property="IsSelected" Value="true"/>
<Condition Property="Selector.IsSelectionActive" Value="false"/>
</MultiTrigger.Conditions>
<Setter Property="Background" TargetName="Bd" Value="{DynamicResource {x:Static SystemColors.ControlBrushKey}}"/>
<Setter Property="Foreground" Value="{DynamicResource {x:Static SystemColors.ControlTextBrushKey}}"/>
</MultiTrigger>
<Trigger Property="IsEnabled" Value="false">
<Setter Property="Foreground" Value="{DynamicResource {x:Static SystemColors.GrayTextBrushKey}}"/>
</Trigger>
</ControlTemplate.Triggers>
</ControlTemplate>
</Setter.Value>
</Setter>
</Style>