Did you try setting the Rectangle
's margin to 0
?
<Button x:Name="Button" BorderThickness="0" Margin="0" Padding="0" Width="96" Height="96">
<Rectangle Fill="Blue" Margin="0" Width="96" Height="96" />
</Button>
EDIT: The padding must come from the button control template. Try using a custom template:
<Style x:Key="MyButton" TargetType="Button">
<Setter Property="Template">
<Setter.Value>
<ControlTemplate TargetType="Button">
<Grid>
<Rectangle Fill="Blue" Margin="0" Width="96" Height="96" />
</Grid>
</ControlTemplate>
</Setter.Value>
</Setter>
</Style>
<Button x:Name="Button" BorderThickness="0" Margin="0" Padding="0"
Width="96" Height="96" Style="{StaticResource MyButton}" />