You are not likely to see much of a performance difference between the two, as XAML is ultimately translated into code and compiled anyway. By putting all of your UI into pure code, you lose the benefits that XAML has to offer as a UI-centric domain specific language. It is important to maintain a separation of concerns, and keeping your UI layout and styling separated from the the code that drives your UI (and further, keeping both of those separate from your business logic), you produce a more maintainable application.
XAML is an excellent UI construction tool...much better suited for the job than pure code. I recommend keeping your UI layout and styling in XAML form.
If you wish to learn more about how XAML is built, I recommend reading the following:
Building a WPF Application (WPF)
With WPF, you get two generated files as a result of compiling your XAML: a .g.cs file representing the behavior of the UI, and a .baml file which is a compact binary representation of the XAML node hierarchy. With Silverlight, you get the generated code in a .g.cs (or .g.vb) file, but you do not get the BAML. While the ultimate end result is that any behavior IS compiled, with Silverlight you are still stuck with an XML file that contains your UI node hierarchy rather than a more compact binary representation. (Not really sure why this is, limitation of the Silverlight browser control or perhaps the limited .NET framework at its disposal.)