The Web.Config is purely for consumption by the web site serving up amongst other things the silverlight application files. The Silverlight application(s) in the web site do not natively have any notion of a "App.config".
For the record one very simply way to create custom sections is to use the NameValueSectionHandler:-
<configuration>
<configSections>
<section name="myCustom" type="System.Configuration.NameValueSectionHandler" />
</configSections>
<myCustom>
<add key="someItem" value="someValue" />
</myCustom>
<!-- other sections here -->
</configuration>
In server-side code you can access values from this section via the HttpContext object.
string someValue = ((NameValueCollection)context.GetSection("myCustom"))["someItem"];
The alternative is create your own section handler be creating an implementation of IConfigurationSectionHandler which simply has a Create that accepts an XmlNode which is the "myCustom" node.
However all of this doesn't help you client-side. You basically need to invent your own means of providing configuration to the application.
For simple sets of value the initParams parameter on the plugin's object tag is generaly used. If you have more complex settings then invent your own xml structure to hold this data and download that xml on application startup.