All forms come with a $form[#submit]
property that describes what functions are run when the form submits. The default is formname_submit, of course, but you just need to add new ones to that array.
So, you should use hook_form_alter and add another item to the $form['#submit'] array.
You can get the form id easily using the Devel module, or by looking for in the HTML of the pages themselves. (Hyphens should be translated to underscores if you take the latter route)
I get system_image_toolkit_settings for that form on my installations, but that might be dependent on which image library you're using (I use GD).
Though, I'll admit I'm scratching my head a bit about what submit handlers you're wanting to add to that one ;p
Edit:
Some sample code in reply to OP's comment:
What you're basically looking for is this: (from http://drupal.org/node/144132)
function my_module_form_alter(&$form, $form_state, $form_id) {
if ($form_id == 'my_form') {
$form['#submit'][] = 'my_additional_submit_handler';
}
}
Of course, you'll need to follow that up with function my_additional_submit_handler
in your custom module for anything to happen.