The first thing I tried didn't work: I guessed that the problem had to do with the wider window borders on Vista. I figured UpdateAnchorRules in VCL was somehow calculating incorrectly due to the difference between the design width and the actual width of the window on Vista. Looking at the VCL source, it was clear that changing the anchors would cause UpdateAnchorRules to be called again and (hopefully) calculate correctly, since it now had the actual width of the form available.
I added
TAnchors t = BlahBtn->Anchors;
t >> akRight;
BlahBtn->Anchors = t;
t << akRight;
BlahBtn->Anchors = t;
to my form's constructor.
No joy. The behaviour was entirely unaffected.
I figured this might be too early in the process, so moved the same code to the FormShow method, equally unsuccessfully. As a last try, I changed the design of the form to no longer have akRight for the button and changed the code to
TAnchors t = BlahBtn->Anchors;
t << akRight;
BlahBtn->Anchors = t;
...which failed too - behaviour entirely unaffected, other than that I broke the positioning of the button on XP in the case that the saved size of the form (which I read out of the registry and apply to the form in FormShow) wasn't the default.
Having added a metric tonne of debug code outputting the width of the form, width of the button, left of the button, ClientRect of the form, etc. at various points during the form's lifetime, I found the problem. For some reason (presumably still window-border-related - I didn't manage to find out exactly what the reason was), VCL was opening the window with the width 4 pixels below what it should have been. The width got corrected shortly thereafter, but by that point, the anchoring (and UpdateAnchorRules) had already fixed the positioning of the button 4 pixels too far to the right.
The fix was:
void __fastcall TFooBarDlg::CreateParams(TCreateParams &Params)
{
TForm::CreateParams(Params);
int i = GetSystemMetrics(SM_CXSIZEFRAME);
Params.Width=Params.Width+(2*(i-4));
}
This corrects the initial width of the form, using the differing size of the border as reported by Vista. It causes the correct behaviour on Vista while retaining it on other Windows versions (and Vista with "classic" look).
Hope this helps somebody.