Basically (alpha, red green and blue on a scale of 0 to 1):
Result.Alpha = BackColor.Alpha + ForeColor.Alpha - (BackColor.Alpha * ForeColor.Alpha)
Result.Red = ((ForeColor.Red * ForeColor.Alpha) + (BackColor.Red * BackColor.Alpha * (1 - ForeColor.Alpha))) / Result.Alpha
(Replace 'red' with 'green' and 'blue' above to get the corresponding functions.)
For 32 bit color, in VB.NET (somewhat optimized):
Shared Function Flatten(ByVal BackColor As Color, ByVal ForeColor As Color) As Color
If ForeColor.A = 0 Then Return BackColor ' Prevent division by zero
If ForeColor.A = 255 Then Return ForeColor ' Shortcut
Dim BackAlphaDbl As Single = CSng(BackColor.A) ' Convert to single to prevent in-calculation 8 bit overflow
Dim ForeAlphaDbl As Single = CSng(ForeColor.A)
Dim ForeAlphaNormalized As Single = ForeAlphaDbl / 255 ' Precalculate for triple use
Dim BackcolorMultiplier As Single = BackAlphaDbl * (1 - ForeAlphaNormalized) ' Precalculate
Dim Alpha As Single = BackAlphaDbl + ForeAlphaDbl - BackAlphaDbl * ForeAlphaNormalized
Return Color.FromArgb(Alpha, (ForeColor.R * ForeAlphaDbl + BackColor.R * BackcolorMultiplier) / Alpha, (ForeColor.G * ForeAlphaDbl + BackColor.G * BackcolorMultiplier) / Alpha, (ForeColor.B * ForeAlphaDbl + BackColor.B * BackcolorMultiplier) / Alpha)
End Function
I figured this out with paper and pencil and verified it by overlaying images with GDI+ and testing the resulting colors. The only difference between this function and GDI+ is that GDI+ rounds up and down inconsistently as the foreground alpha changes. This function is more precise.