You're running into one of the 'features' of VBA here. If you refer to some objects, like the option button, without a property specified, VBA assumes you want the default property, not the object itself. In the case of the option button, the default property is .Value, so in your code, OptionButton1 is not the option button object, but rather TRUE or FALSE depending on whether or not the OptionButton1 is checked.
Your best bet will be to change your function to this:
Private Function Option1Checked(option1 As Boolean) As Integer
//option1.ForeColor = vbGreen
If (option1 = True) Then
Option1Checked = 1
Else
Option1Checked = 0
End If
End Function
The downside here is that you cannot change the foreground color of the option button to green without referring to it by name.
An alternative that would get you the functionality that you want would be to pass the name of the option button to your Function.
Private Function Option1Checked(ByVal option1 As String) As Integer
UserForm1.Controls(option1).ForeColor = vbGreen
If (UserForm1.Controls(option1) = True) Then
Option1Checked = 1
Else
Option1Checked = 0
End If
End Function
Sub MyCountingRoutine()
Dim str As String
str = OptionButton1.Name
counter = counter + Option1Checked(str)
End Sub
Make sure you include the Else in the If..Then statement in your function, otherwise you will always get 0 back.