First of all, why your code doesn't work as is: ref doesn't have the same meaning in F# as it does in C#. In F#, 'a ref is a type. It's not magical, either - it's really just a record with a single field contents, defined as follows:
type 'a ref = { mutable contents : 'a }
It is not a pointer or reference to a local variable. Thus, when you write this:
let x = 0
let y = ref x
you do not have variable y referencing x. Instead, you have variable y of type int ref, with the value of contents initialized to the value that x had (that is 0). You can change that:
y.contents <- 1
and this will not change the value of x. It will only change the value of contents
F# also provides some syntactic sugar for 'a ref. Specifically, this:
y.contents <- y.contents + 1
can be written shorter as:
y := !y + 1
Thus := is a shorthand for assigning to contents, and ! is a shorthand for reading its value.
See Reference Cells in MSDN for more information on ref.
Now F# has a magic cast associated with 'a ref type, that lets you pass an instance of that type to a foreign function that expects a byref argument (both ref and out in C# map to byref in IL). In this case, if function changes the value of the argument, the value of contents in your ref instance changes accordingly. In your example, ref parameterSet created a new instance of ref, passed it to function which changed it, and then discarded it. What you should have done is this:
let parameterSet = ref(new ParameterSet())
let retVal = currentPanel.GetParameterSet(name, parameterSet)
...
// use parameterSet.contents as needed
Alternatively, you could use let mutable to declare a mutable local variable, and then use the magical & operator to pass it to the function directly as byref:
let mutable parameterSet = new ParameterSet()
let retVal = currentPanel.GetParameterSet(name, ¶meterSet)