What is a better approach to serialize custom class: using XMLSerializer or BinarryFormatter and [Serializable] attribute on class?
It's not possible to answer this, without knowing how you will use the resulting file, and the lifetime of it.
The decision is based on the fact that it is harder to "upgrade" the binary format. If your object model changes, it won't deserialise correctly. But if you've implemented a custom XML serialisation/deserialisation, then you can handle the "new" cases appropriately, and life will be good.
So decide more about how you will use it, who you are sharing information with, and what the possible changes to the model are.
FWIW, I sometimes use both types of serialisation in a given project.
That really depends on how you use the serialized class. If you want to pass it to other programs or want to easily debug it, use XML (but mind that XMLSerializer might produce non-compliant XML output, like multiple root elements).
In all other cases, you can use the binary formatter. But note that XML is more suitable if you change the class later - you can use XMLIgnore
and the like to keep the XML format intact.
The decision will sometimes also be made for you based on what the serialized output will be used for - while you could expose a WebService to take a binary array that is a binary serialized item, you couldn't utilize the web service easily from anything but .Net (and the end client would probably need a reference to the type).
Using XML means that the service could be exposed to any end client regardless of the platform/environment on the end client