The principle you should try to follow, when considering vendor specific extensions, is progressive enhancement. The basic idea is that you should design your site to work for as many people as possible; design it to work in plain HTML 3.2, on a terminal client, with no JavaScript, CSS, or any sort of vendor specific extension.
Then, you add on style and functionality, using more advanced standards that not all browsers support, using designs for visual media, using CSS and JavaScript which may be disabled, and finally, using vendor-specific extensions, which are not at all supported by other browsers. But because you first started designing given a common set of functionality that everyone supports, you don't need the extra functionality from the new features for the site to work; it will continue working in old browsers, with old features, as the new features were added as enhancements, not as something essential for your site to work.
So, you shouldn't rely on these vendor-specific extensions, but use them as enhancements. If for some reason, you do rely on them—if your product cannot function without an extension available on a particular browser—then it may be worth using one. But in that case, your future is inextricably linked to that browser. If it falls by the wayside, you're SOL. If it winds up being used only in Korea, and your product is in German where another browser is the dominant one, again, you're SOL.
Now, there are different sorts of vendor-specific features to choose from when making these decisions. There are some features that will always be specific to a particular vendor; features that are not applicable on other platforms, or which have been proposed to a standards body and that entire approach has been rejected. These features are ones that you want to use very, very judiciously.
There are other features which are on the standards track. They are in a current CSS 3 draft, they are implemented in two or three different browsers almost compatibly. These features are ones that you may be able to begin relying on, if your user base all support them, as they are pretty certain to be implemented by a majority of major browsers (used by a majority of potential users) at some point in the future.
So, in sum, I would say that you should feel free to add effects to your pages that are supported only in an individual browser. But you shouldn't rely on these, unless you have a good reason to believe that they will soon be supported by all of the major browsers. The current state of HTML supported by the major browsers can do quite a lot; try working within the current standards, and functionality common between the browsers, as much as you can, while keeping in mind that adhering to standards isn't paying the bills, it's providing functionality, and a certain amount of style, that is.
As a direct answer to your question, about the pros and cons of vendor specific extensions:
Pros
- Get access to features that may become widespread sooner
- Provide better value to your customers
- You can do really cool, fun stuff
- Help encourage implementers of other browsers, and people working on standards, that this feature is a good idea
Cons
- The feature may go away, and never turn up even in later versions of the same browser
- If you rely on such extensions, you may limit your user base severely
- The feature may change significantly before released, leading to broken pages, or some new sniffing mode that is really hard to figure out.
As to whether you should use a JavaScript based approach or the CSS extension approach, that's a judgement you should make on a case by case basis. If you have been able to follow the principles of progressive enhancement, they are both fine to use. As long as it is not an essential feature of your site, or you find alternative ways of doing the same thing for all of the major browsers, I would say that CSS would be preferred over a JavaScript solution, as it will generally be smaller and faster loading.
Something to consider about validation is that while it's generally a good idea to ensure your HTML validates (though not always essential; see Google for example, where they consider minimizing download size to be more important than validating), validation failures due to vendor-specific properties in your CSS are less critical. In CSS, unknown properties are defined to just be ignored, so even if it doesn't validate because you use a vendor specific property, you know how it will behave in browsers that don't support that property; they will just ignore it. And if you've followed the approach of progressive enhancement, then your site should still work fine if those properties are ignored, meaning that there's no negative impact other than your site looking a little bit less nice.