Yes, it is definitely possible!
It can be a little tricky to get it all working correctly, but knowing a couple key things should help a lot.
The first step is to have StyleCop working correctly on the developer machine with custom rules:
1) StyleCop has to be installed on the developer machine
2) The custom rules have to be installed in the same directory as StyleCop (or, optionally, into a child directory). If they're installed anywhere else then they're unlikely to work correctly, if they work at all.
3) The Settings.StyleCop files in that installation directory, the directory containing the project being analyzed, and all directories above that one, have to combine to enable your custom rules. This one is verified best by double-clicking the Settings.StyleCop file in the project directory and making sure your rules are enabled in the editor.
The next point is even trickier, having StyleCop working correctly on the build machine.
It's pretty much the same three points again, but on the build machine which will be running StyleCop for the check-in policy. So the build machine has to have StyleCop installed, the custom rules installed, and all the right Settings.StyleCop files to create the same overall StyleCop rule settings on the build machine as there are on the developer machine. This is easiest to break by simply not realizing that the build machine has to have StyleCop installed on it. Even if StyleCop is installed, StyleCop can run via VS integration (as you're not doubt familiar with) or StyleCop via MSBuild integration (which is less common and typically less well understood). The build machine is often set up to use MSBuild integration, which means that the custom rules need to be installed in the %Program Files%\MSBuild\Microsoft\StyleCop\v4.3 directory instead of the %Program Files%\StyleCop directory you might be expecting. Additionally, if you've modified the base Settings.StyleCop file in the %Program Files%\StyleCop directory, then you'll have to copy that file into the %Program Files%\MSBuild\Microsoft\StyleCop\v4.3 directory in order to get the same effect on the build machine. Note that doing a copy/paste with a verified working Settings.StyleCop file from a developer machine is the best way to make sure that the build machine has the correct settings. StyleCop is extremely picky about those XML files, and the slightest typo can lead to bizarre behaviour that will take days to debug. Do not hand edit. You have been warned. 8 )
So there are definitely lots of ways to not get it exactly right, and you're going to see strangeness just like you've described until you get it there. Fortunately, by keeping in mind the three points I've listed and the difference between "StyleCop on the build machine" and "StyleCop on a developer machine" it should be possible to get it all working smoothly eventually. Well, it's consistently worked for me at least.
Good luck!