How can I build a native Mac OS X installer for my application, on a non-Mac platform?
For example, I have a windows pc and a Java application. I want the windows pc to build an installer (possibly inside a .dmg archive) that works with Apple installer.
So, a couple of quick questions
First, why do you want an installer. Most Mac users prefer apps that are just drag installed. If you are not writing Mac OS X specific code it is hard to imagine you need to place bits in special places, like Application Support or LaunchDaemons. Assuming everything you have just goes in one folder why bother with an installer at all?
Second, why would it be a problem to build the Mac installer on a Mac? Surely you have a macintosh around to test the app (you are not just blindly shipping it for Mac without testing it on a Mac, right?).
Okay, having said that, assuming you still have a good reason to actually build this on a PC, there are some bits that are not going to be easy. Basically a .pkg is a bunch of text scripts, localizations, an archive file (Archive.pax.gz), and a bill of materials (Archive.bom).
Assuming not much changes between builds, you can make the installer on a Mac, and then just rebuild the bom and the pax.gz, replace them into the existing .pkg, and batch a few pieces of metadata. The pax should be easy enough to deal with (pax is a standard archive format), but the bom file may prove a bit trickier, since I do not believe it is publicly documented, and I doubt the tools for creating them (mkbom) are part of darwin (not opensource). So you are going to need to figure that out and right a custom tool to create the bom file.
In other words, this is likely to be a large amount of work.
The usual way to install a app on a Mac is to drag the app to the application folder. Most programs come as a DMG containing the app and a symbolic link to the application folder. Why would you want it any other way? You need to think Mac to build a great Mac application! Look and feel is very important, especially for Mac users.
As others have already pointed out, you are really taking the hard solution to this problem, and your users will curse your name for it, unless you have a really good reason to do so. It is true, that certain types of applications will require installers for Mac OSX. These generally include:
Ok, so that last point is a bit sarcastic, but do you get my drift here? :) Basically, if you are writing a normal, end-user application, you should distribute it in the normal way that Mac users would expect, which is a DMG file containing your application's bundle. Or if you want to be really fancy, stick an alias to the "Applications" folder inside of the DMG to help the user drag the program in there. Unless you are writing something which must install itself into the system, rather than simply be run by the system, there's no reason to use an installer here. Also, keep in mind that this is OSX, which already contains a fully-functioning Java JRE, so you don't need to worry about packaging the JRE in an installer or anything like that.
Now that you've been properly scolded for asking this question, I'll answer it with the assumption that your software falls in one of the categories above. Really, your best bet would be to go with a commercial solution such as VISE installer (which, again, I can state confidently that a vast majority of Mac users will recoil in terror when installing a product made with this tool), which allows you to do exactly what you are looking for here -- basically, to make a cross-platform installer which builds for the various platforms you want to support from a single installer file.
Again, though, your best bet here is to do what users of that platform are most comfortable with (which is why all the responses to your question urge you not to make an installer). That means, however, if you really must make an installer, you should use a non-cross-platform framework; windows users will feel most at home when presented with a standard MSI installer, and Mac users will feel most at home with an Apple Installer pkg. The PackageMaker program is notoriously limited, though, so if you must, you should use iceberg instead. This will mean a bit more maintainance for you, since you'll need to tend after two (or more) separate installers, but if your software is really so complex as to demand this, you should be willing to make the sacrifice for the comfort of your users.
Put everything into one JAR file, add it to a ZIP. Done.
But seriously, you're wanting to distribute your application to Macintosh users without testing it first? What planet are you on!?
It is difficult to create a .dmg on Windows, but it is certainly possible to create a .app file structure that you can then zip, as others commented have mentioned. There are times when a regular .pkg will not cut it and you want to provide dialogs, pre-Installation checks, etc. You can do that with BitRock installbuilder, you can build installers for mac, Linux, Windows, Solaris from each of the other platforms.