How would one go about finding the parameters of an undocumented Dll function? I have searched all over the internet and have found one way involving decorated functions but I cannot find a way to get those. Any help would be appreciated. Thanks!
The only way to do this is by disassembling the function and seeing how it uses the registers and stack. IDA Pro is the best tool to do this, but it's not something that is trivial.
If the only piece of information you have is the undecorated function name, then unfortunately it's not possible to deduce the function parameters from that alone.
If you're good with assembly it might be possible to disassemble the machine code for the function and reverse engineer it. But that's reasonably hard to do for all but the simplest functions.
I'm not really familiar with the PE format that Windows uses, but I'm pretty sure there is no real easy way to do this. If the symbol table has not been stripped you may be able to find some information (not sure how Windows stores debugging information in PE) but it almost certainly wouldn't help you with parameter types. The best thing to do is load the DLL into a debugger and experiment with it... monitor the raw memory on the stack frames, send various variable types, etc.
Even if you find a good resource on the debugging information in a PE file, there almost certainly won't be any information for a private function.
Is it COM Dll? If it is a COM Dll, then register it, use OLE view to know the Interafaces and parameters.
You need to disassemble the application using, as Paul noted, something like IDA Pro (or the free version of the same).
A good introductory resource is the Wikibook, x86 Disassembly. Specifically, take a look at the section on functions and stack frames. Deducing function parameters can be straightforward for simple functions taking a few parameters of standard type.
Probably the best way to get started with this sort of thing is to create a small test DLL, create a few functions with known parameters, and then disassemble your DLL to see the patterns. Learn disassembly from your own functions (for which you have the source code and know the full signature) rather than plunging into disassembling third-party stuff.
First of all, download Dependency Walker and open your DLL in it. You will see symbols exported and imported. If your function name looks like _MyFunction - it is "C" style (not decorated) and you have not too much to do with it (may be disassemble as said before)
If it is more like ?_MyFunction@LoNgSetOfSome@_StrangeChAracTers it is C++ - decorated and you may try to "undecorate" it using {unofficial} info from here
I did a fairly in-depth answer here, ReactOS is your best bet as it seems again, everybody here is a bit off base.
I would strongly discourage attempting to disassemble system DLL's.
A MUCH more adventagious (and I do not believe discussed so far from the looks of things), technique is to enumerate the contents of PDB's.
PDB files are debug symbols as you may know, however, Microsoft is required, due to the action from anti-trust court cases, to release great volumes of otherwise un-documented information.
Fully accurate, usable and updated information for massive amount's of the Windows API is only documented via PDB file's. The calling convention, argument count and even argument types and names are documented their (however not the specifics regarding the use of course:).
Review the DIA SDK, dia2dump is a good example distributed with Visual Studio, to investigate further, it also provides a solution to undecorate function's, to speak specifically to your question.
Also, kernel32 provides UnDecorateSymbolName, so you can use that also if you do not wish to link to the debug sdk libraries.
workingprogress:
I was having the same problem, I used dia2dump SDK and my problem got resolved. I posted alot and got verbosity from everyside. I wounder why people not use dbghelp and prefer to decipher or dissassemble dll's or Exe's that's more cryptic way.
I use dia2dump sdk and i got EVERY symbol whether exported,non exported etc with code. Just compile dumpbin project and see the code as well
Thnx RandomNickName42 and I can imagine how much technical you are. Alots of sowfware engineers, developers and every one suggesting me on forums to dissasamble and lookup. But you told that MSF has provided every thing for you.