How can i retrieve the bytecode and make a hash to see if someone has manipulated with my bytecode in-memory or on file?
EDIT: Does signing the binaries protect the code from being modified and executed? As much as I want to protect my users from making sure they are running my software. I would also like to protect the program (server) from being used by a hacked client.
How do i server side detect if someone tampered with my client?
Maybe you want to sign your jar files instead?
What you want should be possible via Intrumentation, by adding custom Transformer. See http://java.sun.com/j2se/1.5.0/docs/api/java/lang/instrument/package-summary.html
Custom classloader also does the job, as it gets bytecode when class is defined.
You can create your own classloader and do the checking manually, or you can sign your code and let the java runtime do the job for you.
So you are trying to prevent some process with the same (or higher) privilege level than your application from manipulating your application?
That's a task that's doomed to fail. Because if you add your security checks, what would prevent the attacker from modifying your isSecure() method by replacing it with a simple return true;?
I think you need to clarify your requirements (at least I'm having trouble understanding what you are looking for).
In security-related areas, you always need to answer two questions, befor you can even start to tackle a problem:
In your case, I believe you are trying to protect a Java client's class files from being modified. In that case the answer depends on what the (potential) attacker can do.
If the attacker actually has admin privileges on the machine the client is running on, then there is essentially nothing you can do. As saua above points out, if you cannot trust the the system you're running on, you're doomed.
If the attacker can only modify the class files before they reach the client maching, then signing your JAR files will let your clients detect the manipulation.
Signing the jars will protect the code from being modified. Signing involves creating a signature based on your private key. The public key is embedded in the jar with these signatures. Java will validate the signatures against your public key and refuse to load modified classes.
A hacked client will be a little harder to prevent. First an attacked would have to reverse engineer your protocol. You could take a step toward preventing this with a java obfuscator, but ultimately the attacker could just watch the wire and reverse engineer the protocol from the traffic. Even if you encrypt the client-server comms (this isn't exactly easy, considering using a protocol that already does it for you ... SSH or HTTPS) you will ultimately still be suceptible to a man-in-the-middle attack.
What exactly are you trying to protect against?
How do i server side detect if someone tampered with my client?
You can not. On the internet nobody knows if you're a dog ;-)
Seriously: the only option server-side for making any assumptions about the client, is in the information sent back over the network. By encrypting the protocol and making it sufficiently hard to reverse-engineer, you can make it hard for an intruder to hack the client, but not impossible.
NGSCB (formerly known as Palladium) is designed to make this more secure, but this has its own set of issues.
On the client, you can call getResourceAsStream with the path name to the class file in your jar.
That answers one part of your question ("How can i retrieve the bytecode"). Other answers cover the larger issues well.