Is there a small library for RSA or DSA without any dependencies like GMP or OpenSSL? (Written in C or Asm)
I used this SSL library for a small HTTPS server in busybox: http://www.matrixssl.org/
It has a commercial and an open source version and is the best I found.
You may find LibTomCrypt
useful. It's written in C, supports RSA and DSA (along with a host of other algorithms), and is public domain software. You can read about its features here and download it here.
Okay I found myself one after searching not so commen coder sites http://z0mbie.daemonlab.org/ it's under PGP/RSA-RELATED if someones interested. But are there any others?
Take a look at axTLS it's a BSD licensed TLSv1 SSL in C for embedded systems. It's built for POSIX sockets, but is easily ported.
The one thing it's missing is RSA key generation, but it can read X.509 certificates.
It's about 32KB of code space with a small RAM footprint.