I need to write a UDP server which uses self-signed SSL certificates, and the client to connect with this server as well. Can anyone help me with this?
UDP is an unreliable transport protocol. SSL/TLS is a stateful negotiation. In order to implement an SSL-secured connection over UDP, you'd need to redo much of the work that is done by TCP.
This has been done by some programs such as OpenVPN, but it's definitely the exception rather than the rule.
I am not certain you understand what these acronyms mean. Read up on the difference between UDP and TCP.
It's called Datagram Transport Layer Security, DTLS; TLS modified for datagram protocols. The Wikipedia article includes links for libraries and sample implementations.
Further to @Steve-o's answer, there doesn't appear to be an implementation of DTLS for Java. I found:
An SO question about DTLS on Android/Java, which someone answered saying they ended up using JNI to call a native library implementation.
A thread on the "security-dev" mailing list in which a Sun guy said they had no resources to implement DTLS.