I have a set of randomly distributed point in a coordinate system and I'm interested in recognising if some of the points form any geometric figures like a square, circle, parallelogram or simple lines.
Do exists any libaries or algorithms for recognising patterns? Or maybe some tipps how to program something like that?
Haar and HOG are the standard algorithms.
A good library for image recognition - admittedly more for faces and vehicles and such, rather than geometry but that's just a question of training - is OpenCV, which has Java bindings if you dig enough.
It's quite a complex issue. One of the many tools you might want to read about is the Hough transform, it's used mainly to detect lines but can be adapted to other shapes.