It is different in various Lisp dialects. The following uses Common Lisp:
First #' is a short notation for (function ...). So the following are only textually different, but Common Lisp reads them as the same:
#'(lambda (x) (* x 2))
and
(function (lambda (x) (* x 2)))
#' is a readmacro, and transforms when Lisp code is read by the Lisp system.
If Lisp evaluates
(function (lambda (x) (* x 2)))
the Lisp system creates an function object and FUNCTION returns it as its value.
So, whenever you need a function as a value, then you need to write it like that. A function as a value is needed, when you want to pass it to another functions as an argument, when you want to return it from a function or when you want to store it in a variable. Examples:
(map #'(lambda (x) (* x 2)) '(1 2 3))
(defun (y) #'(lambda (x) (* x y)))
(defparameter *twice-fn* #'(lambda (x) (* x 2)))
Now, what is (lambda (x) (* x 2)) ?
It is two different things depending on context.
Lambda expression
The lambda expression can be used instead of a function name:
(function foo) and (function (lambda (x) (* x 2)))
and
(foo 17) and ((lambda (x) (* x 2)) 17)
Above two are legal Common Lisp syntax. There a lambda expression can be used directly.
Note that the following two forms are illegal in Common Lisp:
(#'(lambda (x) (* x 2)) 17) ; illegal in Common Lisp
(function #'(lambda (x) (* x 2))) ; illegal in Common Lisp
Macro
During the Common Lisp standardization process a macro LAMBDA has been added (it was not part of the first description, CLtL1, of Common Lisp). It makes it possible to write slightly shorter code. Example:
(lambda (x) (* x 2))
In above case LAMBDA is a macro. During macro expansion it will be expanded to:
(function (lambda (x) (* x 2)))
Remember that in above FUNCTION form, the inner lambda is part of a lambda expression, denotes the function and will not be expanded.
So, now the three examples from above can be written as:
(map (lambda (x) (* x 2)) '(1 2 3))
(defun (y) (lambda (x) (* x y)))
(defparameter *twice-fn* (lambda (x) (* x 2)))
It is slightly shorter, looks a little bit less cluttered and looks slightly more similar to Scheme code. It is just looking a bit better for programmers used to read and write Scheme code.
Summary
a) (function (lambda (x) (* x 2))) is the 'true' way to write code that returns a function as a value.
b) #'(lambda (x) (* x 2)) is a shorter notation of above
c) (lambda (x) (* x 2)) is even shorter, but uses macro expansion to create the form of a).