This is explained in depth in Intermediate Perl.
The general syntax for variable lookups is:
 SIGIL  BLOCK  INDEXY-THING
For a simple scalar that looks like:
 print $   { foo };
You have probably seen this when you need to separate a variable name from things surrounding it:
 print "abc${foo}def\n";
If you just have a Perl identifier in the block and no surrounding mess, you can leave off the braces, which is the common case:
 print $foo;
However, this is the same thing for dereferencing a reference:
 SIGIL  BLOCK-RETURNING-REFERENCE  INDEXY-THINGS
If the thing that you get in the block is a reference, Perl tries to dereference it like you asked it too:
 my $ref = \ '12345';
 print $     { $ref };
That's a real block though, and not just sugar. You can have as many statements as you like in there:
 print $     { my $ref = \ '1234'; $ref };
Now you're not just specifying a Perl identifier, so Perl doesn't assume that you're giving it an identifier and it executes code and uses the result as a reference. Consider the difference between these almost identical say statements: 
    use 5.010;
our $foo = "I'm the scalar";
sub foo { \ "I'm the sub" }
say ${foo};
say ${foo;};
In that second say Perl sees the semi-colon, realizes it's not an identifier, interprets the code inside the braces as text, and returns the result. Since the result is a reference, it uses the ${...} to dereference it. It doesn't matter where you do this, so that you do it inside a double-quoted string is not special.
Also, notice the our there. That's important now that you're going to consider something a bit more tricky:
    use 5.010;
our $foo = "I'm the scalar";
sub foo { \ "I'm the sub" }
sub baz { 'foo' }
say ${foo};
say ${foo;};
say ${baz;};
Perl intreprets that last say as code and sees the result is not a reference; it's the simple string foo. Perl sees that it's not a reference but it's now in a dereferencing context so it does a symbolic reference (as gbacon describes). Since symbolic references work with variables in the symbol table, that $foo had to be a package variable.
Since it's easy to mess this up, strict has a handy check for it. However, when you turn it off, don't be surprised when it bites you. :)