If it really has to be a one-liner and it doesn't matter if the code is understandable, I think the following statement should work:
Date yesterday = new SimpleDateFormat("yyyyMMdd").parse(
""+(Integer.parseInt(new SimpleDateFormat("yyyyMMdd").format(new Date()))-1));
It formats the current date as "yyyyMMdd", e.g. "20100812" for today, parses it as an int: 20100812, subtracts one: 20100811, and then parses the date "20100811" using the previous format. It will also work if today is the first of a month, since the 0th of a month is parsed by a lenient DateFormat as the last day of the previous month.
The format "yyyyDDD" ought to work as well (D being day of year).
For the first day of the current month, you can use a similar trick:
Date firstday = new SimpleDateFormat("yyyyMMdd").parse(
new SimpleDateFormat("yyyyMM").format(new Date())+"01");