I was looking over Martin Fowler's recent book contents - Domain Specific Languages and I noticed some ANTLR example - that got me thinking that writing compilers will become more and more popular since people needs in this matter will increase.
So, will the compiler theory still be as arid (being subjective here) as it was until now or are there any chances that we'll get more applied, programmer oriented materials ?
Even though DSLs may seem to create more opportunities for creating new compilers, I don't think they will make the challenges of writing a compiler any easier. You can either use compiler tools like yacc to generate code to handle your dsl syntax, or you can hand carve your own parser with an eye towards better internal efficiency than what the yacc generators spit out.
Either way, you have to have sufficient knowledge of how to define and manipulate a language grammar to make your DSL work and to avoid loopholes and can't-get-there-from-here problems.
Spiffy tools help to implement the solution, but they don't solve the problem for you. To quote my high school chemistry teacher: "Sure! Bring your calculators to class! Calculators only help you get the wrong answer faster!"
There was an explosion of programming languages in the 70's and 80's. Then Java came along and killed everything off. Now we are in another phase of people inventing lots of languages. So, I'd say it is cyclical, and there is really nothing "new" going on.
However, one aspect that remains constant is that most programmers aren't very good at designing languages. Tools like yacc and ANTLR make some of the implementation easier, but they don't make would-be language designers any better at language design.
There already are some useful tools, look for Xtext, EMFText, Jetbrains MPS, Intentional Domain Workbench and Microsofts former OSLO project with the M language. All these tools make defining languages easier, although it has its cost, however for a DSL you might have a bit other requirements than for regular general purpose programming languages.
As high-quality DSLs get easier to build, we are more likely to see more of them. There are several obstacles:
Choosing a good problem domain for a DSL. It has to broad enough to have appeal to more than the author, and narrow enough to have good solutions (C# doesn't count).
Implementing a DSL well. Lots of people seem to think if they have a parser they are done. Actually, you need a lot of technology: parsing, analysis, code generation, ... (See DMS Software Reengineering Toolkit for an engine that contains what I think is needed to produce DSLs effectively)
Acceptance of the DSL by the community. Its amazing how many people insist on coding in just the programming language they know, and nothing else.
So, will the compiler theory still be as arid (being subjective here) as it was until now or are there any chances that we'll get more applied, programmer oriented materials ?
I'd say that compiler theory is actually pretty rich, but may not centered around C style languages. If you want to look at some powerful tools commonly used by academic language designers, I suggest that you check out functional programming languages (ML, Scheme, LISP, Haskell, OCaml, Scala, Clojure, etc.). Personally I prefer Haskell with Parsec, but there are many options. I think the common consensus is that the structure of these languages is more conducive to language design and implementation, at least in a theoretical sense.
Like Kristopher said above, programmers don't necessarily make the best language designers. I've seen some really cool DSL's and I've seen some pretty awful ones (my opinion, of course, YMMV). Knowledge of language concepts is a must for designing any language, DSL or otherwise (Type theory, category theory, various code analyses, machine optimization, etc). Not to mention, if you're designing a DSL, you have to have a fairly intimate knowledge of the domain you're targeting.
Tools off the shelf like yacc, ANTLR, flex, and cup can make building your compiler easier like buying wood from a lumberyard to build your house is easier than going off into the woods and cutting down trees. Both get you the material for the structure, but you still have to know how to build the house. We will definitely see more DSLs in the near future and these tools will help. Will the DSLs be worth using or even useable, however? The tools won't make a difference here, at least in my opinion. Language design employs a lot of real computer science and/or mathematics. Good language designers will have to at least be familiar with both, and good language implementers must be familiar with language design tools.