Are there types of killer applications, classes of algorithmic problems, etc., where it is better, in the long run, to create my own language?
PS: Just to be sure, I mean a new programming language and a compiler, not a new compiler for an existing language.
EDIT: Thank you for the answers. Can you provide some examples, where it is absolutly unnecessary to create a DSL or cases in which a DSL might be a good idea?
Let me just quote Paul Vick, ex chief developer of the VB compiler and now working on Project Oslo and the M language:
It is always "feasible", to use the word in your (original) question, but it is not very often useful and very rarely optimal given the abundance of well-supported and mature languages and frameworks that exist.
It is an interesting intellectual challenge, however.
One reason could be to create it as an experiment to learn about language design and compiler building.
Another reason could be to build a scripting language into an application when you don't have the option to add a third-party API.
It definitively depends on the situation. As nosklo said - If you have a good idea, a brand new concept or something like that I would highly recommend you to do that.
In general I would suggest to rely upon established technology.
But if you're interested in creating your own "language" you should check out: YACC & Lex
You might want to read parts of Martin Fowler's upcoming DSL book, if you're thinking of writing you own language.
I can't really think of a business case to create a language from scratch other than it being a tremendous learning experience.
Edit: for DSL's there are plenty of business cases, but the key here is not to get carried away and Keep It Simple.
It certainly is relevant for a person to write their own language for educational purposes. To learn about programming language design and about compiler design. But real-world uses are few and far between.
In writing your own language you are:
So, if you plan to write your own language for your project then the features that it provides that other languages don't have need to offset the above costs.
Take games development for example. They often need mini-languages within their games or scripting languages. They use these languages to script out a huge amount of the in-game events that happen. However, even in this case, they almost always choose existing scripting languages and tailor them to their needs.
I suggest that the key questions are, "What problem am I trying to solve?" and "Who gets the ROI?"
If you're trying to build your own skills and experience, then full speed ahead, but not in a production system that's supposed to solve someone else's problem.
Only if your team's core business is programming languages.
I've worked on a programming language that was created in a financial company.
Clearly, for the architect himself this was a great challenge and improved his own skills.
Inevitably, the language couldn't grow or improve at anywhere near the rate that something like C# or Java could - they have teams dedicated to doing that.
The language soon stagnated as no one new wanted to take on the task of improving someone else's pet project.
The original architect left. The language withered and died after 10 years.
Those 10 years were hell for anyone who had the misfortune to work on a dead-end language.
So go ahead, create your own language but please don't ask anyone else to actually use it. Please don't expect anyone else to thank you for it.
Seems like the main reason you'd want a new language is that you start discovering patterns in your code that existing languages don't handle well. But there are a bunch of problems with creating your own language. You'll be missing out on all the libraries and frameworks that are built up for existing languages. You'll spend a lot of time designing and implementing the new language, which is all time you don't have to spend on the real programming task. You'll spend a lot of effort convincing other developers that they should use your language. And, you'll have a hard time recruiting and training new developers.
Why not write in a language like Lisp that lets you extend the language as you discover new patterns? Then, you get all the power of a new language with all the benefits of an established language.
Designing languages can be fun. But you don't have to ristrict yourself to programming languages.
If I build a moderately complex application, I like to add a kind of macro/scripting language to make it easier to execute complex repetitive tasks. Most users won't use this functionality, but the few that do use it are very grateful. Besides, i make sure it is valuable for the support people to help them fixing customer problems.
It's completely reasonable if done to extend your skills and to learn.
Other than that, if you have to ask the question, then it's not. If you're trying to figure out whether you can deal with a certain class of algorithm or a certain problem domain better than existing languages, first off you need to be an expert in the area you are addressing. You will know it is appropriate when your skills and experience tell you so.
And you could be wrong about that, too, but you'd need another expert to convince you of that (or to show you that you aren't the expert you think you are). A lively discussion that would be, not a simple Q-and-A as you'll find here.
You can, just don't catch yourself in the anti-pattern "Recreating the square wheel".
Meaning you are recreating whats already been done, only poorer than the original(s).
Wouter was known for creating a new language for any new idea. You can draw inspiration from his work: Wouter's programming language page.
Learning exercises aside, it is reasonable to create your own programming language only when you understand other languages, your specific problem domain, and the way that existing languages address that problem domain and this understanding is thorough enough that you know a new language is a reasonable solution without needing to ask the question.
Except for self-educational purposes, I would like to claim that there is today no need whatsoever to create your own language. In any circumstance. Ever. Regardless of what you want to do, there are boatloads of existing languages you can take/adapt to your needs.
I don't think you can program without creating a new language, so it's good to realize that's what you're doing and understand the issues.
An off-the-shelf language like VB, Java, C#, etc. is just a base language. As soon as you add classes, methods, etc. to it, you have added vocabulary and semantics. There are many ways to implement languages - parsing & translating, parsing & interpreting, macros on top of an existing language, adding classes&methods to an existing language.
How do you know if you've done this? The measure I use is edit count. If one-sentence requirement A comes along, I proceed to implement the requirement in code. When I'm done & got all the bugs out, I check in the code, and the code repository gives me a list of the changes I made, B. The smaller B is, the better the language is. Averaged over the space of real & possible requirements, that measure tells me how "domain specific" the language is.
If it takes N code changes to implement 1 requirement, and you sometimes make mistakes, then the number of bugs you introduce is roughly proportional to N. In the limit where N = 1, it is almost impossible to introduce a bug without trying to.
Note that this is a direct challenge to the "code bloat" we see nowadays.
ADDED: In response to your request for an example, see differential execution. I won't say it can be understood quickly, but it does significantly reduce UI code.
Only big companies create their own language. They don't forget to include many libraries and put it on top of their own OS for building the associated ecosystem.
I think of Microsoft with C# and Apple with Objective-C (though in this case the ownership is not official).
When to create your own language?
When you want to, as a large hobby project.
For a domain-specific language. These can be quite elaborate; look at what goes on in the Interactive Fiction (or text adventure) community by checking out the archive.
When your aims are very ambitious, and you think you can make a real advance, like Paul Graham's Arc project.
Also, in any sufficiently adaptable language (perhaps C++, definitely Common Lisp) in the process of developing low-level constructs.
When to avoid it like you would, I hope, avoid a cliche like avoiding it like the plague?
When it's to be the basis of continuing development for real projects. It will always wind up lagging seriously behind what's commercially available for cheap, and will cripple further development. I worked for a company with its own version of COBOL, and never want to work at another company that maintains its own language. We watched other versions of COBOL get better capabilities, and better tools, while we were stuck with the same problems. (I don't want to work with COBOL ever again, but that's another story.)
The situations in which you might create your own language don't fall into this. Hobby projects are not used for real development. Something like Arc will succeed (and get multiple implementations and further evolution and development) or fail (and nobody else will use it). A small domain-specific language is only part of a project, and since it's small it can be improved over time. A text adventure language is used to write individual games, and those games, in addition to being hobby projects, are almost never used for continuing development.
My perspective is that DSLs are generally a "weak idea", and it's more productive in the long run to use a standard language and build your domain-specific needs as a library of the "non-DSL".
However, it may turn out to be that your needs are custom enough that it's preferable to have a DSL(not just a slightly modified gcc or lisp implementation) for your company. Many companies use drop-ins of current languages that target what they are doing, without writing/maintaining their own language. E.g., I've heard that PHP has a nice drop-in; Lua is designed around being a drop-in, ModelView uses Python and AutoCAD has AutoLISP as its scripter.
What language is good at is compositionality or putting the same components together in different ways.
If your domain problem just needs you to set a bunch of orthogonal switches, a language probably doesn't add much over forms, a graphic UI or a straight text-config. file. (I'm assuming here that a file full of key, value pairs isn't what you mean by a "language".)
OTOH, if your configuration is like real language eg. verbs and nouns can be put together in many different (and novel) combinations to any degree of complexity, then a language is going to become almost inevitable, because the combinatorial explosion of trying to specify what you want by any other method overwhelms.
When is it reasonable?
When you feel like it!
Don't listen to these people who have snarky comments that basically say:
"Don't do it because its too hard and X language is better than any language you can come up with".
The thing is, creating a DSL happens all the time. A framework is a DSL. A macro is a DSL. Every time you write a function for your program, that is part of a DSL. Sure, its within the bounds of the grammar, but vocabulary is part of a language. This is why industries often create their own vernacular: its more efficient!
If "don't do it" was the right answer, we would all be writing COBOL and Fortran.
Probably never.
Lua is the best choice you can get if you want to embed the language in basically any other language.
Small Domain Specifc Languages are currently in, and it make sense in some applications.
Other then that, reasons are mainly academical.
Creating a language when its not needed, is really a bad thing to do due to the complexity involved in developing and maintaing it. I have seen many projects that introduce some kind of scripting language specific only to that program, and it was the thing that was slowing down development of the base thing by huge amount. Good examples are for instance automation languages like Phantom, AutoHotKey, AutoIt. Those tools would be IMO much better if they used some well known emeding langauge like Lua.