Thanks for the question! Long reply and late, but hopefully worth the read.
Conventions
I call these temporary tools "scaffolding" by analogy to construction industry, in manufacturing they're commonly referred to as "jigs and fixtures" and if you prefer a more academically sounding name it's "tooling and production technology".
“Proper” Way
It might seem that the decision whether to invest into the scaffolding can be determined (as the project management science would have it) by a simple cost-benefit analysis: estimate the development cost, compare to predicted benefits and if the value of scaffolding is greater than the cost there is a good point in building it:
COST BENEFIT
Spec £500 Time saved over 2 years £3000
Development £2000 Savings on training £600
Total: £2500 £3600
Well, but what if a similar tool is already available off-the-shelf, but doesn’t provide quite the required functionality and hence the same level of benefit? The analysis gets an extra dimension “buy vs. build” (note that for many community tools it's only licensing cost that is free, many other expenses still apply, hence these tools need to be evaluated under the "buy" category):
COSTS BENEFIT
BUILD Spec £500 Time saved over 2 years £3000
Development £2000 Savings on training £600
Total: £2500 £3600
BUY License £500 Time saved over 2 years £1000
Setting up £500 Savings on training £300
Total: £1000 £1300
Obviously, an off-the-shelf tool is cheaper to buy; however, its return on investment (ROI) is just 30%, when for do-it-yourself solution it’s 44%.
Well, if only things were that simple, because in real life actual costs and benefits are hard to identify and quantify in advance. How do you quantify an opportunity cost of developers working on scaffolding instead of project deliverable? Or a benefit of having some production knowledge incorporated within scaffolding so new starters don’t have to spend time learning a manual build or release process? Or a benefit of anyone being able to build any version of software in one step in less than half an hour instead of a senior developer spending most of the day trying to complete the task?
Additionally there inevitably going to be some variance between predicted and actual figures and the bigger the expected variance (also referred to as “risk”) the less you can rely on the figures to materialise. The risk gives our analysis a third dimension and given that some of the costs and benefits are entirely contingent the whole cost-benefit thingy really start getting out of hand.
Practical Approach
I believe there is simpler approach. Instead of trying to treat the scaffolding as a capital investment and quantify costs and benefits for years to come, try to assess the immediate situation and take more agile approach, i.e.:
Would it take significantly longer to automate the task than to do it manually?
Can a task be completed reliably without the scaffolding and any requirement for highly specialised talent or superhuman abilities?
Does the cost exceed benefits that can be extracted in the immediate future?
If any of the answers in “no”, when you’re most likely be better off investing into the scaffolding.
Some Notes
There are a few more things to keep in mind:
Scaffolding is highly specific, temporary and sometimes even one-off. Avoid the trap of approaching its development in the same way as you would approach application or component, the scaffolding is much closer to a working prototype by nature. Remember that writing an application that covers the same functionality will take roughly ten times as much resource as the scaffolding script whereas writing a generic component will take 10 times as much effort as the application. Benji Smith has made a brilliant illustration of the point in his creative outburst "Why I Hate Frameworks". Tools are supposed to serve as means to achieve a goal, but not to be a goal in itself.
Scaffolding should be separated from the application; it’s not really a deliverable and in most cases should not replace parts of application aimed at maintenance and configuration.
Having custom scaffolding gives a real advantage over competition:
Speeds up development.
Avoids duplicate work.
Decreases the need for accuracy and magical superpowers that are as a rule in short supply and very expensive.
Reduces human error.
Provides consistency of the outcome.
Gives the ability to do things in the exact way required (as opposed to off-the-shelf tools).
Helps preserving knowledge within the team (it never quits and can always be reverse-engineered).
Boosts morale, since developers don’t have to spend their time on mundane tasks.
By being crude non-critical solution scaffolding can serve as a test ground for new tools, techniques, technologies, ideas that later can become a foundation for a killer app (FogBugz is a great example of an in-house tool that had been initially developed to test FogCreek founders’ assumptions about the correct way of handling software issues, because they strongly believed it has to be done in an exact manner that was not supported by any COTS tools. As we all know later the scaffolding was capitalised upon to become a standalone flagship product). By building scaffolding you create intellectual property.
Test Driven Development (TDD) is a notable and practical way of building scaffolding for testing. TDD provides some excellent guidelines on “when enough is enough” that can be extended on other bespoke tools.