How to reduce maintenance costs

Question

It has come to the point where 4 out of 5 developers are full time dealing with maintenance or support issues.

This is mainly due to the total lack of accountability (read:reviews etc) during the development process and having dozens of small in-house legacy applications everywhere which everyone is scared to replace.

Management is hitting dept hard about little progress and projects are way behind so things like 'reviews' and 'testing' are seen as a waste of time.

How would you even begin to reduce this huge overhead?

Answer 1

If management doesn't understand what a "feedback loop" (i.e. reviews, testing), then switch jobs maybe?

From experience, the culture of a company is a difficult thing to change / takes time: if you find yourself in a situation like the one you describe, then save yourself.

Answer 2

Don't try to eat the whole elephant at once!

Identify particular applications, or parts thereof which tend to cause much of of the maintenance overhead and which seem to be relatively easily improved... Yes, I know finding such targets is not easy, but by temporarily focusing more of the effort on these "easy wins", you will accomplish two things:

• increase credibility from management and other stakeholders
• decrease, if only progressively the amount of time spent on maintenance.

and hence, have more time for cleaning up other parts of code as well as be given more time to do things properly (eg, with reviews and unit tests), as time goes.

Answer 3

That project is likely doomed then. If management is not in support of best practices such as code reviews and testing, little can be done to help.

I would try to convince management that implementing test suites and doing code reviews would save time in the long run.

Without proper process, bug fixes will just be half-thought patches that are very likely to introduce new bugs, which will require more time to fix, which will cause more bugs which will cost excessive time and money.

Taking a month or two now to fix things up and start a proper process would actually cost less time and money than continuing with the current process.

If things don't change, I would strongly consider finding work elsewhere, you're better than this, your question proves it.

Answer 4

If you need something just to start. The overhead is beatable with "freeze" approach. This is good management term, well understood in situations like this.

Answer 5

I have worked on projects like these, it's a death march. We found that the solution is to "pay down the mortgage". Find the things which are costing you the most time and work to correct those. Once they're done you'll have freed up time to work on other, smaller issues. A good idea is to pull TDD into your process, if you can lay down enough tests then you can start to make changes without as much worry about breaking more things.

Answer 6

Usually it helps to talk to management and explain to them what the problem is. Basically the same things you mention in your question. Let them know that you aren't making progress because the team spends a lot of time doing application support. Also let them know that the applications have serious bit rot and need to be fixed. Explain to them that fixing these issues takes time and investment.

Once management understands why things are not working properly, you can come up with a plan together to get things under control. You can vouch for the technical needs and be sure and be sensitive to the business needs as well. Try and come up with a plan where say 50% of the time is spent fixing the app and the other 50% is spent solving business problems. These percentages are negotiable, so see what you can get!

Good luck...

Answer 7

The options are:

• Automate
• Delegate
• Fix root cause
• Get more people
• Organise
• Skip and take the risk
• Or a combination of the above.

But the first step is to collect some data and then do a quick analysis:

1. Categorise present and historic issues by root cause, source, frequency and effort required.

2. When use Pareto Charts to visualise the areas that require most effort or come from the a single source. Paying attention to these areas will give the biggest benefits and will free people to deal with other maintenance problems.

Fix Root Cause

Always aim at fixing the root cause of an issue. The solution need not to be expensive, “the art of engineering is to do with one dollar that any damn fool can do with two”. Use analytical techniques such as Five Whys to get to the bottom of a problem.

Reading the question one of the root causes seems to be related to the current software development process and the pressure that is exerted to add functionality to existing system quickly, which leads to low quality code delivered through a stream of projects increasing the burden of maintenance. This would normally be caused due to an incorrect project planning and budgeting when project budget does not include any maintenance costs and the plan ends abruptly once the change is delivered to the users. The correct way would be to include and track maintenance costs over the lifetime of the deliverable which then would be added to the organisational budget. The figure will give more incentive to deliver robust code in the first place and help making a better choice between various project options.

Try to see if there are any other "hidden incentives" for maintenance growth.

Delegate

IT Department or a project team can often be seen as a free resource within the organisation to train users, run data conversions, reports, do routine system configuration, data corrections and other tasks that could really be done by other departments.

These tasks can be either delegated back to other business units (through providing them with the tools to do the job) or outsourced to external vendors (in case of non-business critical or specific system you might be able to source a better quality service or hosted application from outside the company).

Either simplify the tasks of educate users to carry out more complex tasks. Organise and support user community so they can resolve most common problems without having to rely on your time.

Automate

Automate things that can be automated, including day-to-day monitoring. There things cannot be fully automated at least automate them partially, provide a way of maintaining a library of scripts to do parts of a process. Help team members to learn scripting tools and languages.

Organise

Group similar issues together and do them at once. The resolution time might go up, but it takes less work to reset 5 passwords at once within the same system that do each one individually.

Provide a way for users to report and track the issues that will give you an ability to deal the with the maintenance in a more organised way, that is to say instead of having someone on the phone waiting for an issue to be resolved straight away have an issue tracking system with priorities, so the things can be dealt with in a more orderly way and give you more room for manoeuvre as to how and when you deal with specific category of issues.

Skip the Maintenance and Take the Risk

Assess the risk and impact of not doing some of the maintenance. If sufficiently low then make everyone aware the work will not get done and move on.

Answer 8

You've just described the situation common in most IT shops!

Is there a Manager in the House?

There have already been some good suggestions here. I agree with those that have said that it's a management issue. You can try all you want, but it's management that needs to understand the situation and management that has the power to set the direction for the company. Management has to decide that there will be a development freeze, or that addressing the sins of the past is a higher priority than the project of the moment, or that a rewrite is needed, or that a better approach is justified, or that perhaps some people need to have their keyboards taken away.

Right now, better coding is needed, certainly, but communication is the skill needed most. If management is beginning to feel the pain of past mistakes, perhaps it is ready to listen but then again, the managers might be the same ones who got you into this mess by making the common mistake of assuming software development is easy. So you have your work cut out for you.

We're on our Own

If management doesn't change, then try some of what many of us in IT-land have to do:

• Take incremental steps and sneak in refactorings gradually when a project to rework the code isn't authorized.
• Utilize the benefits of some specialization. You appear to have several developers at your disposal? They're not going to like it, but to gain some efficiency, a few of them are going to have to be assigned as primary resources on the support end, especially if you are in a position where management won't change and expects you to do support and projects at the same time. Support work will interfere with projects and projects will interfere with support; the difference is that the project can be managed while support is unpredictable. You have to isolate them to reduce the productivity drain caused by the frequent context switches. If you really want to be as fair to your devs as possible about it, then let people rotate between support and projects when appropriate. Take a good look at your team's skills; everyone will say they want greenfield projects, but certainly some are better at it and others may be better at working with existing code.
• Learn from your mistakes. It appears your shop authorized lots of custom apps but didn't do a great job with business analysis and testing, leading to the support load you have now. If you can't completely correct the existing code base, at least you can try to do a better job of requirements and testing (and development) to limit the increase in support new stuff will bring.
• Get users involved. Users will be valuable if your shop doesn't have access to formal business analysis and testing resources. They can also assist with communicating to management about wanting things to be better.
• Unpaid overtime. I know, just reading that phrase sucks. Based on your description, I have to assume you're already quite familiar with the concept. But if you're happy where you are working and want to stay, you may have to make what is hoped are temporary sacrifices. Example: in my shop we used to do builds manually through our IDE. But I invested time on the side to research the build script language that comes with our development tool and although we still don't have continuous integration with automated testing, the process of making and migrating builds is considerably less involved; a monkey could do it. It saves time and makes the task easily transferable to any team member regardless of their skill level (as Totophil said: automate). Those are the kinds of things you can do to help the team save time. Creating a team wiki or better user documentation are other examples of things that reduce support or the hassle involved in doing it (although management doesn't appreciate these things and might not like use of regular working hours for it). Ask yourself: is the company one you want to stick with? Do you like your team and your users? If so, some sacrifices may be worthwhile. If not, then start looking.

Several of the other suggestions are great and Totophil's post is awesome, but it sounds like you're already in a deep hole and working for questionable management; it would be difficult to do all the best practice recommendations.