My productivity is decreasing as the project becomes larger. How to increase productivity as size of project increases?

Question

I initially started off with a small project, editing php files and such in notepad++. It used to be easy to think of a feature, and add it on as a separate file onto the project. As the project became larger, my productivity began to decrease because I couldn't remember all of the functions I made, and where they were stored etc... Then, I added an IDE (PhpEd), and SVN, and then noticed a large boost in productivity.

I find myself slowing down in productivity again (because everything is becoming too complex again).The project has gone from about 20 or so files --> 100 files, and is becoming difficult to manage all in my head (even using an IDE). I am wondering if people have advice on what I can do to increase productivity again. (What is the next level? if there is one)

Any software tools or tips on how to approach program design/make things simpler to visualize at least?

I know there is no silver bullet, but any ideas would be helpful.

For example, do you guys use certain tools to get through the day besides an IDE/SVN. Also, do you write code in a certain way so this won't be a problem in the future? (specifics please).

Thanks!!!

Thanks for everyone's input! Everyone has such good advice!!

Answer 1

Untyped and procedural languages break down in large projects. It's always been this way, this is why OO was made and things like decoupling and encapsulation are considered good design methodologies.

Answer 2

Sure, first use oo programming and inheritance. Next, I would use framework like code igniter for start, add module for layout (so you would have consistent site layout) and maybe few others, point is not to spend too much time on common things. Unfortunatelly migrations for CI are not great, yet they exist, so something along those lines would be useful to deal with db. With those, you can have your productivity again as you will not spend time on small things.

Thing with bigger projects is that they have multiple aspects and the more those exist, the more time you need to spend taking them into account.

Hope this helps.

Answer 3

Never be proud of editing code in a notepad! IDE saves your time and increases your efficiency. When project becomes large, you should take care of the project management and stick to a efficient "Process Pattern", such as RUP (Rational Unified Process), EUP or OOSP. Your SVN is a part of SCM. Of course, there are far more activities defined in a Pattern.As for the file management problem, you can divided them into different packages and save them in different places. If you don't have any idea of "Software Engineering", You can refer to some books written by Scott W Ambler or others about SE(Software Engineering). Remember, Software is far more than CODE!

A good developer knows that there is more to development than programming. A great developer knows that there is more to development than development. By Scott W Ambler

Answer 4

Draw and/or write it out. If you say it's 'all in your head', then take some time away from the coding and document your work. This can include paragraphs explaining why you did something.

Diagrams and other visuals will also help you keep it organized.

I've found some programmers ignore the non-technical factors in projects. This is especially true for lone developers who don't have the structure of a group to guide them. Your work is more than just code and you should look at all aspects of your process.

Answer 5

Test Driven Development might be useful to you.

Build your tests first, then write the code to pass those tests.

When you add/change any code, have a regression test suite you can run to make sure you haven't broken something. In my experience, this saves loads of time, particularly as an application grows in complexity and size.

Answer 6

Productivity on large code bases is less about tools and more about the code you write. Spend time improving the quality of your code, both new and existing. Yes it slows you down in the short term but it allows you to maintain your pace as time goes on.

This applies regardless of your tools or language. The best tools in the world will not help someone who is out to make a mess. To start with I'd keep some simple ideas in mind:

• Code quality is an investment in future productivity.
• If your gut tells you code is bad, complex, wrong, ugly... it probably is, fix it.
• Stop writing bad code, now, not tomorrow.

For much more in depth information on how to design great code I suggest having a browse around here.

Answer 7

I used to work on a project that had roughly a million lines of code (C#,C++) and it is still growing.

1) To manage such a code base, the folder structure of the svn archive was created in such a way that it would mimic various architectural layers of our product. This way it became easier to find our way through the archive.

2) Also we had our own custom build tool. When you have a huge code base, it might not be feasible to build everything anytime you want to test a small feature. Our custom build tool had lots of options, that could build the archive at any granularity.

3) Usually what I have observed is if you get the generic functionality/helpers right, building new things on top of it becomes easier and also avoids code bloat.

4)Set up a wiki, if the number of developers working on your project are more. Wiki documentation is effortless, search able and helpful if done right.

That's all I can think of right now. Hope this helps

Answer 8

Using a PHP MVC framework would greatly reduce the amount of code you have to keep track of. It will take care of many of the things you probably had to write yourself like session, database layer, etc. By using a framework, it will help you organize your files logically.

Answer 9

I have found that test-driven development (TDD) / behavior driven development (BDD) helps me keep things in order as they get larger. Even if you just write unit tests, leaving aside functional, integration, and other tests, that alone can be a huge help. As your code base turns into a lumbering, shambling Swamp Thing of Doom, your tests will keep it shambling in more or less the right direction and will prevent it from tripping over a rock, falling face down in the mud, and drowning because it's too heavy to get up.

If you write enough tests and they're well thought out they can make it so that you can make changes and know right away if the change broke something. Without tests, after a certain point you kind of can't change anything.

I don't know the PHP scene too well but it appears that there are testing frameworks for PHP, including one called PHPUnit.

Answer 10

What about this myth floating around that OOP will save the day? If that's true, how come the Linux/Kernel guys are still pushing new versions on a weekly basis? In other words are all ASM/C programmers doomed to "Hello World" complexity?

Not sure what exactly you are asking for. I mean work on the problems before and when they occur:

• If you have to many files and LOCs just reduce and reorganize them. For example by building or using rock solid frameworks for certain tasks (PDO instead of mysql()). In other words, don't repeat yourself. Short, catchy, similar function names: get_user_ID(), get_task_ID(), db_query()....

• Is communication in the group a problem? Try wikis, try internal IM, try using more comments, ask the group what they dislike and want to be improved. It totally depends on the project.

• If you work in a group, don't be all over the place. Work on your parts and improve them. That saves time you need to get into the others programmers mindset. One change at a time. Don't open up 200 files and edit totally unrelated things.

• Automate and speedup everything, if possible: deployment, versioning, documentation, tests.

• Big projects are big projects and these are always hard to fully understand. KISS is good, but in my experience it sometimes helps to just break down the thing to independently working components communicating via XML, language specific interfaces etc... In short: Make a lot of small projects out of the big one.

• I keep a personal todo/remember list as "myname.todo.txt".

• Fast deployment is mandatory! For example install a local lamp/xampp and directly view your changes.

Answer 11

You have a huge project for PHP, in my opinion. It can be done, but you really need to organize.

Break it down into independent pieces as much as possible, each with just a small interface.

"Finish" as much as you can. Then treat that "finished" code as an API, so you don't have to look at it anymore. 4.5 days of the week, assume that someone else wrote it and that you only know the interface.

Answer 12

Get in the habit of regularly studying your code and looking for ways to eliminate repetition (ie, apply the Don't Repeat Yourself or DRY principle), and be ruthless in refactoring that repetition away. The more you push common code snippets into reusable methods and classes, the easier it is to write new code, and the more leverage you gain on the problem.

Using a good IDE will help you find commonalities and apply refactorings.

Also look for open source frameworks and tools that automate aspects of your software. Using them is a massive application of the DRY principle.

Answer 13

1. Refactor after every iteration - maintains clean code base
2. Breaking Dependencies - helps the codebase to be simple to understand
3. Have automated unit tests and test harness - reduces the turnaround time for a new feature
4. Update documentation and tracebility after every iteration - helps to understand the code.

Answer 14

Big project in PHP? Not just silver bullets, there are no lead bullets and even clay bullets are scarce.

Different language!

Answer 15

My first thought is that you need to write stuff down. Your comment about "it's all in my head" worries me that

1. Eventually you're going to forget something seriously important and waste time figuring our or rebuilding something because you can't remember.
2. Future programmers on the project are going to hate you for not documenting anything.

I find a wiki, even if I'm the only user, is phenomenally helpful in organizing my thoughts. The editing process is easy and quick, which allows me to literally core dump the contents of my brain into a searchable, digital medium.

It doesn't matter which wiki you choose, just take some time to really learn the syntax so it you're not constantly switching your attention to formatting.

Then just sit down and dump.

Once you've got stuff on the page, you can start reorganizing your thoughts, and it's easier to see the entire system architecture from a high level when it's right there in front of you.

I've also found plenty of clever and interesting ways to fix my architecture by just having it all in front of me in a completely different medium.

Answer 16

As Steve McConnel says, the main challenge in programming is managing complexity.

You have several ways to manage it:

1. Using the right tools (good IDEs, VCS, etc) , you can manage an higher amount of complexity - but as complexity increases, sooner or later you will not be able to manage it.
2. Using the right techniques (OOP, TDD, etc) you can reduce complexity - but it will soon increase steadily.
3. Using the right approaches (refactoring, unit testing, etc.) and a lot of discipline you can reduce the pace at which complexity increases - but it will increase nevertheless.
4. Using clear interfaces may allow splitting the project among more people, each of which will have to handle a lower complexity - but as the team grows, the complexity of team communications will increase ever more.

In my personal opinion, the first two points (tools and techniques) are indeed good, but not something that really changes. For instance, I have colleagues in my team that use old editors, and are almost as productive as colleagues using Eclipse.
The other two points, on the other hand, will help much more: discipline, good designs, and refactoring will help maintaining complexity as low as possible, and good team working allows for tackling bigger projects that alone would be really too complex.

However, as you already pointed out, there is no silver bullet. In the end, the complexity of any successful software grows to a point where its maintenance may become so expensive to require a complete new generation.
And that's life in software :-)

Answer 17

Stop pretending that you can keep it all in your head. At some point (looks like you have reached it), it's simply no longer possible. How to deal with a project too large to memorize? Well, the same way you handle projects that were written by another person, or that you write to be maintained by someone else.

The single most important item: Write good code. Use consistent, meaningful names for classes, methods, variables etc. Write functions (resp. methods) that do exactly what the name says, no more, no less. No tricky shortcuts, unexpected side effects, timebombs, surprises. You must be able to use a function with confidence, without reading its implementation to check what it does and how.

Organize your code. Seperate GUI code from business logic code. Keep them in separate folders. If a file contains both, refactor it. If you have generated code, keep that in a separate folder (so you are never tempted to modify it).

Use a Revision Control System. (Can't be said too often)

Answer 18

Going on a limb here but breaking the project in smaller projects/API/modules/packages/assemblies (call them what ever you want) should be the next logical step here.

• Started small, few files, small editor, command line builds, all was good...
• Got bigger, versions got trickier, moved to SVN and IDE... BIIIG boost in productivity
• Then project got even bigger and the feeling of being overwhelmed creeps back again...

The human brain can only process so many things at the same time, there is just so much our working memory can handle. Hiding smaller details behind a higher abstraction level will enable you to forget about these files until there is something wrong under the hood. If that ever happens you only need to open that particular hood to fix it there. Use different level of abstractions and your projects will become suddenly much smaller where only a handful of units will be meaningful while the rest is just to make them work. OOP is very good at hiding implementation details while exposing higher level functionalists. This being said there are other paradigms than OOP that you can choose from.

So my advice to you at this point

• Break down your projects in smaller chunks each with an interface that will give you a single point of access to the rest.
• Use Unit testing and other testing techniques with a good testing framework to test each chunk individually. This will enable you to test the interface and see if it is useful.
• Never access the stuff behind the interface, if you feel the need, change the interface, the tests and then use the interface. This is the most important part, ignoring this will get you back in the problem you have now (too much coupling), therefore reduce as much as possible the points of interactions between the chunks, hence the interfaces.

This will allow you to reduce dramatically the number of things you have to remember about your project and will scale well to the next order of magnitude.

Then your next step will be at looking at OOP more in depth and learn about architecture and design patterns. They will help you reducing even more the domain of your system.

usefull tidbits

• http://www.simpletest.org/
• http://www.phpunit.de/
• http://laughingmeme.org/2003/08/05/a-few-tips-for-writing-useful-libraries-in-php/

Answer 19

Profile your process!

Just like profiling code you can profile you development process.

Look were the most time is spent and try to optimize that part of your process.

Answer 20

I think you should read Brooks's classic book 'The Mythical Man-Month, Anniversary Edition'. Although the original book describes work done in the mid-60s, the underlying truths are still accurate. And the later paper on 'No Silver Bullet' (which is included with the anniversary edition) is very insightful.

Put succinctly, the book explains that as programs get bigger, they get harder to manage (understand, maintain, develop) because there are more things to think about. Almost all the design techniques and programming techniques around are used to try to limit those problems, but the problems exist as software gets bigger (that's the "essence" part of the sub-title of'No Silver Bullet').

Answer 21

The biggest bang for your buck will likely come from:
Version control (e.g. Svn, Mercurial, Git etc.)
Build server (e.g. Hudson)
Tests
Refactoring

Answer 22

The cold hard fact is that developer efficiency is going to go down with project size. This has been known for decades. There are methods to help, but they do require some discipline.

The best solution is to go to a higher abstraction level. Write routines that will serve as building blocks, that you can use as if they were standard library or language constructs. Document their interfaces, and program only to the interfaces. If you ever feel like you need to know how a routine you're not working on is implemented, you're either using it wrong or didn't document the interface enough. Be slow to add to an interface, slower to delete anything, and remember that changing elements of it can bite you badly.

Locality is your friend. The more you can focus on a small area, the better off you are. Programming to interfaces helps this. Keeping routines cohesive helps this, so that the routines are doing one thing at a time.

Object orientation is very useful, since it promotes the above two. It promotes encapsulation and coding to an interface, and it groups related pieces of code together.

Test-driven development is valuable for enforcing programming to an interface. Write tests based on the interface, not on the implementation. This has the nice side effect that the test suites themselves help define the interface. If it isn't being tested for, you don't count on it. Make sure you can run test suites easily, and get in the habit.

Refactoring is going to be necessary. Plan on it, particularly when changing anything. You need clean code. Moreover, you will inevitably find that you've put functionality in the wrong place.

Remember also that none of this is going to solve the problem entirely. Large software projects are inherently difficult.

Answer 23

You need to rearchitect your code building process. Possibly your code as well.

I also suggest reading Code Complete.