When have you been put on a new project that was far more challenging than anything you'd worked on before?

Question

I work for a small trading company. It's a very small team of four developers, two of whom -- myself and another guy -- do the coding for the actual algorithms and develop the main application used by the traders.

To be honest, the work we do here is generally very simple from a programming standpoint. The traders come up with a relatively straightforward mathematical idea, we implement it as an algorithm, and then we try it out and tweak it to make it as profitable as we can.

I think the boss considers us to be pretty bright guys, and he is at least somewhat aware of the fact that our normal responsibilities aren't too challenging for either of us. Now just recently he's had this extremely ambitious new idea: he read in some academic journal about "intelligent" trading algorithms that basically learn from the trades they make over time and self-adjust to maximize profit; and he wants us to create our own. And I'm not just talking about tweaking various parameters up or down; this is supposed to detect patterns, identify types of traders (e.g., informed vs. uninformed), basically function as a rudimentary AI.

Needless to say, this is a huge jump in difficulty from what either of us has grown accustomed to doing here. I think we're both enthusiastic about the challenge, but I'm also pretty skeptical about our ability to create this thing in any reasonable amount of time, or even at all (at least as our boss has imagined it).

I'm just wondering who else has had this experience -- of suddenly being assigned to something far more challenging (perhaps even ridiculously so) than any of their previous work for the same company. Were you able to actually achieve what was asked of you? Did you have to scale back the goals of the project to be more realistic? Were you pleased and/or surprised with the end result (if in fact you ever reached it)?

I'm also interested in the personal interactions that might have taken place. Did you tell your boss at the start that it was too much, or did you opt to just bite the bullet and give it a go? If you were working on a team, how did the size and difficulty of the project affect the other developers?

Basically I'm looking for others' perspectives on this situation, and any helpful advice that someone who's been in the same position might have.

Answer 1

You are not alone

I'ld recommend reading

Edward Yourdon's book Death March: The Complete Software Developer's Guide to Surviving 'Mission Impossible' Projects, which has a second edition simply titled Death March.

Answer 2

There are a few AI techniques out there for learning/tuning/discovering parameter adjustments. They've all got their strengths and shortcomings. A short list:

1. Genetic Algorithms
2. Fuzzy Logic
3. Nearest-Neighbor algorithms.
4. Support Vector Machines.

There's also plain and simple least squares minimization.

Answer 3

Don't jump directly from the routine to the impossible. Instead, break this up into iterations, each acting as a proof of concept and prototype. Success, or lack thereof, at any stage will help you decide what to do next, even if that means giving up on the idea. However, so long as you fail early and often, you won't wind up putting in more resources than the project deserves.

Answer 4

My suggestion would be to consider doing some investigating to see how well do these "intelligent" trading algorithms work and try to set some thresholds for what is wanted and what can be done.

Another line of thought is to consider applications that are self-optimizing like MS-SQL server and see how they do things and see how that can apply to your trading scenarios.

Answer 5

Answering you question - Yes. When I started we all were given this baby ASP.Net page and we did it. Then we were given a slightly more advanced page and we did that. Then we got split up, and got our own tasks. Mine was to rewrite, from the ground up, the 3rd or so most essential section of our website, by myself. There was so much baked-in-weird logic that 2 years later, I'm still finding stuff that is odd and I think is wrong (but is right for some strange reason) - in fact I got interrupted writing this answer to go over one of those cases!

So anyway, we've all been there before. But think back to when you were learning CS and you were given a problem you had no idea how to solve. Break it into smaller pieces that you know how to do. Iteratively solve more and more complicated problems until you get closer to the goal. Whiteboard and draw lots of diagrams and arrows. Don't try and design the entire thing at once, it'll be too big and monolithic and you won't get anywhere.

Answer 6

Make use of domain experts: for example, have your boss talk to the traders, so that a trader can understand, express, and explain your boss' idea to you as a "relatively straightforward mathematical idea".

Answer 7

That is a tough situation to be in! I agree with the comments in Stevens posting above, that is sage advice. I would add that If you really do feel totally out of you're element, and are worried about burning the budget, you might consider bringing in a consultant who has expertise in that area. This can save a tremendous amount of trial and error, and research time, and the responsibility is shared.