On being a contractor at a shop where they don't use database indexes

Question

I've been freelancing for 5 weeks at a Rails shop where I just learned today that the "Technical Lead" of a team of about a half dozen engineers, never applies indexes to the databases backing the websites being developed. I have a week left on my contract and have no idea if they intend on offering an extension. I feel like I owe it to the director who hired me to report this, but realize that if it trickles down to the team/lead, it will sour relations.

But I feel adamantly that it does need to be reported, and a policy developed regarding using database indexes. Bear in mind this is complicated by the fact that I previously worked for this same company directly as a salaried employee a year and a half ago, and at that time I profiled a couple of applications for them for major clients, and discovered horrendous performance issues. Both those accounts were lost, one before, one shortly after, the first time I left, and now a mere fraction of the employees work there than used to. We are starting to land clients of the same stature again, and I fear that the same continued lackadaisical attitude toward performance could literally mean the end of this company.

Thanks in advance for any advice.

Answer 1

If this were asktom, he'd say fire their DBA. Under no circumstances can a database be expected to perform/scale without any indexes. Things will only become worse as the application scales(?) to more users-think of the IO contention from all of the constant, multiblock reads.

If it were me, I would consider it my professional obligation to inform management of the potential performance risks. Depending on the site/company, a lackadaisical approach to performance could lead to lost clients and revenues.

Depending on your RDBMS, not having indexes will also lead to concurrency problems.

Answer 2

It depends. Does the lack of a database indexes affect response time in a noticeable way? Furthermore, at the intended level of deployment, will this issue become immediately obvious? As CS people, we would gawk at a 10ms response time when only 0.1ms is truly needed, but in certain situations, other factors will limit far quicker than the DB backend. [Fancy user interface code, for example, could be resource intensive]

Just because something is bad practice and unscalable doesn't mean it's inappropriate for the target audience or client. If you can make a strong business case as to how this will hurt customer satisfaction or lose sales, then report the activity.

Otherwise, you're just whining to what will most likely be deaf ears.

Answer 3

Ordinarily as a contractor you should keep your mouth shut. But you used to work there before as a salaried employee, and they allowed you to come back as a contractor, so someone likes something about your work. If you know who respects you, you can discuss your suggestion with that person. Otherwise the company will go under and you won't have to worry who you have sour relations with.

Answer 4

I feel like I owe it to the director who hired me to report this, but realize that if it trickles down to the team/lead, it will sour relations.

There's no real perfect advice for this. You could tell the director about this and everything goes fine or it could explode in your hands.

But as a programmer I feel that all issues should be reported. Good or Bad things can happen but to keep the level of respect and integrity (because they could find out later that you knew all along and never told them)

Answer 5

Talk to the technical lead first (they are the technical lead, after all).

First, get their side of the story; ask them why they doesn't think indexes are required. There may be a good reason (but I doubt it).

Try to convince them why your side of the story is correct. And use hard data as well. Set up copies of the DB on your local machine (full or representative data, not just the schema) and time the queries in both non-indexed then indexed form.

Also time inserts, updates and deletes in both forms - the only possible thought I can come up with why they'd not like indexes is because the table is mostly written and infrequently read but even then, I'd have a hard time justifying no indexes.

If they can't convince you and you can't convince them, fire off a letter to their boss expressing your concerns and your (full record of) discussions with the tech lead.

Don't say they're wrong, just state that, in your professional opinion, you don't believe the database is as efficient as it could be - this puts the blame on the inanimate database rather than the (possibly incompetent) tech lead and will be less likely to affect your chances of future work there.

Then let the company take care of it. It's their issue, not yours.

Answer 6

Talk directly with the person involved first -- and be friendly/helpful, not combative. Maybe you have your information wrong or perhaps there is a good reason -- "we're waiting until we get some performance data to tell us where we need to add them." Don't go above someone's head first, unless the situation is critical and you don't have time to discuss it with them.

If, after talking with the team lead, you still feel like you owe it to management to let them know about the issues, then document what you think the problem is and how you would solve it. Don't be critical of anyone, just be helpful.

Answer 7

This may be an indication that the app does not need a database at all, e.g. just store things in files.

But in general database exist to do clever things with indexes.

Answer 8

I know it's going to sound obvious, but you didn't mention it, so I guess I will. Have you talked to this "technical lead" about this? Maybe he thought there were some applied, or has some other "good" reason for not having indexes on those tables (though I can't really think of one -- "we haven't gotten around to it" is fine if the site is fast, but not if it's actually affecting real perceived performance).

Once you get a reason from him (anything from "I never got around to them" to "indexes are the devil"), don't think it is a valid one, and you can show that the time investment in looking at the DB and creating the indexes at least may help with a business-visible problem (ie. searches take too long), then approach this director. I work as a consultant, and I consider it part of my job to tell those who hired me what I think they need to know about their technical systems (which they usually only casually understand). They may choose to ignore my advice (ie. "he's so smart and doesn't like indexes, we're going with him"), and that's fine. I've discharged my obligation.

But knowing that there is a problem and not telling them because I think it'll get someone mad at me? I guess I've never actually had the situation come up, but in my "ivory tower", I think the answer is clear. I'd tell the person I report to (or however high I need to go to get above this "technical lead") my opinion of the situation and leave it to their judgment about what needs to be done.

Answer 9

What is the "Technical Lead"'s reason for not using indexes?

First thing I would do, is try to prove to the technical lead why indexes are required (make sure all communication is documented in case they try to take credit).

If the technical lead refuses to see the light, then you have to go higher, silence is consent.

Answer 10

I assume that since you're talking about reporting this, that you're going around the technical lead, and that therefore the technical lead was not receptive to your concerns. If that's not the case, a technical discussion with the tech lead would of course be the first thing to do. In general I've found that taking technical concerns outside of the technical team generally signals some sort of extraordinary problems, some sort of technical concern that can't be resolved but has to be.

So, assuming you're in a situation where the lead/team is incompetent/won't listen to reason-

Given that you've been called back, you can assume that director likes your work and you have some technical credibility with him. If they've lost clients and they're not complete fools then they know the tech team is messing up and will appreciate useful suggestions. My personal feeling is that if I were in this situation and didn't say anything, I'd be one of those guys who just does crappy work and "get's over" until he get's caught. I don't ever want to be one of those guys.

A side note - this doesn't have to be confrontational or sour relations, a simple information "I've noticed that X isn't being done..." is sufficient.

Answer 11

I agree with the answer Pax gave above, but I'd like to note that you should try to set the sample dbs up with realistic amounts of data inorder to ensure that the difference between indexed and non-indexed scenarios is apparent.

Answer 12

Your fiduciary responsibility is to the person who purchased/authorized your contract time, not the technical lead (IANAL). It's plainly a good idea to discuss the issue with the technical lead, but it's up to your contract manager whether you should concern yourself with things outside your direct contractual responsibilities.

Some questions to ask yourself and/or the contract manager

• Is it proper for you to care about the success of the company? This may sound cold-hearted, but consider: You are not an employee, so they have explicitly chosen not to pay you to worry about that kind of thing. They have even less loyalty to you than to their employees, and no loyalty is expected from you in return. This is the big upside of contracting - you can go home and sleep soundly. They don't pay you enough to lose any sleep. It is completely ethical to hold this view.
• If the company did hire a contractor to do business and/or performance analysis, they would probably pay that contractor more and explicitly expect precisely the kind of feedback you're contemplating.

Based on the limited data you've given here, I would say you could:

• Ask the contract manager whether he wants your comments outside of your immediate obligations. This can likely wait until the end of your contract term.
• At the beginning of the next contract, ask that contract manager whether this kind of thing is within the scope of the contract. I.e. chalk it up to experience that you even had to wrestle with this dilemma.
• Look into "upgrading" your contract profile; maybe you can get paid more doing performance optimization consulting.