Process « Code In Review

Archive for the ‘Process’ Category

Rejecting Defects

Thursday, March 12th, 2009

Not all defects are created equal – sometimes things are actually by design.? Sometimes the problem was a one-off type error that can’t be reproduced.? Sometimes the person reporting the problem simply gets it wrong.? For this reason, every project I have ever worked on has had some mechanism for rejecting defects.?

There are generally three possible reasons a defect is rejected.? The first is that the problem cannot be reproduced.? In this case, the rejection is done by either the testing staff (if defects are vetted before being assigned to a developer), or by the developer assigned to the defect.? A large number of non-reproducible defect reports is often a symptom of multi-threading issues.? Sometimes though, weird things just happen, and the effort required to protect against the 1 in a million chance of it happening again is too great to warrant attempting to fix the problem.?

The second reason is that the problem identified is actually correct behavior.? For these rejections, it usually requires a domain expert to make the appropriate judgement.? If a large number of rejections of this type are being seen, it usually indicates a usability error since users are not able to determine “correct” behavior.? Often rejections of this nature will spawn secondary defects or new requirements centered around improving the interface that caused the user to believe there was an error.

The final reason a defect might get rejected is because the defect is no longer relevant.? It has been Overcome By Events (OBE).? This rejection is a favorite of developers, and is usually cited when a consistently reproducible item suddenly stops being reproducible.? This is related to the first type of rejection, but in this case the defect is not as transient as in the first case.? In this case, the defect is a known problem that ceases to occur.? This can happen for a variety of reasons, especially in tightly coupled code.? A large number of these types of rejections is often a sign that development may be getting out of control and that unrelated changes are affecting code in other places.?

Sometimes developers like to “pad their stats” by marking items in cases 1 and 2 as fixed instead of rejecting them.? They like to have these problems show up as being fixed instead of rejected under the belief that since the problem no longer occurs, it was “fixed” by something.? I don’t like this method because it obscures actual changes being made to the code base.? This artificially inflates the number of changes that a given version appears to have had made to it.? Conceivably, if one were unscrupulous, you could create an entire release without doing any work simply by saving all these items in the work queue and then marking them as fixed all at once, giving the appearance of a large number of improvements when the code did not change at all.

However, I am assuming that rejected defects are subject to some sort of review process to ensure the rejections are appropriate.? Generally, this falls to either the testers or the customer, and they simply need to validate the explanation given for the rejection.? If the problem can be reproduced, or if there is continued disagreement, the defect should be re-opened and sent back for further review.? Otherwise, rejection becomes the resting place for all the unpopular changes and requests.

Posted in Best Practices, Development, Process | Comments Off

Doing the right thing, despite your customer.

Monday, March 9th, 2009

It seems to be pretty accepted in the software world the you should not give your customer what they ask for, because they don’t know what they want.

Of course I’m exaggerating slightly – this is all a semantic subtlety that hinges on the idea that you discover what your customers “need”, not what they want.? Unfortunately, this clever wordplay is lost on many developers and every customer (real or imagined) I’ve ever met.? If you are developing commercial software, you certainly have the ability to decide what you will and will not do.? More often than not though, even commercial software has one or two customers that are large enough to demand the type of attention normally reserved for custom software, and if the development is being paid for by a customer, it becomes even more difficult.? So how do you explain to these customers that despite the sheer brilliance of their request, you are actually going to do something else entirely, and by golly they are going to love it even more??

1. Be Honest and Trustworthy

This one should be obvious, but so many people miss it in their desire for profits, success, or simple laziness.? If your customer can’t trust what you say and you can’t be honest with your customers than there really isn’t any point.? No matter what you do, your customers won’t like it, and you will always struggle to understand them.? This does not mean that you must air all your dirty laundry but if you can’t do something because of a mistake in the initial design, admit the mistake and work to correct it.? Your customers want you to improve and be better because it provides direct benefit to them.? Seek out those opportunities and your customers will recognize that and reward it.?

2. Understand Their Request

This is the step that most folks have fixated on.? Understanding the request means more than a simple literal understanding of how to implement the feature exactly as suggested.? Instead, try to understand the problem they are attempting to solve with this feature.? Are they suggesting a new feature because the existing ones are too difficult?? Is there an equivalent way to solve the same problem using the existing feature set?

Of course sometimes the customer simply wants what they want, and it is important to understand that to.? Sometimes a duck is simply a duck.

3. Acknowledge Their Need

This goes hand in hand with step #2.? By taking the time and effort to actually understand their request, you are signaling to them that they are important.? By working with them to help define their need you are acknowledging that they have a legitimate problem that they need solved.

If you don’t care about solving this particular need of the customer, then it will never get solved.? This may be a legitimate outcome.? Sometimes the cost of solving a problem outweigh the gain that could be had, and sometimes the problem is so far outside the bounds of the system that it simply doesn’t make sense.? In these situations, you must rely on the trust you have built up and simply explain this to the customer.? I have even been known to suggest competing alternatives that would address their needs.?

The important thing to remember is that even if you do not want to address this particular problem, it is still a problem for your customer, and they will attempt to solve it no matter what you do.? Ignoring potential solutions because they won’t result in an improvement to your bottom line is a lost opportunity to build up the trust from step #1.?

4. Outline the Cost of the Idea

Since your customers are unlikely to have access to your codebase, they don’t have the information needed to accurately evaluate a given request.? Instead they make assumptions based on their own experiences and needs.? They do not consider everything that needs to be addressed to implement a new feature.? Often times, simply explaining what a feature would cost is enough to dissuade them from a literal interpretation of their request.? It is amazing how flexible a customer can become when faced with a 10 month timeline for a problem they need solved next week.

Of course, this is often abused by the less scrupulous as a customer control mechanism.? If you begin to inflate the cost of ideas you don’t like and deflate the cost of those you do, eventually you will be caught out.? When that happens, the loss of all the goodwill built up by your efforts in step #1 will severely hamper, if not cripple, your future efforts on the project.

5. Suggest Alternatives

Often customers will suggest the first idea that pops into their head.? It is part of your job to suggest alternatives.? Most problems can be addressed in multiple ways, so take some time to consider them.? Figure out the benefits and drawbacks of each approach.? Determine the relative costs of the approaches.? Arm the customer with as much information as possible so they can make the most informed decision possible.? Be careful not to use this step to overwhelm the customer with technical questions and alternatives they don’t care about.? Focus on the problem and how it could be solved, and do not neglect ways to solve the problem using the existing feature set.? Often times the existing features are sufficient, but the customer is unaware of them or doesn’t understand how to use them.

6. Accept the Outcome

Whatever is finally decided, you should accept it.? Holding a grudge because the customer overruled you will simply hurt the quality of your work.? The purpose of these steps is to vet the feature request in such a way that by the time you reach this stage, everyone should understand what is being done and why it is being done.? If you still cannot agree or accept the outcome, you should probably revisit step #3.? It may be possible that the problem is simply something you don’t want to solve.

Posted in Customer, General, Process | Comments Off

Subversion, Command Line, and Redirecting the Standard Output

Tuesday, January 13th, 2009

I'm going to interrupt my estimation tool project in order to comment about an interesting mini-project that I finished last night.? Being a big fan of code reviews, I have recently managed to convince my team at work to require reviews for all code checked into the main repository.? Once I had everyone on board, I used a handy subversion hook provided by our review tool to enforce our new review policy.? This has worked well for the past month or two, until just recently when our project began to move into the pre-release testing phase.

The problem is that we have a large number of test models and the inability to commit new models without a review was beginning to hold up our progress on testing.? So it was requested that I exclude the test model directory from the review requirement.? It sounded pretty simple, so the first thing I tried was to look at the hook provided by our review tool to determine if I could have it exclude files or directories.? Unfortunately, it couldn't.? A quick web search turned up nothing resembling the hook I needed, so this left me with only one basic recourse – write my own.? Besides, it was an interesting diversion from my usual responsibilities.

Since we use subversion, setting up the hook was simple.? Essentially, I edited the pre-commit template to first call my own custom application that would determine if the changes being committed could be excluded from the review requirement.? If they could not be excluded, I called the review hook.? Simple, except I still had to figure out how to find out which files were being changed and figure out how to access that information programmatically.

The first problem was easy – SVNLook was made for just this occurrence.? With it you can discover a wealth of information about a given transaction.? The problem is that it is a command-line application that writes its output to Standard Out.? Of course, I know all about the theory of redirecting the standard output and so forth, but having spent my entire career doing graphical desktop and web-based applications, I had never had the chance to actually attempt it.

Fortunately, C# makes the entire process ridiculously easy:

????? System.Diagnostics.Process svnLook = new System.Diagnostics.Process();

????? svnLook.StartInfo.FileName = "svnlook.exe";

????? svnLook.StartInfo.UseShellExecute = false;

????? svnLook.StartInfo.RedirectStandardOutput = true;

????? svnLook.Start();

????? string output = svnLook.StandardOutput.ReadToEnd();

That's all there is to it.? The local variable “output” holds the entire output of the svnlook program.? The ReadToEnd method is useful because it blocks until the stream is closed, allowing you to block until the execution is complete.

There are other ways to do this (such as using the System.Console.SetOut method), but this was by far the easiest method I found.? Note that you must set the UseShellExecute property to false if you redirect the standard output, otherwise you will get an invalid operation exception (according to the documentation).

All things told, it was a refreshing change of pace for me, and a fun way to spend an hour or so at work.? I may even write and post a full-featured hook one of these days (of course, I'd probably use a different language to make it as portable as possible). In the meantime, I guess I should get back to my own mini-project…

Posted in C#, Development, Process, Source Control, Subversion | Comments Off

Commit Wars

Monday, December 15th, 2008

Quick show of hands…? How many of you have gotten into a commit war?? I think we all know what I'm talking about – some projects even have rules that exist solely to prevent commit wars from occurring (they generally take the form of limiting the frequency of commits for a given file).? These things generally start out innocently.? Some well-intentioned developer opens up a file and decides that a couple of variable names aren't as clear as they should be.? So he renames them.? Since he is already in there, he reformats a couple of lines of code to meet his personal preferences and commits the file.

The original developer is then informed of a defect in her code.? Nothing serious – a simple mistake in the index being used.? She opens the file and is confused by the naming scheme – it seems the new names conflicted with the index name, and resulted in the defect.? In addition, the code was reformatted in an odd manner.? A firm believer in “the one true brace style”, and already on edge for taking the blame for a defect introduced by someone else, she reverts back to the original check-in and reformats the entire file to fit her preference.

The first developer notes the change in the file and the loss of all his insightful and very carefully chosen variables names.? Now furious, he undoes the commit and sends a nasty note to the original developer instructing her to leave his changes alone.

Not one to take something like this lying down, the original developer begins to nit all of his code in reviews.? When he doesn't fix her complaints, she comes in afterwards and changes it anyway.

Preventing Escalation

In hind sight, commit wars are remarkably simple to prevent.? Simply by mandating a standard code format you can avoid the majority of commit wars.? In addition, it is critical to ensure that your reviews focus on real problems and not on simply stylistic or implementation preferences by the reviewer.?

Of course, the most crucial element to preventing a commit war is to create an environment and team where there is enough respect that the developers do not feel the need to look over each other's shoulders.? If no one can trust the rest of the team to do things right, then every little change is subject to enhanced scrutiny, and possible retaliation.

Finally, if every change requires a peer review, most of these problems will never make it past the review.? Too often, teams believe that only the major changes should be reviewed.? But by putting a lightweight review process in place to review every code change, you can avoid the sorts of unnecessary changes that create commit wars.? Of course, if you actually expect to get things one, you had better make sure that your reviews are quick and painless…

Posted in Best Practices, Code Reviews, Coding Style, Development, Process, Source Control | Comments Off

Solutions are not Requirements

Thursday, December 11th, 2008

Yesterday, I described a particularly prolonged form of Project Suicide.? In my description, I touched on a point that I feel should be expanded upon.? Essentially the type of project suicide that I described revolves around the difference between how a client views a requirement and how most software engineers view requirements.

Clients inevitably frame requirements in terms of a problem:?

I need to be able to add 2 numbers.

Engineers inevitably frame requirements in terms of the solution:

We will provide 2 text entry dialogs.? In each dialog, the user will be able to enter a single number.? We will provide a single button called 'Add', which when pressed will add the 2 numbers in each dialog and display the result in a third text dialog.? The third text dialog will not be editable.

These are 2 very different beasts.? Most engineers will look at the first statement and immediately begin complaining about the lack of specifics.?

How do you add the numbers??
Where do you enter them??
Where do you display the result?

Eventually they will derive out the second example, and set it down as the official requirement, completely losing the fact that all the customer wanted was a way to add 2 numbers.? Everything else is just implementation details of a particular solution.

The problem with treating solutions as requirements is that you remove your ability to address the core problem.? You cannot anticipate future problems, because you are essentially reading from a script that tells you exactly what to implement.? That is not to say that you shouldn't explicitly detail your solution.? It simply means that your solution is not the requirement, it is simply a mechanism to satisfy the requirement.?

Solutions are not requirements.

Don't even bother phrasing a solution as a requirement.? Understand the solution and why it was suggested.? Document the underlying problem that is solved by the solution.? That is your requirement.?

Detail the solution in a design document for the requirement.? That way, no one will be tempted to treat it as gospel.? Designs are changed, modified, even thrown out, and people do not bat an eyelid.? But attempt to change a requirement, and everyone gets upset.

The other advantage of avoiding the temptation to treat solutions as requirements and treating them as designs instead is that on quality development teams most designs will go through some sort of vetting process.? This could be a formal sit down review, or a quick peer collaboration to make sure that nothing is missed.? Requirements rarely achieve even that level of scrutiny simply because by their nature there is little to debate.? They are coming in from the customer.? If the customer wants his spreadsheet software to also control his coffee pot, then at the end of the day the spreadsheet software will control his coffee pot.? This will happen one way or another, so there is little point in reviewing it internally.

Dealing with these vague requirements with multiple possible solutions will initially be a little bit overwhelming if you are used to the tightly constrained world of solutions as requirements.? But with time and effort I think you will find that the flexibility it offers you to address the core problem, not simply implement an attempted solution, will result in a much more satisfying, successful development cycle, and a happier client base to boot.

Posted in Best Practices, Customer, Development, Process | Comments Off

Project Suicide

Wednesday, December 10th, 2008

Have you ever had an argument with a client where you resorted to telling them they got exactly what they asked for?? Out of all the forms of project suicide I have seen, it astounds me that not only is this still common, but it is believed to be a legitimate response to customer's complaints.

The problem seems to stem from this mindset of software engineers to think in terms of solutions – not in terms of problems.? Customers and clients tend to think in terms of problems – they may suggest solutions, but they are limited by the current behavior and appearance of their toolset.? We as engineers like to take these suggested solutions and set them in stone.? We want to write them down and examine their every detail.? We try to force the customer to commit to design decisions so that we can move forward with our implementation.? The customer, who doesn't think in terms of solutions, simply goes with the flow, gratified that his or her suggestion was so useful.

This is why, after spending weeks ironing out the exact layout and specification of the entire feature and then an additional month implementing it, followed by weeks of testing, when the customer finally gets their hands on the release with the shiny new solution, she hates it.? So she comes yelling and screaming to the team demanding you correct your mistake, but of course, the team is protected.? Every step of the way, you've gotten the customer's signature that says this is what she wants.? It is part of the process we are all taught and follow blindly – when in doubt, ask.?

So when she comes yelling and screaming, you pull out all of those approvals and proceed to explain how you did exactly what she wanted.? As a matter of fact, you prove to her that you gave her more than she wanted.? When you are finished, all she can do is tell you to fix it and wait for the next release to see if you have a solution to her real problem.

That is project suicide.

From this moment on, the working environment is changed.? Everything must be documented, approvals must be given, and all decisions must be effectively supported.? Once this line has been crossed, you are no longer working with your clients as a team, you are working as an obstacle that must be dealt with.? The trust has been broken.? The client hired you to solve a problem, not satisfy a process.? Once you are in the business of satisfying a process, your margin for error has disappeared, because the process specifies exactly what is expected, and no allowances can be made.

It always seemed much better to accept the short term consequences of an honest mistake than the long term consequences of a refusal to accept it.

Posted in Best Practices, Customer, Process | Comments Off

What I Want in a Process

Saturday, November 29th, 2008

I have been a part of a number of different development styles and processes.? Some were well suited for the task at hand, and others were broad strokes that were applied in the sake of either consistency or simplicity.? Still others were simply band-aids applied to address some disaster in the projects past.?

I much prefer an organization that has taken the time and effort to find and fit a development process that matches their development style as opposed to mandating down from on high the one true process.? Of course, these projects are often in conflict with the so-called “process-nazis”.?

I have encountered 2 types of “process-nazis”.? The first type is usually a new employee or project member that comes from a team or school that used the “one true process”.? This person will inevitably attempt to discredit any guidelines that are different from the process they are accustomed to simply because of their difference.

The second type is usually a believer.? These are folks who are always looking for a quick-fix to their problems.? They usually latch onto the current “in” process and promote that for every situation.? They are identified by their blind devotion to the strict tenants of whichever process they are promoting.? No impurities are allowed to pollute their vision.

I tend to avoid these “process-nazis” like the plague.? I have found that the longer I spend with them, the more sense they begin to make, until slowly the reality that each project is different begins to fade into a blurry thought that is easily swept away.? This inevitably ends in disaster.? If you have ever shipped a release that failed to even install you will understand the dangers that occur when you blindly apply processes without respect to the actual needs of the software.

This happened to me on my last project.? Our build process was essentially brought forward from the build engineer's previous project.? It emphasized speed and assumed a large QA component along with rapid turn-around? The problem was that the new project needed to emphasize perfection, because once the software was released, our customers would not be able to receive another version for weeks.? So the build process that worked perfectly when you could simply rebuild a new release at any time, failed spectacularly when the rebuild process could not occur until much later.

In hind-sight, this failure surprised no one, but at the time, no one thought to ask questions about the process.? We were all more concerned about simply having a process.? These days, I am far less worried about what the process is as opposed to why the process is the way it is.? When I encounter a new process, I try to determine why the guidelines exist.? I ask the team why they do things the way they do things.? If I see an opportunity to suggest an improvement, I will attempt to determine why the current requirements exist, and, depending on the response, I may suggest the improvement.

Chances are that if the team has already taken the time to tailor their process, they will be receptive to new ideas, especially if you take the time to determine why things are the way they are.

If the change is approached simply as “this isn't mine and so isn't right”, you will probably not make much headway, but by being pragmatic about the benefits you can likely make some substantial improvements.? Of course, most of the time, when I find a team that is interested in customizing their process, they generally have compelling reasons for each step in the process, and when they don't they are always eager to re-evaluate and re-visit any and every step.

This is what I want.

Posted in Development, Process | Comments Off

How Long Will It Take Me?

Friday, November 28th, 2008

So now that you know what do, the next question that comes up is how long will it take me?? Estimating software is notoriously difficult – a lot of people don’t think it is worthwhile because developers are often so bad at it that their estimates don’t seem to have any connection to reality.? I think this is a mistake.? The only way to get better at estimating software is to estimate how long it will take to implement different features.? Eventually, with practice and feedback, you should learn to get fairly accurate with your estimates.

Estimating tasks offers diminishing returns, which is likely why it has such a bad reputation among developers.? There is little point in attempting to accurately estimate a feature that is months away from being started.? By the time you actually begin work on that item, the environment is likely to be different enough that the original estimate is useless.?

Of course, you want some idea of how long everything will take, so I recommend running through every item and putting together a rough order-of-magnitude type estimate for each one.? Don’t spend more than a minute or two thinking about each item, the point is to just give an idea of the total length of the schedule, not to accurately estimate it.? I will usually just split the items into one of three categories:

Long (1+ weeks)
Medium (1+ days)
Short (1+ hours)

Once I pick a category, I can usually take a stab at the number of hours, days, or weeks.? It is often way off base, but with enough items, on average I have found it to be as accurate as any other schedule, and I don’t have to worry about specifics just yet.

Of course, now I have an idea of my overall schedule, but I need some more accurate short-term forecasting.? To do this, I select about 4 weeks worth of tasks using the rough order-of-magnitude estimates.? Usually, the items in the “Short” category are fairly accurate already, so I focus on the “Medium” and “Long” categories.? To estimate these items, I break them up into smaller and smaller tasks until I only need to provide “Short” estimates for any given task.? Add up the length of all the decomposed tasks, and you have a fairly accurate estimate of the overall task.

As an added benefit, this will force you to think of what a given task requires before you begin implementation, hopefully helping improve and refine your design.

Finally, this is an on-going process.? Every few weeks, you should revisit your tasks and re-estimate their magnitude.? As you pull items off the queue, you should be decomposing the tasks and providing accurate estimates.? This should provide you with an accurate short term (3-4 weeks) schedule, and a fuzzier long term schedule that will progressively get more accurate as your cycle progresses and you begin to understand the domain and intricacies of your design.

Posted in Best Practices, Development, Process | Comments Off

What Do I Do Now?

Friday, November 28th, 2008

Have you ever been working on a project and realized that your pile of TODO’s has far outpaced your ability to address them?? This is a common problem for projects and it can lead to all sorts of problems.? So how can you get this beast back under control and make sure that you are making progress on the things that are truly important?

Well, the first thing you have to do is make sure that all of your tasks are rated and prioritized.? I always rate items by severity first – if it is not obviously a defect I mark it as a feature and set it aside.? This first rating is always done as fast as possible.? I usually just go with my first impression – the key here is not to get it correct, you can always change it later, but rather to separate out the items that do not matter so I can focus my efforts on the items with the largest impact on the customer.?

Once I have all my defects rated, if I have any urgent items, I immediately begin work fixing those items.? The rest of the tasks can wait, but our customers should not.

Now that I have rated my features and defects, it is time to prioritize.? I usually prioritize each item based on its risk. Some things are easy to implement and others are difficult. All things being equal, you should always do the riskiest items first when your schedule is more flexible to meet any unanticipated problems. Wait until the end of your cycle, and Murphy’s law will strike you down.

With a small team, I would just send the list of items around and ask everyone to mark it if they consider it a risky or difficult feature to implement. A simple yes or no is sufficient.? Add up all the yes’es and you have your “risk assessment”, with the highest scoring items being the riskiest.

Alternatively, if you have easy access to your customer, ask them to rate the “risk” associated with each feature.? In this case, risk refers to the worst-case scenario of not having the feature or fixing the defect? – if not having something would be annoying, it is not risky. If not having the feature or fix would result in them not using your product, it is high-risk.

At this point, you have everything you need to begin assigning items from your TODO list.? Sort the items by priority and start assigning the items with the highest priority.? If items have the same priority, use the severity as a tie-breaker.? Every week or two, revisit the list and repeat this process to re-evaluate and prioritize new items.?

Using this process you can rapidly focus on the important items and maintain that focus throughout your development cycle.

Posted in Best Practices, Process | Comments Off

The Danger of Bad Tests

Monday, September 22nd, 2008

Bad tests are worse than no tests at all.? It is a simple fact that seems to go unstated or unnoticed on most projects.? Someone decides that there must be a certain number of tests (or a certain percentage of code must be tested), and so the developers (who are already overworked and not suited to write good tests anyway) throw together a set of tests that ostensibly navigates the required functionality.

I am convinced that this seemingly unwritten rule is the reason that developers avoid writing tests.? We have all been forced to spend hours wading through some test failure only to discover that whoever wrote it had obviously spent more time commenting the code than actually testing what was needed.? Take, for example, my experience today.? I had to fix a defect which allowed bad data to be written to our database, so I wrote some code to validate the data before it was written.? It was a pretty simple fix as the code was already written, we had just missed an entry point.? Satisfied that all was well, I checked in the code, and promptly broke the build.?

Now, to be honest, this is a fairly common occurrence with me.? We use an automated build system that detects check-ins and schedules builds immediately.? Unfortunately, I have a habit of checking the differences in all my files before checking them in and including comments on the check-in to indicate what I had intended to do.? This naturally takes some time, so I frequently end up spanning the automated build process.? By the time I have completed my check-in, the build has already started, and will usually fail because of a dependency on an item I have not checked in yet.? This has had the unfortunate side-effect of training me to ignore build errors.? Its a bad habit, but the system is setup such that I expect a failure during my check-ins, so I wait for the second build before I worry about it.

In this case, the error was due to a test failing.? Originally I had assumed the test was failing due to my slow check-in process.? The test was in a related area, but was testing our handling of duplicates.? After the second build failed, I realized that the test was failing because of my change.? After looking at the test, there was no reason for it to be failing due to the changes I had made.? It claimed to be testing how the system saved duplicates – perfectly valid data in our case.? I checked the validation code and sure enough, duplicates were not allowed.? Intrigued, I stepped into the debugger to see what was going on.? Imagine my surprise when the validation failed!

Now I was in trouble.? I had been relying on the existing validation code and if it was flagging valid data incorrectly, I would have to go back through all the validation rules and make sure it was written correctly.? Not a task I was looking forward to.? Fortunately, in addition to my bad habit of not trusting the build, I have also learned not to trust our automated tests.? They were written with the goal of meeting a certain percent code-coverage, and have been shown to be fairly useless in terms of correctly evaluating the behavior of the system.? So I started looking at the actual data that was being written, and as it turns out, the data was in fact invalid.? The test, which was attempting to ensure that valid data could be written to the database, was writing invalid data.? Not only that, but the test had been happily running and passing for over a year.?

False positives such as this cause two problems.? The first problem was that the system was telling everyone that there was no defect in the way in which we handled duplicate data.? In this case there was in fact an underlying problem that was being hidden by our use of invalid data.? The problem had already been discovered in a released version and had been fixed, but this test never indicated that the problem existed, despite that being its reason for existence.? The second problem is that it falsely indicated an error when there was none, causing a couple of hours of development time to be lost in a very tight schedule.? Lapses such as this cause the situation that we are currently in on this project, where we have simply turned off a number of our automated tests because they are failing and no one can explain why.?

Good tests are hard.? They are hard to write, and they are hard to automate.? Unfortunately, the dangers of writing bad tests far outweigh the cost of writing a good test, so most developers simply ignore writing tests unless they are forced to, and then development slows down as everyone spends their time chasing imaginary bugs.

Posted in Process, Testing | Comments Off

Code In Review