Archive for August, 2008

Ask the Client

Sunday, August 24th, 2008

Jim has a problem.? He's been tasked to write this new feature which will require him to store a good amount of data on disk for later use.? Speed and scalability are not critical, as the client doesn't believe the storage requirements will grow beyond current estimates.? Suzie, of course, believes that Jim should store all the data in a database because it will make things better down the road if and when things change.? Bill is adamant that this all overkill and he sees no reason to store the data in anything other than a simple text file, thank you very much.? Why fix a problem that doesn't exist yet?? Karen doesn't really care what Jim does, she just wants Jim to hurry up and finish the interface so she can start working on the analyzer code she's been tasked with.

After several meetings attempting to hash out a workable compromise, everyone finally agrees on a resolution – ask the client what she wants.? Whatever the client picks, that is what Jim will implement.? The developers all believe they have reached a perfect solution – not only will they finally get an answer, but they are involving the client in important decisions.

Sometimes asking the client can be the right decision.? Anytime you are implementing a feature that behaves differently (no matter how slightly) from the expected behavior in the given domain, you should always present your reasoning to the client and work with them to either fix the underlying problem or bring the feature in line with the expectations of the users.

On the other hand, “implementation details” are rarely of interest to clients (there are some exceptions, but in my experience they are the proverbial proof to the rule).? I am well known in some circles for my habit of waving my hand and dismissing a discussion as “implementation details” whenever a member of the team would attempt to engage the client in a discussion of underlying technology.? If the client has the knowledge and ability to make the hard technical decisions that your team cannot, why would they want to hire your team in the first place?? Instead, they are trusting you to provide them with the appropriate technological foundation to solve their target problem.

The other reason I distrust developers who want to involve the client in a low-level technical discussion is that inevitably they seem to want to use the existing state of the system as proof that feature A can't be done and should be scrapped.

Ed: Well, Steve, you see, we can't really do an unlimited undo/redo because we are actually running in separate processes, and in order to do it properly we'd have to correctly synchronize all the inter-process data calls and make sure that the interfaces are all correctly locked and thread-safe.? Unfortunately that will pretty negatively impact your performance.? On the other hand, we could just implement a message-passing system that will handle all the synchronization for us, but then we'd need to guarantee that the messages are all processed in order, and we'd still need to address the issue of whether or not we could undo the messages and still maintain the correct internal state across both processes.? Of course, we could always just combine all our processes and do away with the whole synchronization issue, but then you wouldn't bee able to scale up if the computing needs outgrow the abilities of a single machine.? So what do you think Steve?? Should we implement a thread-safe API, a message passing system, or return to a single-process implementation?

Steve:? Ummm, well that sounds hard, so maybe we should leave everything as it is…

Ed:? Good choice Steve.? Good choice…

These types of discussions are just another way to confuse the client into doing what you want, as opposed to getting at what the client wants.? Inevitably the more the client is involved in these types of discussions, the less likely the end result will be what the client needs or wants.

Please notice that I am not advocating keeping the customer in the dark.? By all means, provide them with technical information about the way the system works.? But always phrase the discussion in terms of the abilities and benefits that the different technologies and approaches provide to the users.? The client probably doesn't care whether you implement a polling system to pull data from the server or a push system to send data to the clients, but they do care about the differences in user experiences that result from those differences.? Stick to those discussions and provide the underlying implementation details only as necessary to illustrate your points.? In the end, you will find that the client will be much happier with the outcome, regardless of the particular decisions that may get made under the hood.

Posted in Customer, Development | Comments Off

Working from Home

Thursday, August 21st, 2008

Recently there has been some chatter about a ComputerWorld article asking whether IT workers should be allowed to telecommute:

Get tough on telecommuting: 6 questions to ask before you say yes

While I get the feeling from the article that it is more concerned with System Administrators as opposed to Developers, this article was interesting as I have just started working from home, and encountered many of the questions that were asked in the article.?

Right now, I am in a trial period, meaning that I still go into the office once or twice a week to evaluate the setup, but starting in two weeks I will be working from home full time.? To be honest, I do not particularly want to telecommute, but I am moving to the West Coast and the company I work for has asked that I remain on my current project (located on the East Coast) instead of taking another position.?

I am a firm believer in the value of having a team that is all in a single location and easily accessible.? It is much easier to walk down the hall and spend 15 minutes hashing out a difficult technical issue face to face, than to try to hold the same conversation via the phone or chat.? I tend to be a visual person, and so I like to see ideas sketched out or look at the code as it is discussed.? i even prefer having the face-to-face discussion because it allows me to gauge reactions and gain insight into what someone really thinks of an issue as opposed to what they are willing to say professionally.?

On the other hand, a common problem with telecommuting, measuring productivity, is not really an issue for a developer in terms of the difference between being local and being remote.? I'm still fixing defects and implementing new features.? I have the same deadlines and the same requirements as I had before.? It also helps that I have been working with this team for several months now and have just helped them get through a successful release of our software, so my manager knows me and trusts that I can get the job done.? He knows what to expect.

The one looming unresolved problem is that I do not foresee telecommuting being a long-term solution, but given my relocation, telecommuting is the only option that will allow me to meet both my personal goals and the companies goals.? I have worked out a tentative plan to transition off the project come this winter, but as with all things, it will depend on what is best for everyone involved.

Posted in General | Comments Off

Defect Severity

Wednesday, August 20th, 2008

I always find it annoying as a developer when I am assigned a “High” severity
defect only to discover that someone would like to have a label changed, or
thinks that the page layout is confusing or any number of valid defects that
simply don't merit the attention that a high-severity item merits.? I've found
testers (and developers) who use severity as a means to push a fix through the
process.? Others who seemingly assign severity based upon the severity relative
to the other defects he or she has found (“Hmm, this label is *really*
confusing, probably about twice as confusing as the last one so I'd better make
this a high severity instead of a medium severity…”).? This really shouldn't
be what severity is for, that is why most defect-tracking systems also list a
priority (there are plenty of low severity items that for one reason or another
should be done right away).

Priority and severity are often used interchangeably, but have different
meanings.? The priority is usually meant as the relative importance of a
defect.? This is usually a factor of the severity, number of complaints,
likelihood of occurring and ease of implementation.? The priority is almost
always subjective, and usually the customer has the final say.? If your end user
says you have to change the name of the “File” menu to say “Data”, inevitably in
your next release you will have a “Data” menu item.

Severity, on the other hand, tends to reflect the negative impact of the bug
on the system.? A high severity defect has a highly negative impact on the
system or end user, while a low severity defect has a minor impact on the system
or end user.? Severity, like priority, can be subjective, but I've found that a
fairly simple set of guidelines dramatically reduces the number of poorly
classified defects.? Essentially the system breaks down like this:

  • Low Severity
    • Annoying or otherwise confusing to the end user.? The behavior of the system
      is “correct” simply not clear or as expected.
  • Medium Severity
    • The operation of the system is not correct.? There is a reasonable
      work-around that allows the end user to accomplish the desired
      functionality.
  • High Severity
    • The operation of the system is not correct, and there is no
      work-around.
  • Urgent
    • The defect causes a deployed system to crash.

These fairly simple guidelines allows most everyone to operate from the same
page.? Now when I see a high severity defect, I have a fairly good idea of what
to expect, and when the urgent defect comes across my desk, I know I'd better
stop what I'm doing and investigate right away.

?

Posted in Process, Testing | Comments Off

Testing in Debug Mode

Thursday, August 14th, 2008

Here we go again.? You might be able to tell given the tone of my posts here that I am in the middle of attempting to put a new release out the door.? We are almost ready to send it out, but we keep running into roadblocks.? The latest problem had to do with, you guessed it, testing in debug mode.?

Essentially, the problem went something like this.? The testers would fire up the latest build of our software and start testing it.? Usually about 30 minutes into a testing session, they would get a random appearing error and the software would crash.? This behavior was not consistent.? Sometimes the software would work fine, other times it wouldn't take longer than a few minutes before they would get the error.? They wrote up all the information they had and sent it back to the developers.? The developers fired up the code and waited.? And waited.? And kept waiting, but everything worked as designed.

Thinking that perhaps it was a machine problem, the code was moved to the test bed and the software fired up with a debugger attached.? Still no problem.? The developers declared the problem a transient issue and the defect was triaged from the release.? Test cleaned the machine and installed the next build, and sure enough, they encountered the exact same behavior.

Eventually, we tracked our problem to a quirk in how .NET compiles debug and release modes.? In debug mode, .NET will artificially extend the life of all variables until they fall out of scope.? This ensures that any time a breakpoint or other event occurs that cause the debugger to break into the code, the developer can inspect the value of all variables that are currently in scope.? In release mode (when run outside the debugger), .NET makes no such guarantees.? In our particular case, we were not particularly careful with how we were disposing of some unmanaged objects in relation to some threading we were doing and could occasionally wind up in a situation where they could get disposed of before we were ready to dispose of them.? Of course, this never occurred in debug mode, because the lifetimes of all the objects were artificially extended, but the second the testers installed a release version, it became all too painfully obvious.

Developers love to test in debug mode.? They get to do all sorts of nice things like break the code, watch the debug statements flyby, inspect memory and variables on the fly, all great and useful stuff.? Testers hate it when things are tested in debug mode.? its not what the end user runs, it doesn't behave the same way, its not installed on a clean machine, all valid and accurate points.? So when it comes time to test, the testers all carefully setup a clean machine, install the latest version and test as if they were users.? The developers meanwhile update to the latest version of the code, hit compile and are up and running before the testers have even created their user accounts.? The developers all race through their tests, declare the version “bug-free” and wait for the inevitable agreement from the testers.

The testers meanwhile are busy complaining about how the software doesn't work if the latest version of .NET isn't installed, or how it will crash if it is installed to a directory other than the default directory, and so on.? This is why every version should be thoroughly tested in an environment as close to the production environment as possible.? No compilers, no debuggers and certainly, no debug versions.? Let the developers run their tests in debug mode.? It is far more productive, especially early on or when there are rapid development cycles, but always set aside some time for release mode testing.? You never know what might fall out.

Posted in Testing | Comments Off

Controlling Installers

Wednesday, August 13th, 2008

This is a topic that I don't really know what common practice is but I seem
to consistently encounter resistance when I suggest it.? I am a firm believer
that any and all installers created should be controlled and versioned the same
as any other artifact of development.? That means it goes in source control.? To
me, it just makes sense – that way at any point in time you can go back and grab
the exact bits that were built and (possibly) delivered.? There is no substitute
for having the exact copy when an obscure bug report filters in.?

Just recently we delivered a new version of our software to testing.? Our
group has a very limited testing staff, and so it ended up being a day or two
before they were able to install and begin testing the software.? Within 15
minutes of installing the software, we began to get all sorts of bug reports.?
Nothing was working.? Of course during this time, the development staff had
continued pressing on to meet their deadlines, and had not encountered any of
the issues being encountered by the test staff.? Dutifully we reverted back to
the version of the software being run by test in an attempt to verify what was
occurring.? Unfortunately, we don't control copies of our installers, so we had
to rebuild them.? Once that was done, we installed and ran a few tests, but
everything seemed fine.? We showed test what was happening and they agreed that
everything seemed OK.? We decided to re-install the software on our test boxes.?
We installed the software and sure enough, all the problems went away.? A
miracle, right?? Or maybe someone secretly fixed all the problems and delivered
them and that's what was built…

In general, there seems to be two lines of resistance to including the
installer in source control.? The first is the belief that source control
doesn't handle binaries well.? This may have been true many years ago, but any
mature source control system should handle binary files just as easily as text
files.? When people claim that is the case I immediately suggest they upgrade
their system.? The second line of resistance seems to be that since the build is
deterministic and completely automated, it will always produce the exact same
result.? In theory this is true, but in practice there are far too many
variables to rely on a bit-for-bit copy every time.

For example, as it turns out, in our case the problem was due to a licensing
issue.? We had received a new runtime license for some software we are using,
and had forgotten to put it on the build machine.? The build incorporated the
old license (which has expired), and so after it was installed and test began to
run it, the software was unable to retrieve a valid license, and failed.? Of
course, by this time, we had updated the license on the build machine, so when
we attempted to reproduce the issue by rebuilding our software, we built the
correct license into the software and viola! problem solved.

So, it is not enough to revert the source code back to the original state and
have an automated build, you really should be reverting the entire build machine
back to the state it was in when the software was built.? That should guarantee
that you reliably build the exact same bits as when it was originally built.?
But then again, if you are going to go to that much trouble, why not simply
control the installers themselves?

?

Posted in Source Control, Testing | Comments Off

Testing for the Sake of Testing

Thursday, August 7th, 2008

So there you are, its the day before the next version of your application is scheduled to be released, and Joe discovers a bug.  After looking at it, you verify that sure enough, if you attempt to open a file that is already open, the software will use the data in memory instead of loading it back from disk.  The defect is not a show stopper, but it would be nice to fix and so Sue fires up her IDE and before you know it, she’s fixed the problem.  With the annoyance averted, the software is wrapped up and shipped out.  Everyone pats themselves on the back and starts planning the next version.

About a week later the customers start calling.  It seems that every time they open any new file, all their other open files are closed and all their data is lost.  They are understandably not happy, your boss is not happy, and everyone wants to know how, with all the testing that was done, such an obvious bug could have gone unnoticed.  Of course, the problem is that the bug didn’t exist when the software was tested, instead it was introduced accidentally and quite innocently because Sue didn’t have time to analyze all the entry paths to the method she was modifying.  Management demands that new processes are implemented to prevent this from ever happening again, and a whole new set of draconian procedures are implemented in the hopes that there will not be a next time.

Most of us have been in this situation at some point in time.  The details may change – sometimes the defect is huge and crashes the application and simply must be fixed, and other times its not even a defect, just something somebody didn’t like.  Unfortunately, as developers, when we encounter a problem, we feel a strong inclination to just “fix it” and sooner or later we give in to the temptation.  Just as inevitably though, when we do, Murphy rears up and bites us.

After spending so much time working on a piece of software, it is often difficult to accept that there are problems.  The first reaction is to always fix this “one last thing”, or tweak “one little setting” to make it better.  This is perfectly natural, but it is most often wrong.  Every line of code we write has an effect on the software we are modifying.  That is kind of the point of writing the code.  Sometimes the change is simple and the effects can be known completely (I changed the name of a local variable, but left its meaning the same).  Other times, the end result may be more complicated (I changed the return value from an integer error code to a boolean because I only saw a success or failure error code).  As developers, we tend to classify our changes as the former – if we think there are unintended consequences, we will continue to work until we have tracked them all down.  Testers, on the other hand, tend to view all changes as the latter and want to start all their testing over every time a new change is made.

Of course, a happy medium needs to be met.  Early on in the development cycle, the point of testing is to uncover problems for the developers to fix.  The job of testing is to make sure that things pretty much work the way they are supposed to.  As the release date approaches though, the purpose of testing changes.  It is no longer a general search for problems.  It becomes a very targeted investigation of the characteristics of the software.  How does it behave under load?  What happens when I press this button with this window open?  Does the system allow me to enter in a string where it should only allow a number?  The difference is subtle, but important.  Testers are no longer uncovering problems for the developers to fix, but are instead looking for problems the end-user may encounter.  Their focus has changed. 

A lot of people, when faced with this situation, want to know why the developers don’t simply fix the problems as they come up.  We shouldn’t be testing just for the sake of testing, or if testing has found them, then they should be fixed, the argument goes.   However, it is almost always better to know what problems you have up front and solve them in a controlled and consistent manner, as opposed to having to rush a fix out the door and not properly testing it.  It is very difficult to make a successful code change when faced with increasing schedule pressure.  Most defects are not worth the risk. 

In the end, don’t try to tempt Murphy for the sake of wanting to fix everything.  Objectively evaluate all your defects, and if you can’t justify the risk or you can’t justify delaying the release to properly test and fix it, let it go.  Fix it in the next version.  Put out a patch after you have enough time to properly address it.  Shorten your release cycle and ship a new version quicker.  Or, if it is important enough, delay the current version and do it right.  Test for the sake of testing.

It will work out much better in the long run.

Posted in Testing | Comments Off

Hello and Welcome!

Wednesday, August 6th, 2008

Hello and welcome to my little neck of the woods.

My name is Brian, and I am a software engineer.? I have been fortunate to work for a number of great firms and have been given the opportunity to work in a variety of roles from developer to tester, from architect to technical lead and was even given a short stint as a project manager (not sure who thought that was a good idea).?? I've worked on most aspects of a project's life-cycle, from advanced proof of concept projects all the way through to end-of-life maintenance (not my favorite).? I've worked on commercial websites, contract software, and even spent a short period of time as a consultant. Along the way I've collected observations, thoughts and opinions on a wide variety of things relating to development.? This site is just a collection of my thoughts on these things I've encountered and what I think I think about them.? I reserve the right to change my mind, say something stupid and hopefully continue to learn new things.
Enjoy!

Brian

Posted in General | Comments Off