All stick, no carrot

The Factory School of software testing follows the ideology that the software world can be sliced up into very neat pieces or categories. All traces of the messy affects humanity can have on a project are removed, you're left with "metrics". Factory is an attempt to simplify or platonify (as Taleb would put it) your world for a variety of reasons. The factory school depends on things like test case pass fail counts, bug count metrics, test estimation, and very reproducible work. Numbers numbers numbers. No people.


My conclusion here, is that this style of test management leads to an overly heavy handed approach to management. I'll illustrate this through some experiences I've had while working for other companies as well as an anecdote from the linkedin forums.


At one company I worked at (for a very short stint) whenever a customer found and problem, the tester who worked on that feature was required to sign a sort of incident report which explained the issue and was an admission of guilt that we tested the feature and apparently didn't do a good job. Interesting thing here is that detailed test cases were required as well as test case reviews. Factory ideology and a culture of blame, together at last!


At another company testers were made to do upfront estimation along with the programmers. We had to estimate down to the half hour the amount of time it would take to write and execute test cases. These were two separate estimations. Estimations were consistently wrong of course (except in come cases where fake testing was occurring) and there were consequences ranging from talks about why that particular person can't estimate correctly to disciplinary actions. Note that there was no reward for doing good estimation even if it were possible.


There was a discussion thread on linkedin talking about the nature of how people run detailed test scripts. The question was whether people run them step by step or based on memory. A qa manager mentioned that she would create a few "rigged" test cases that were designed to fail. In the case that someone ran the rigged test case and it didn't get failed there would be disciplinary action. If that occurred a few times, she would have that person removed from her team somehow.


The common thread I see here is harsh discipline when messy humanity is noticed but also a distinct lack of reward when the desired procedure is followed. This type of environment restricts the serendipity often needed for real innovative and interesting testing. Things like pass fail numbers for test cases or bug counts or lines of code or number of tests don't tell a story by them selves. A story needs context to make sense. A story about software needs a human to make sense.


Love to hear your thoughts and experiences on this!

Safety language overview

The topic of safety language has popped up twice lately, once at the CAST 2012 critical thinking tutorial with Michael Bolton and once on Twitter in a tweet from Ben Simo. Safety language is also known as epistemic modality. Epistemic modality translates into modes of knowledge. The word epistemic comes from the philosophy of knowledge, Epistemology.

Anyway, I'm no academic, so here's the practical stuff.

Safety language is a way to vocalize that you are currently thinking critically about something. It is the verbal suspension of certainty. Communicating in this manner shows that you have separated an observation from an inference or judgement about that observation. It is a precise statement that says you do not currently know and need more evidence.

Examples:
could
may
might
should
seems
perhaps
possibly

Think about the difference between these two phrases.
It is going to rain today.
It looks like it might rain today.

To me, the different modal verbs show distinctly different stages of critical thinking. Namely, certainty and uncertainty or suspension or certainty. These examples are pretty simple but they can be applied to observations made when testing software. When performing black box testing, you have a fairly limited scope of what you can observe. You have the UI you are interacting with and possibly some log files and a database (though that may not longer be black box testing at that point...?). At any rate, you are not looking at the code so your understanding of what it actually happening is limited.

Due to the lack of knowledge about what is really happening, when communicating observations it may be wise to be precise about what you do not know. Love to hear your thoughts and experiences in using safety language.

This is soloist Demondrae Thurman playing Slavish Fantasie.