There have been a few instances in my work life where I have been vehemently against something, only to later do a 180 and become a big fan, after finally figuring out how wrong I was.
One example that comes to mind was in my days at Digital Equipment Corporation in the Alpha Workstations development group in the early 1990s. This was very much a disruptive time and place to be within the company. We were a brand new division of Digital, and we were pioneering a brand new CAD process for creating our brand new products. And unlike previous development teams at DEC that all used internally-developed CAD tools, we were using third-party, Unix-based tools.
It was my first exposure to Unix, as I and many of the other engineers there had spent almost all of our time using DEC’s proprietary operating system, VMS. Since the company was pretty much counting on the hardware engineers to be able to navigate this new operating system without impacting the development schedule, they sprung for an outside consultant to come in and teach us how to migrate from VMS to Unix.
I remember sitting at this training arms crossed, skeptically listening to the benefits of this alien outside operating system. Everything about it seemed hacky compared to VMS. Where VMS had very literal and descriptive commands like DIR to show a directory, DELETE to delete a file, and SHOW PROCESS to show what was running, Unix had strange abbreviated commands, like ls, rm, and ps. And when the trainer got to grep and awk, I almost lost my shit.
“grep?? What the hell does grep stand for?” I asked incredulously. The poor trainer did his best to explain that the point here was to create simple and speedy-to-use commands that could also in fact be chained together in powerful combinations. He said it didn’t really matter what the commands were called, because you would memorize them and they would become second nature, just like the VMS commands you are used to.
I didn’t buy it in the least, at the time. But he was definitely right, of course. In retrospect, I was a bit scared of the change, and was probably also intimidated by the thought of suddenly going from expert OS user to novice. I came around once I started actually using Unix. The more time I spent with it, the more I realized how much I could do with it compared to what I was doing with VMS. It was in fact hacky, as I had originally suspected. But therein lied its power.
Fast-forwarding about 20 years or so, I once again found myself on a development team caught in a sudden change of wind direction. This time it was a software development situation, and our management had announced we would be migrating our development process over to Agile.
I like to think I’ve always had a kind of pragmatic approach to building things, one that skips rules that are impeding progress when necessary. But I had spent years working in big hardware and software teams, and it had taught me to expect there to be a pretty rigid, waterfall-style development flow.
The next thing you know though I was sitting in an Agile training meeting, arms again crossed, listening to heretical talk about how we didn’t need to plan things out six months in advance, and could just do it in two-week “sprints.” It was Unix vs. VMS, all over again. No weeks writing detailed specs, no spec review phase before development starts. Insanity!
When I had started at Digital, I worked in a mainframe development group. Spec reviews were sometimes held between the hardware team, and the VMS team that was writing the operating system support for the new machine. These reviews were attended by dozens of people from each team and took a full week. They were held at hotel event rooms and featured fully catered lunches, and the VMS participants had to drive almost an hour to get there to attend.
This ridiculously-expensive development process soon went by the wayside, and while I do miss getting the free lunch, it’s quite obvious that there is no reality in which that was an efficient use of everyone’s time.
When I develop code today I am still a very big fan of planning things, and documenting what was done. But gone is process of spending days and days writing monster specs that detailed every aspect of a new product or feature.
I wrote a lot of huge design specs in the past, but almost always was left with a sense of disappointment, when the reviews of them were either poorly attended, or resulted in few meaningful comments. The honest truth was and is, people just don’t have the time, and maybe attention span, to pore through page after page of detailed specifications for an entire product or large feature in a single sitting, or event.
Everything new starts out imperfect and therefore does need to be reviewed at some point, and will get its chance either via some internal process to a company, or later by its customers. That latter situation is not usually the one you want. Finding problems is less expensive earlier in the process, and some kind of design review before implementation still seemed like something we could not do without.
Mr. Smith Goes to SCRUM School
This was very much on my mind when we transitioned to using Agile. It seemed to me that we had abandoned any sense of product planning, and in trade, management would attempt to exercise a level of control over scheduling low-level activities like individual feature development that was neither practical, nor necessary.
Many people on our development team were privately unhappy as well, but said nothing. So as a project lead, I was the vocal skeptic of the transition, figuring it was my job to raise concerns. In response, our management used a classic trick from the management playbook, which was to put the strongest opponent of a thing in charge of it. I was told I was going to be the Scrum Master of the team, and I was sent off to Certified Scrum Master (CSM) training.
Not going to go into too much detail here about Scrum or other Agile processes, since most either know what it is at this point, or don’t care. But a very brief description of my new job role would be, I was the foreman of our agile development team, and it was my job to make sure people were able to get their tasks done on time, over the two-week sprint cycle we now worked in.
This sounded to me then and sounds to me now like running a job shop, where workers are just cranking out things in assembly-line fashion. But the devil is in the details. What I learned after going to Agile school and then also putting it to practice for several years was that it was certainly possible to create a poorly-designed product, implemented by a micromanaged development team. We had some low points approaching that at one point.
But I also learned it was possible to adapt the Agile process to work for us, keeping the good parts, but kind of ignoring the stuff that didn’t make sense.
When I was at CSM school, the trainer there warned against just this. He said that people most often failed when trying to apply Scrum to their development teams because they were practicing “CRAGILE”, or “Crappy Agile.” A situation where a development team practiced Agile in name only, but only followed a select set of practices and not all of them.
An example of that might be, subverting the idea of breaking long tasks into shorter, achievable ones by creating vague stories like “Development, Part 1”.
The issue there would be one of ignoring the idea of Acceptance Criteria, and thus not being able to measure completion. If you can’t objectively actually say whether a story is done or not, you are really just practicing “Agile Theater”.
The Things We Left Behind
There were a lot of interesting ideas in CSM school I took away, but also to me at least there was a certain sense of over-zealousness about all the Agile development process details. I found myself among others in the class who were primarily full-time project management people, not developers. They were all pretty excited about things like metrics and tracking sprint performance and such, for instance, which I was less enthralled with.
Measuring the outcome of sprint cycles is an important part of the Agile process, and the idea is to use what you learned to improve the process going forward. This made a lot of sense to me and we continue to hold Sprint Retrospectives from time to time, which take the form of brainstorming sessions about improving our process, and avoiding going down dark paths like interrogating people about why this or that story did not get completed within the Sprint.
But there were things left behind from the classic Scrum textbook. The trainer who came in to train the rest of our team on Agile for instance was adamant about breaking work down into stories that are no more than a day long. We never even attempted this, it might be the first CRAGILE process change we did. It took a lot of work to convince developers that they could break a project they are working on that would take a month to do into smaller pieces of a few days or a week.
Breaking it down further to a day or less would have most certainly resulted in “Development, Part 1” types of stories, and “Agile Theater” to appease the management. The other big issue with such a fine-grained scheme was the amount of paperwork, in the form of entries in the Atlassian JIRA tracking system we were using.
The Agile Tail Wagging the Development Dog
I’m not going to slip into a whole review of Atlassian JIRA. It’s a complex and capable tool, with many features and also flaws. The biggest problem I think we faced with JIRA was making the tool work for us, instead of the other, tail-wagging-dog way around. In our early days of trying to plan sprints, we were very focused on completing all sprint content on time, and also in distributing tasks evenly to members of the team so no one was overburdened or left without something to work on.
It is definitely a “Cragile” alteration to attempt to load-balance all of your developers at the start of a sprint. The “correct” way according to Scrum would be to avoid making assignments of stories upfront, and have people take an available story during the sprint when they are free.
That process did not work well for us though, because we did not have completely interchangeable developers, and the tasks we put on our sprints required a lot of further estimation to be done to get any real sense of how big they truly were. We ended up with a process where we would assign all the sprint stories to people, they would break them down into smaller subtasks with better estimates, and then we’d have a meeting to try to balance out the work.
And that meeting took forever, the first time we attempted it.
Each of our eight developers had several stories, each with several subtasks. Trying to figure out how to add and delete stories and subtasks on and off the sprint, and swap around owners for them was a nightmare. The JIRA system was not set up to do this kind of thing, and was also very slow, which I will charitably attribute to some issue with our server hardware and not JIRA itself. The net result was, it would take us many hours to plan each sprint, working out people’s time on spreadsheets and then making tedious changes in JIRA for everyone.
You would think this should have been a sign we were on the wrong track and we would return to the classic Agile process. But no. A bunch of software engineers have other ideas of their own, often involving coding their way around obstacles.
We wrote a program using Atlassian’s REST API that let us extract all the subtasks for a sprint into a spreadsheet-like program, where we could easily move them around, to other people or sprints. It did all the math about who had how many hours of work in the plan for each sprint, and then let us do a bulk update to JIRA later, after all the changes had been made. Our planning meetings went from hours to minutes. We’ve used that process ever since.
In the end, the process has to work for you and your team, not the other way around. And if you need to ignore some rules to get there, that’s just fine.
Things We Kept
Not everything in Scrum Master Training went ignored though. Perhaps the best takeaway there was about how to run an efficient and useful standup meeting. When we moved to Agile and were told to run daily status meetings, everyone balked. Who had time to go to all these constant meetings? They were supposed to be quick, but ended up running long because everyone wanted to dive into too many details.
CSM school covered things like how to be a ruthless Scrum Master, and keep people from getting off track for too long. We worked on getting everyone on board with just exchanging status and not going too deep. And we got comfortable with sharing if we were waiting for someone, or blocked by something.
After a while, a funny thing happened. We would hold sprint retrospectives, to talk about what went right and wrong with a previous sprint. People started listing “the standups” as an example of something going right, and would even get cranky if we cancelled one.
Because we were getting value out of them. There were many times I would ask a prompting question about people’s obstacles, and someone would say “I’m waiting for Joe to finish his support code.” And then Joe, was at the meeting, would say something like “Oh, I didn’t know you were waiting for me.”
These types of interactions ended up removing delays that in our previous waterfall process, could have ballooned into weeks of waiting and exchanging email.
The Best Worst Process We Have
There were certainly downsides to moving to Agile. We had to adapt to having more informal planning processes, and it sometimes would come back to haunt us in the form of rework. We eventually got better at planning in investigation stories (known in Scrum as a Spike) to help us better estimate work and to work out details that would have previously been part of the Spec development process.
The benefits for our product team over the older waterfall process were pretty measurable though. Things just got a lot more streamlined, and we integrated our product validation into our development much earlier, which had quality benefits for us. Since our product was relatively new and had a small customer base, we really needed a process that let us adapt quickly to changing priorities, and this is an area that Agile excels in.
Hacky? Yes. But like Unix, therein lies its power.
So Agile development has been around a while, but is it here to stay? I’d say it depends on your definition of what constitutes “Agile”. I’ve seen a lot of articles recently about “Beyond Agile” or “Agile Development is Dead” and so on. People are looking for the next thing, and who knows what it will be.
But I would say whatever comes after it is going to be an evolution of Agile not an abandonment of it, because I really can’t see us returning to the highly-structured, years-long development processes I used back in my earlier engineering days.
But… If we could find a way to integrate a free lunch or two into the next development process, I would be on board!
Explore Further
Next Week: Rewind Time, once again. We’ll wrap up the year with another look back at Memos past, with updates and highlights of past stories, discussions, and interviews. The Mad Ned Memo takes a short break after that for a few weeks, but returns in January for a new year of nerdy old computer stuff!
The Mad Ned Memo covers topics in computer engineering and technology, spanning the past forty or so years. Get your weekly dose of nerdy computer tales and discussions delivered right to your inbox, and never miss an issue! This newsletter comes to you ad-free and cost-free, and you can unsubscribe at any time.
The Mad Ned Memo takes subscriber privacy seriously and does not share email or other personal information with third parties. For more information, click here.