Everywhere I look I see software rapidly falling into obsolescence that will need to be re-written soon. A good example is how rapidly JavaScript (ECMAscript) is evolving. In a few years nobody will get excited about maintaining a ECMA5 code base… this is already true for some early Angular 1.x apps, am I right??? Another driver of obsolescence is the continual force march of updates and upgrades from big corporations like Microsoft, Apple and Oracle. In that case, it is all about money. It recently dawned on me that Scrum is also a driver of early obsolescence! In a nutshell, Scrum leads to technical debt and conceptual debt, but it can be managed if a) you can take a medium to long, view technology wise, and b) you have a healthy level of skepticism about anything that is hyped up.
Call me crazy, but I think software should last more than 2-5 years. There is a LOT of old software out there that needs maintenance. Some of it is built really well and stands up to the test of time (abstraction, configuration, logging, not locked into a particular vendor, etc). However, a percentage of that software starts off as or morphs into a totally unmaintainable rats nest. I’m talking about the kind of spaghetti that makes developers run and hide. Low quality software leads directly to higher costs and expensive fires like data breaches and glitches that release prisoners early. CEO’s hate having to deal with technology risk, because they think they can’t control it. I’ll explain below why Scrum feeds into this problem.
An overview of Scrum:
Scrum is a development methodology, a way of organizing work on a software project. The most commonly implemented idea in Scrum is a daily stand-up, where all team members gather together for a quick status report. Each team member lists what they did yesterday, what they will do today, and anything they are stuck on. Work is broken up into time boxed segments called sprints (usually 1-2 weeks long). To the team, the current sprint is all that matters. Each sprint has to produce something of value to the end user, so it can’t be all UML diagrams and refactoring. What gets worked on during the sprint is selected by the product owner, via a backlog of tasks they have ranked in order of importance to them. These tasks should already have estimates associated with them. Ideally, those estimates were reviewed by two or more technical people who are actually doing the work. The unit of effort can be hours, or more abstract things like story points, or even bananas.
What, estimating in bananas??? It sounds nuts, but it gets into the fact that estimates are often misconstrued, wrong, and used as political footballs in the game that is middle management. When you say 8 hours, what does that really mean? Is it 8 ideal hours? Is it one work day with room for meetings and water cooler sessions? Does it mean the work will be ‘done’ tomorrow? When an estimate is first uttered, it magically turns to stone and becomes a commitment to those who heard it. They may try to talk it down which causes the stone to shrink, again magically. Estimates are rarely changed unless extraordinary information comes to light. At the end of the day, as a middle manager, with bananas you have more leeway (if your boss will still take you seriously). A login page might take 4 bananas, but that upgrade to the reporting module might be 40 bananas. How many hours is a banana is worth? In Scrum, that is determined by looking at past sprints. Measure how many bananas got done vs how many person hours were available on the team that sprint. To tell how much ‘work’ you can get done each sprint, compare estimated bananas vs actual completed bananas, adjusted for variations in team size. That figure is called velocity. A team’s velocity might be 100 bananas or 1000. The actual number doesn’t really matter.
There is an ongoing circular debate about how to solve the estimation problem. Topics include relative vs absolute estimates, the estimation process, the units to use in estimation, who should do the estimating, estimate poker sessions, etc. Estimates by nature are probably wrong, so why fuss about them so much? Have the person doing the work make the estimate and then get out of their way. I’ve never seen actual work get done while philosophizing about estimates.
Speaking of philosophy, here is a better question to ponder: what is your team’s definition of done? Does it include writing some documentation? Unit tests? Peer review? QA testing? Deployment? Don’t forget those very important aspects of software development in your estimates. Skip these and somebody will be disappointed later on. The strategy most immature companies use is to entice their staff to becomes heroes which makes things even worse later on. They say it takes a start-up about 2 years to paint itself into the corner…
I’ve also found a) tasks have a funny way of getting completed within their estimates (but to varying degrees of quality and done-ness), b) it can be much faster to just do the task rather than sit around estimating.
To be fair, some estimating is a necessary part of the planning phase of any project. No approach is perfect. It is wise to double or triple estimates to allow padding and get the job done right. You only know how long it actually took until after the work is done. This bothers middle managers who need to file reports to their bosses showing how efficient and frankly amazing they are at herding cats and seeing into the future. To cover up that huge gap in managerial control, the Scrum people invented banana velocity to explain what is going on in a quasi-economic sense. I suppose banana velocity sounds sufficiently geeky so non-technical people just accept it. In repressive office settings dominated by cubicles, bananas go by the term ‘story-points’ which sounds more official, but not concrete enough to be an hour or a dollar which still leaves the all important leeway.
Scrum’s Strengths:
As a developer, I enjoy working in a Scrum environment because of the balance it strikes between structure and being left alone to work. Agile software development and its offspring Scrum and Kanban, achieve balance by making the software development process flexible. Work is based on real goals. Output is measurable. Features are launched regularly, sometimes with quick turn around. Scrum’s iterative releases are especially satisfying for the team. The stand-ups keep everyone in sync and problems are surfaced right away. Communication between technical and non-technical stake holders is baked into the process!
On a green field system, where the code is new, Scrum is fantastic because you can fly though features and put things in the hands of customers quickly. For one off contractor style projects Scrum does a great job.
Where Scrum falls apart:
Developers enjoy Scrum because they are left alone to work, but that enjoyment is a form of moral hazard. As a wise master put it: ‘the untroubled ease of programming beneath [Scrum’s] sheltering branches‘ (see The Tao of Programming section 7.1). Moral hazard is when one party is insulated from risk (in this case, the happy programmer), and they behave in a way they enjoy that may cause another party to suffer a loss (in this case the business they work for). A happy developer, working on small tasks week by week, does not see or have reason to care about the big picture. With Scrum, the big picture takes a back seat to the prioritized tasks on the backlog. The lack of big picture thinking and focus on the short term is what makes Scrum so dangerous to software quality.
As the sprints wear on, things begin to change. The velocity begins to drop and estimates begin to creep up, but that’s okay because we are talking bananas and there is room to fudge it. Eventually weaknesses begin appearing in the system design and a few fires break out. Perhaps a scapegoat will be blamed for the headaches? It is the system, not the people that is causing the actual problem.
There are three main ways in which Scrum causes the code base to be more expensive to maintain as it progresses, which leads to obsolescence.
1) Scrum is like playing drunk jenga in reverse:
Scrum says nothing of technical details, leaving that ‘to you’, as in, an exercise you can do on your own. Seriously, unless we are talking about homework assignments, nobody does those. Given the lack of specific direction there, and the fact the Scrum product owner in a purely technical sense is likely clueless, I don’t think most Scrum teams address technical details as good as they could, or at all. The new bits continually accumulate on top of the older bits forming a kind of upside-down pyramid that is prone to collapse. Imagine building a 2000 square foot house, where every two weeks living space has to be delivered to the customer. Every two weeks the customer immediately moves into the new living space. For the next sprint, you either start with a small room and build a larger story on top, or plop down a new disjointed foundation and knock holes in the walls to join the rooms. Even with a blueprint the decisions made in the early sprints invariably end up being poor decisions later on.
2) Non-technical product owner steers the ship:
At the start of each sprint, called the sprint planning meeting, there is a little dance that happens between the technical and non-technical staff as to how much customer facing work gets done vs how many backend fixes and improvements are made during the sprint. The product owner controls the money, so they get what they want. Unfortunately, the product owner may not understand technical jargon or care that the RAID array needs its drives updated. System improvements and maintenance tasks usually get pushed to the next sprint until they manifest themselves as a fire which interrupts the sprint and defeats the process. I like to call that spontaneous self combusting technical debt. When that happens, sometimes it is a random surprise, but it is often due to poor management. Scrum seems to bring out the worst decisions in non-technical managers.
Over the life of the software system, Scrum is sort of like driving with a blind fold on. Sure the first few blocks might be okay, but you’ll be in the ditch a mile or so down the road. I’ve seen this unfold many times. A non-technical product owner (someone who I often respect and work well with) ignores the overall technical health of the system for a short term gain (need to get paid, need to make client X happy ASAP, etc). On a lot of software projects, especially under Scrum, the non-technical leadership has too much sway on technical matters. They may not trust technical people, they may want to assert their dominance, or they may be blindly following Scrum to the letter. Either way, we need a methodology that tells non-technical leaders to listen, but that hasn’t been invented yet. Only a few are smart enough to listen and lead. I’m not saying each sprint should be 100% ruled by geeks. That is just silly since technical people often have no clue about business. It needs to be a healthy balance, and I’ll get to that later.
Related to estimation and non-technical product owners, consider an example. Let’s say the backlog has twenty tasks each taking 10 bananas (200 bananas total) and at the very bottom some technical task estimated at 100 bananas that refactors the system making everything else easier. The 100 banana task delivers nothing to the customer, but reduces the effort of the ten banana tasks to just one banana. The net cost of everything is 120 bananas, but requires halting new deliverables for 100 bananas worth of work.
I’ve observed that tasks with smaller estimates are universally made higher priority. This is the – let’s get the cheap stuff done first school of thought. I have rarely seen a non-technical product owner opt for the 100 banana task even though the payoff is clearly there. With Scrum, the product owner gets hooked on immediate results from each sprint. They are turned off from devoting two or three entire sprints to something ‘big’ and ‘risky’. Thus, the big important technical stuff never gets done. This leads to problems and the technology gets blamed instead of the real culprit – the process and people managing the technology.
3) Commoditization of developers:
Others have pondered, Is Scurm is the McDonald’s of software development? Comparing any software development process to McDonald’s is a bit rough. Though it is true, Scrum views developers the same way McDonald’s views burger flippers. Scrum treats developers as interchangeable resources capable of producing bananas.
Career-wise, under Scrum, developers rarely get a chance to do big picture thinking beyond the current sprint. The backlog, which is prioritized by the product owner, dictates the engineering effort each sprint. No work is supposed to span multiple sprints. There is a limit to career growth that can happen in this setting. The post Why Agile and especially Scrum are terrible goes into more detail on that subject. It hypotheses that some companies use Scrum to turn low wage C players into more than the sum of their parts. Bizarrely, this may be seen as a ‘profit’ for the company using that strategy. In reality the best people in software either find a culture they fit with, program in comfort under the shade of the corporate tree branches (and could give a shit about banana velocity), or they create their own companies.
How to use Scrum correctly:
So, what can we do about the fact that Scrum leads to problems down the road? In the software world, as usual, the answers lie in The Mythical Man Month, which was originally published in 1975, way before Agile was a thing (roots in the mid 80’s at the earliest).
In software, there are three main wheels turning: the business, the architect/designer, and the implementer (aka coder). Scrum places the business in control of the implementer and kills the architect. This problem can be remedied with the following:
To address weakness #2 Non-technical product owner steers the ship:
a) Get a product owner who understands the business AND the technology.
or
b) The product owner is replaced by a committee made up equally of business types and technology types who prioritize the back log on a regular basis.
To address weaknesses #1 drunk jenga design and #3 commoditization of developers:
The architect/designer skill needs to be resurrected and given prominence. Work will still be broken into sprints, but that work must adhere to the big picture technology-wise. In other words, there is a plan in place, a road map that lives outside whatever happened to bubble up to the top of the backlog this week. That plan should influence what bubbles to the top of the backlog.
This role could be filled by a lead or senior dev. It should involve junior devs on a rotating basis to give them a career boost and learning opportunities.
The Mythical Man Month advocates for an empowered designer. In reality, many software architects are powerless ‘technical leaders’, spouting off Power Point email attachments in excess of 10MB. Software developers are largely free to ignore all the white tower bullshytt. If the software architect has no teeth, you might as well just laugh behind their back with everyone else. They gave up their coding skills for Power Point anyway, didn’t they?
To ensure quality and consistency, many open source projects nominate a benevolent dictator for life – BDFL. Hey, if you want to get big things done, history has shown a dictatorship is the way to go. The people’s feelings be damned! The BDFL makes decisions that sometimes screw over a few small groups, but they are working towards the greater good for the project.
All Scrum projects would benefit from a technically oriented benevolent dictator (not necessarily for life). This role is not the Scrum master. This person must understand the business. They must also be insulated from petty politics and short-term incentives. The equivalent might be giving the CTO a 25% stake in the company, and starting with the understanding that sometimes the business won’t get what they want right away. The business side needs to trust the CTO’s judgment implicitly like Kirk trusted Spock.
2 Responses to Three Failings of Scrum and the Old School Solution