Three questions engineers should ask themselves to help prevent tragedies like that of the recent Boeing 737 Max 8 fatal plane crash

On the morning of 10 March, under a beautiful Ethiopian sun, 157 men, women, and children boarded flight ET302 from Addis Ababa to Nairobi, Kenya. The Boeing 737 Max 8 airplane took off at 8:38 a.m. and six minutes later, the plane crashed and all travelers were lost in twisted carnage. The media has reported that the, maneuvering characteristics augmentation system (MCAS) anti-stall software system, played a significant role in the catastrophic failure of the plane. Every programmer should ponder the pain and suffering felt by the families left behind caused by, or compounded by, a possible software glitch.

Like many programmers my age, I spent most of my career writing code that solely moved slips of green paper from one ledger to another. Errors and half-baked ideas were only costly and embarrassing but didn’t harm anyone. But the craft has changed. Software can now determine whether a situation means life or death. I personally appreciated this change after I invented a method that enabled blind patients to self-administer medications. My mother-in-law lost her sight very late in life. I watched the burden that new blindness placed on the elderly. She struggled learning coping mechanisms that blind children learn much easier due to the natural higher speed children. I wanted to help. I devised a method that used a smartphone and special medicine labels. The smartphone would remind the patient it was time to take their medicine. The smartphone would use the labels to determine which container contained the correct pill. The patent would then be able to self-administer their own medicine. However, my employer decided to publish this invention as a technical bulletin instead of filing for a patent or creating an actual product.  I realized that anyone could take my idea and implement and support it with varying degrees of quality. My idea was a good idea if it were correctly implemented. However, it would be a disaster if it were poorly implemented and supported. I spent more time contemplating unforeseen consequences for my invention than any project before it.

Code is now more than just numbers. I urge all software engineers to always ask these three questions while working on a project that could affect people’s lives: “Should we?” , “Can we?”,  and “Did we?”.

SHOULD WE?

Many technical solutions seem like a good idea on the surface, but have unintended consequences. Software engineers need to ask ourselves, should we move forward with this tool? Consider a wearable technology that listens for any arithmetic problem and whispers the answer into an earpiece. It has utility and convenience, but what happens when a second grader wears it to her math class? Math facts are never learned.

One real-world example I experienced was after 9/11. Like many people, I wanted Osama bin Laden brought to justice. I began to research a process called stylometry, which is the statistical analysis of a human’s unique use of language and attempts to identify anonymous emails and text messages by word choices, spelling, and punctuation. Others were working on stylometry also. Tools such as JStylo and Anonymouth were released that attempted to tackle identifying the writer of an anonymous text. Moreover, it concerned me that my work with stylometry could also easily be used by an oppressive government.  I decided that, even if I could make some contribution in stylometry, I no longer wanted to do it. I abandoned my research efforts after a few months. We need to always ask ourselves how our creation might be used in ways we did not intend.

CAN WE?

Innovations must be physically possible; this is when the question of can we comes into play. The movie, The Demolition Man, has a great invention that is used in car accidents that seems so reasonable. When a car crashes, foam fills it up and protects the driver from injury. What a great idea! The problem is that this is what stage magicians call magic fairy dust. This technology defies all physics and chemistry. In reality, there is no such liquid that can be ejected that fast and firm up to the foam. Beware of the fairy dust step.

Software engineers should also understand that possible and practical are always moving targets. Several years ago, a young cousin of mine died an all too common death. He unknowingly bought heroin that was laced with carfentanil, an elephant tranquilizer. It was his last mistake. I looked down at the cold body of a young man that I held as a baby and wondered if some engineering concoction could have saved him. Maybe. I researched building such a device that used multidimensional factors—such as heat, electromagnetic, and light absorption/diffusion—to detect the presence of additives like fentanyl/carfentanil. If my cousin had known the drug he bought contained a substance designed to bring down a 13,000 pound mammal then he would not have injected it into his veins.  The algorithms and science theoretically work; however, fentanyl, and particularly carfentanil, is so potent in small quantities that it is impractical, given the current state of technology, to measure infinitesimal changes in those factors so, impractical for now. However, I left details in several public forums that could be used later. Again, possible and practical are always moving targets.

DID WE?

Quality is more important now that software bugs can kill. First, as the creators of these tools, we need to verify that we understand the true nature of the problem we are trying to fix. Does our code address the problem we intended?

Consider the dilemma of fentanyl-laced heroin. The problem I was addressing was overdosed addicts from additives. My solution was detecting any presence of fentanyl/carfentanyl in illegal drugs. I was trying to prevent dead drug addicts. However, these same methods and devices could also be used to control the manufacturing process of drug dealers. Aiding drug dealers in their manufacturing process was certainly not my intention. In fact, the method might be easier in the manufacturing process because it would be dealing with larger quantities than single doses. In other words, the state of technology might be able to detect variations of larger quantities in the manufacturing process easier than the smaller quantities being loaded into an addict’s needle.

We also need to test that our code does not create unintentional problems. Most software engineers have all experienced side effects while developing. A side effect is a state that is modified outside of the current process beyond the returned value of a function. Similarly, entire software systems often have unintended consequences which are like the side effects of single functions. An entire software system can be focused on one set of problems yet make changes in a completely different area with unintended consequences. Let’s take a hypothetical example. Suppose you create an autonomous lawn mowing system. The unit stops and changes direction when it detects an object directly in front of it. Let’s say that triggers playfulness in a pet and it becomes a game that results in the unit never finishing. Silly example but it’s important to observe your innovation being used in various environments and scenarios. Proof of concept projects and observing those projects in the hands of actual users is important.

The interaction of software development and quality control has changed over the years. Mainstream development has evolved from the waterfall model to the scrum model. In the waterfall model, progress flows downwards through  the phases of conception, initiation, analysis, design, construction, testing, deployment, and maintenance. The scrum process, on the other hand, was created for large groups who break their work into actions that can be completed within specific time frames.

Quality control is much easier in strict waterfall methodology. Part of the design process in waterfall is creating user acceptance testing that will be performed at the end. The scrum process requires more work to achieve the same quality control. While it has many advantages, including the fact that it is more responsive to user input and allows projects to grow organically, it can result in more delivered bugs. It is organic because changes are often submitted at the last minute. The Pareto rule, in experiences, applies to last-minute changes and bugs – 80 percent of deployed software bugs come from the last 20 percent of changes. Some teams never retest seemingly unrelated features when applying incremental changes. Scrum requires more diligence in testing. Remember, enforcing strict unit testing into the build process will mitigate the quality control weakness of scrum.

We also need to verify training is adequate for those who will be using the software, like the pilots of the Boeing Max 8. This is especially true if our innovation has the potential for harm. Some reports indicate that the pilots were inadequately trained, and it might have been the most fatal flaw in the MCAS system. Please make sure your users know how to use your innovation. Poorly understood  software is often much worse than no software at all. We have all experienced executing key stroke at the wrong time with dire consequences or discovering the last eight hours of work was unnecessary if we only knew the software would do “X” if we had typed “Y”.