So, I’ve worked designing software and IT systems for a while, both private and public sector, and I have some thoughts on this that may help in understanding how we got to where we were.
First off, I’m sure it’s pretty common knowledge at this point that the software these systems run – generally COBOL – is really old and deprecated. When the majority of these programs were written, the US’s annual unemployed number was much lower; approximately 7.6 million in 1960, give or take. Back then, the processing for unemployment benefits wouldn’t have been done online, obviously. Paper forms and data entry processors would’ve kept the influx of requests steady and predictable, which makes designing stable systems a relative walk in the park for developers.
For some background, in 1985, about 10 years after the rise of most state unemployment system designs were set, a conference held by IT nerds like myself describes 200tps, transactions per second (think of a single transaction as something like trying to log in, or visiting a new page), as “high performance.” The rest of the article goes on to describe the absolutely ludicrous idea of processing 1,000tps and why it’s so economically infeasible, so this is the time period in which the systems were built.
As time has gone on, the systems stayed steady and maintenance of them was pretty straightforward. Unemployment spiked in 2009 but most of the complaints were about heavy call volume rather than issues applying online. Keep in mind that this system still closely matches the state of applying in the 1960’s, which is a data entry processor entering in the information manually to the system the same way they’ve been doing for years. The transaction rate for these systems would’ve stayed stable; claims processing still took a while and state governments wouldn’t have wanted to fork out extra money during a recession for temp workers, or especially the much higher salary of COBOL developers due to the risk of a negative political campaign. So, the systems didn’t change much then, and haven’t changed much now.
Today, the advent of online forms has dramatically changed the way people handle online business. Consider the disastrous launch of the healthcare.gov website. It peaked at 250,000 online concurrent users, each one sending (presumably) multiple transactions in a matter of seconds. When the unemployment systems were designed, 1,000tps was absurd. It was likely that the
healthcare.gov website was responsible for handling 500x that.
This is the environment in which we now find ourselves today. In this article, we’ve already established a record shattering 26.5 million Americans applying for benefits in a month. These people are using online forms that, to save costs, are usually just a modern face hitting the same deprecated 1960’s systems. Is it any surprise that there are problems?
Keep in mind that it’s not just applications that are causing waves of transactions. People go online to check the status of their application, which is another transaction. Other systems need to know who to pay benefits to, and how much to pay them for. Those are heavy on system resources. People will unknowingly submit the same information twice. Checks have to done and audits made. The websites used probably split out requests into multiple transactions. Knowing this, I wouldn’t be surprised to hear that these legacy systems are peaking at hundreds of thousands of transactions every second. When they crash, the transactions just back up until the system is alive again. Then, the wave returns and the system crashes yet again.
In this time of crisis, states are desperately calling for volunteers who can translate and help rebuild these legacy systems. Years of hemorrhaging IT budgets and laying off experienced staff, combined with retirement and a shrinking workforce of experienced COBOL developers, has kept many states from investing the resources necessary to upgrade these systems. The switch from legacy to modern is incredibly expensive, and risky. Up until now this kind of preventative maintenance has been viewed as expensive, political, and unnecessary. But now we’re paying the cost.