View from the Bridge

Technology Debt – when your infrastructure has been ignored too long

“What do you mean “there is an executive level downtime committee that meets weekly”?

As I stated this, I must admit I was a bit incredulous. Although it was only my first week as CIO at a large academic medical center, I had never encountered in my prior 8 years as CIO in both large and mid size health systems.

Before I return to this experience, let me first try to explain what is meant by “technology debt”, or at least my interpretation. For CIOs, now appropriately focused on innovation, AI, ML and digital transformation, it is a given that the organization assumes the CIO has already locked down a solid and reliable foundation of infrastructure to support these go forward efforts. While foundational “chips” in the CIO game, it’s often surprising how weak that “hand” is. This is often as a consequence of the historical accumulation of funding shortfalls or investment compromises that supersede the CIO’s judgment, pleas, and warnings.

The nature of substandard or compromised IT infrastructure, be it network switches, WiFi, PCs,  storage, etc is that, in the short term, they may appear to meet the need, but eventually cause havoc. The funny thing about IT infrastructure is that you ignore it at your peril. Sooner or later, it turns on you, often at the most inopportune time.

Challenges hidden in the architecture can bring surprises that make projects run over budget and miss deadlines or even compromise the entire reliability of the IT technology stack. In these cases, much of IT employees’ time is spent managing complexity and excessive break-fix, rather than focusing on optimizing workflow improvements for the customer.

In the case I began with, that is precisely what happened. Repeated entreaties from a succession of prior CIOs had gone unaddressed. The “red flags” were ignored, as it seemed more beneficial to keep IT investment at a paltry 2-2.5% of budget. At least, until that fateful summer when the entire network collapsed along with significant parts of the on-site data center. Total shutdown for weeks! Finally, a “pain threshold” was reached. An emergency board meeting gave IT a nearly “blank check” to “fix it”. Fortunately, for me, I arrived well after a brand new, state of the art, LAN and WAN were in place and a new data center nearly completed.

The next time I encountered a similar high risk “technical debt” situation was nearly 18 years later at a much smaller organization. An organization, I should mention, that was quite proud of their historically strong financial position. Twenty years of considerable “frugality” had made that possible. However, it came at the cost of under investment and not only in IT. As the new interim CIO, early on I had to assess that deficit and begin a series of “The Emperor has No Clothes” type conversations with the new executive team that was thankfully receptive and understanding.

The very limited budgets that had been historically foisted on IT leadership encouraged sub-standard and compromised infrastructure decisions that while creative in the short term, were not sustainable long term. Examples included:

    • In lieu of Ethernet connected workstations in each ambulatory exam room, providers were asked to carry a WiFi enabled laptop from encounter to encounter
    • In lieu of providing basic MS Office software among professional staff, managers were expected to convert word documents/letters/memos to PDFs to share with their professional staff
    • In lieu of a centrally managed VPN based remote access solution, remote workers were expected to use a “Go-To-MyPC” type of connection, fully dependent on a dedicated single PC in their department being turned on, in good health and not in use by someone else.

That IT infrastructure was broadly deficient, often obsolete and far from industry best practice. Fragile and difficult to support, it really was a tribute to the IT team that they had been able to make it functional enough for most end users to do their jobs, albeit not optimally.

And it’s a widespread issue in the healthcare industry, as a whole. In fact, according to research firm McKinsey, 60% of CIOs reported increased spending on technical debt in 2020 alone. For many, that only grew as the pandemic spread.

The worst scenario is unplanned downtime. A study by healthcare IT consultant, Mark Anderson, 20 years ago found that each 1 percent of downtime (maintaining only a 99% uptime) could cost a 500-bed hospital more than $1.4 million per year and a 1,400-bed IDN $10 million per year in additional operating costs.

We can easily double those figures today. It takes little imagination to calculate the cost of a weeklong infrastructure failure in the tens of millions for even small health care systems. This is the real “Risk” calculation that should underpin any executive level discussion of technology debt.

 

To begin the process of tackling your technology debt, start with a comprehensive assessment.

Examine Your Tech Stack

First, take stock of what systems you’re using now and establish a go forward set of policy-based technology standards and ensure they are well communicated across the organization. Here are some things to consider when auditing your systems:

    • Which ones don’t serve your needs?
    • Which items don’t follow your defined standards or even well recognized industry standards?
    • Which ones contribute to siloed data?
    • Which ones require excessive labor to “keep alive”?

How do you tackle tech debt? The best approach is through a strategic process. One that follows certain principles:

    1. With input from business and IT leaders, agree on a shared definition of tech debt. It could be defined as the negative impact of technology on the business, particularly as manifested in rising operational and technology costs, unnecessary complexity, reduced flexibility, and increased risk of downtime.
    2. Treat tech debt first as a business issue. Reinforce shared responsibility for outcomes. Efforts to tackle the debt must be clearly linked to strategic priorities.
    3. Ensure a solid decision-making process. Using a portfolio approach based on a clear, mutually agreed-upon set of rules and standards that allow IT and the business to work together.
    4. Assign appropriate resources to tackling tech debt. When there is a unified view of the tech debt situation and what the key goals are, allocate funding, mobilize people, and send a clear message from the top that addresses this technology deficit as a priority for the organization.
    5. Avoid a big-bang approach to writing down all debt. Tackling tech debt via infrequent IT mega projects poses high execution risk. Instead, assign a portion of the IT budget for paying down debt consistently, predictably, and over a defined time horizon.
    6. Examine which areas are unavoidably “bankrupt” and consider shifting them to an entirely new technology stack. This may have downstream consequences that make it a last resort, but it can’t always be avoided.

By observing these principles, a health system can design a process for tackling tech debt in a matter of weeks or months, with implementation following almost immediately. Measurements of tech debt will need to be built into financial models, tools, and databases across the business.

The long road of digitization has brought health care organizations beneficial technologies and capabilities, but at a cost and with a focus more on the application layer than the underlying infrastructure. Accumulation of tech debt has left some organizations struggling to bring innovations to market or internal automation for efficiency gains. The good news is that tech debt can be measured and managed. With the right approach, organizations can regain control and refocus their technology resources on creating value for employees and the patients they serve.

Leave a Reply

Your email address will not be published. Required fields are marked *

Top