You’ve Heard of Technical Debt. Now Let’s Talk Information Debt

As a product manager working with development and in technical infrastructure teams, I deal with technical debt on a daily basis, so Geetika Tandon’s recent article on the topic struck a nerve. As an information governance professional, it also made me think of the similar issues that arise with the petabytes of information sitting in our systems.

Tandon defines technical debt as updates applied for short-term gains without any vision for the future — and this lack of future vision is holding our organizations back. 

Just How Big a Problem Do We Have? 

When it comes to information, we can multiply the issues caused by technical debt by hundreds or thousands of times over, due to the massive amounts of information our modern digital business processes generate. According to Statista, the installed based of global data storage is expected to more than double from 5.8 zettabytes in 2019 to 13.3 zettabytes in 2024. Can we even wrap our heads around that size?

A zettabyte is 1 billion terabytes. At the beginning of my IT career 20 years ago, no one had terabytes of storage. We started talking about terabytes on a project in the early 2000s for a large EMC Storage Area Network setup. Now you can buy a 2 terabyte external hard drive for approximately $60. Meanwhile, it is not uncommon for organizations today to have petabytes of data to manage (1 petabyte = 1,024 terabytes).

ROT, Meet Information Overload

The field or discipline of ‘big data’ provides us with an acronym for helping us understand the masses of information we are generating, the 3 V’s:

• Volume — The size of your data stores.
• Velocity — The speed at which your business generates ever-more information.
• Variety — The many forms of information: Text, images, audio files, movies, information classified as private or confidential.

Now take into account the volume multiplied by the velocity and apply the old information governance term ROT: meaning redundant, obsolete and trivial information.

• Redundant — Information duplicated across multiple systems: How many places do you need to store the same email?
• Obsolete — No longer in use for any of several reasons.
• Trivial — Day-to-day work in progress items that don’t meet the definition of “business record,” have no value to corporate knowledge and are not required for regulatory reasons.

Kevin Parker wrote a great primer on ROT for AIIM in 2016. Read it and ask yourself: at what speed are we generating and adding tons of redundant, obsolete and trivial information to our information systems? This information not only eats up storage space, it generates overhead costs to manage it. And just like ROT always does, it gets in the way of our timely and appropriate use of good quality information, which plays a big part in business success. As Parker notes, ROT gets in the way of search, good quality analytics and compounds the already problematical issues of ediscovery across increasing volumes of information.

Related Article: Information Overload Comes in 3 Flavors: Here’s How to Combat It

Defensible Disposition and Information Value Aren’t at Odds

Let’s return to the idea of technical debt, and the concept of short-term value or expediency versus a long-term vision of value creation. Does anyone in your organization have that conversation about what data and information should be kept and for how long?

If you have records management professionals, or information governance professionals with RM in their remit, they can lead a risk-based discussion about what you should discard and what you have to keep. Beyond what you must keep for regulatory reasons, you want to keep some information because it provides value — e.g., it adds to corporate knowledge and organizational memory, or it provides a long tail for analytics, etc. These discussions are as much about what you can throw away and when, as opposed to “how long do I have to keep this stuff”? In technical terms, we should not just be discussing “records retention policies” but rather “defensible disposition.” Can I delete this after three years, rather than seven? Or is there a really good business reason to keep it for 15 years or longer?

So, in “information debt” terms, there’s a tension between managing ROT and developing a long-term vision of the value that information adds to our organization.

The two don’t need to be at odds. We don’t have to keep massive sets of raw data forever in case we want to run analytics on them at some unknown point in the future. Large data sets can be reduced to much smaller and more easily managed amounts of useful information, including some documentation on how that reduction worked, what algorithms were used, etc. This reduces the overall volume of data to retain. You’ll require some strong processes to manage this transformation, as well as discipline. You may have a great process in place for keeping annual sales or production figures, keeping the monthly raw data for a much shorter period, only to find that some well-meaning individual does not trust that process and has been copying terabytes of quarterly data to your cloud storage for the last five years ….

Related Article: Information Governance Is Boring, But Necessary

Just Remember: No Long-Term Vision Lasts Forever

I like the idea of applying the concepts and approaches to manage technical debt to managing ROT across our information systems. The long-term vision you create today must also have a lifecycle of its own. What was a good plan five years ago may no longer meet your operational needs. Or conversely, it may be generating too much ROT and adversely effecting your business processes. Always take a risk-based approach. Get your business, information governance and technology subject matter experts to work together to create a policy that works well for your organization. Just remember, this isn’t a one and done initiative. Build in a time to review this policy at appropriate intervals and change as required.