Entropy: It's not just a good idea, it's the law. In thinking about this week's post on data persistence, the 4th in a series on Information Governance, I recalled two distinct instances where the second law of thermodynamics had an impact on my life: (i) Physics course in college and Ken Kesey's book, Demon Box, a collection of essays that dealt in large part with the notion of entropy. So what is entropy and what does it have to do with this week's post on data persistence? I'm not sure but here's my hunch: Entropy, in this context, can be seen as a game where you are trying to organize, manage and store terabytes upon terabytes of data (if not now, soon "“ and then petabytes and so on"¦) within some defined, structured storage box and the un-captured data (really the people that create, use, rely upon and NEED IT FOREVER, sorry"¦) are doing everything possible to break free from your constraints and behave, well, like children at recess that is, crazily, randomly and chaotically. Two competing yet plausible, if not viable, interests at play. Finding that balanced approach is the key.

Now picture the demon in James Clerk Maxwell's musings on the second law of thermodynamics. Acting essentially as a kind of troll, raising or lowering the bridge on which people or objects pass (in our scenario data passing from unstructured data repositories i.e., the playground to structured data storage devices, i.e., the classroom), thus altering the dynamic on either side through gain and loss, structure and non-structure, this demon or troll either contributes to or lessens the effect of entropy.

IT teams, storage vendors and consultants spend countless hours trying to reconcile the needs of the user with the demands of their environment. Blogs, conferences, seminars and the like are devoted on an almost daily basis in trying to figure out how to cluster, compress, hash and index large, unwieldy volumes of data in an efficient and meaningful way so that the folks tasked with managing the Beast can sleep at night knowing (or at least perceiving) that their data are being managed and stored in a manner consistent with their organizational, legal and regulatory requirements. But at almost every turn in this ongoing saga are the antagonists who perpetuate the myth that data cannot be managed, it cannot be disposed, and it must live forever because it just might be needed.

Good governance, meaning smart strategic information management planning, project modeling, unified user/administration execution and policy enforcement is the only way in which this myth can be quashed. Strengthened by new, exciting storage technologies such as those outlined in this month's Scientific American, and the emerging principles encapsulated by Lean IT (link requires free signup) may help the dismantling of the myth as well. Organizations that are not addressing their data creation, data management and data storage strategies are simply allowing the demon to shut the gate to intelligent governance and thereby adding to the preponderance of unnecessary costs, unstructured and useless data while simultaneously perpetuating bad business practices and inefficiency excesses.