Interesting new developments in Data Governance
Author: Lochan Narvekar . October, 26, 2023
I have been meaning to write this article for last few weeks, as we just finished a sizeable Data Governance project, implementing Data Governance Office (DGO), its operating model etc.
But what was fresh this time was the awareness that my client executive had of the new ways of going about Data Governance (DG) and new DG tools out there. It was refreshing to see a client executive so aware and challenge the team! This prompted me to contemplate writing this article.
I will spare you all the old DG primer. We all know about basics of DG. Of course, it’s curious to see it still be an afterthought and follow a larger enterprise initiative, or come down as a top-down executive mandate, neither are the most ideal situations to start looking at DG. But back to the topic..
What I want to do instead is to put my neck out and look at the relatively newer phenomenon occurring under broader umbrella of DG.
Some dimensions of the older DG programs still remain relevant in this newer approach; i.e. need for privacy, security, and compliance vs. flexibility and self-services. But that can be taken as a constant.
Before we go any further, I must state that all opinions expressed here are my own, and do not represent any product company.
New Paradigm
First of the patterns I see is championed by Product companies like Alation. They do support traditional way of implementing DG, so there you go. But what they afford you to do is to take up a functional area, opportunity, problem, and focus DG effort on it.
Let’s take a quick example; you have Sales reports not working in Microsoft PowerBI, which in turn depends on Snowflake data. Of course, in our fictitious company no report is credible. But, why boil the ocean they would say. Go after this one specific area, and catalog the heck out of it. They enable this pattern by using heavy dose of machine learning helping you to setup metadata repository ground up!
When I say machine leaning, one example is deducing the business glossary automatically from underlying schema; nice!
You can now build on this, assign ownerships and create lightweight operating model and be on your way. Of course, we will leave the good old “People” problem aside; that’s always a headache. But even the smallest of features in these tools, like proposing a top user of a table as a data steward is not bad, comes in handy as technical steward.
When you have plethora of things to do in a DG project, every bit of automation matters, and division of labor becomes easier during implementation.
In summary, I like this agile way of implementing DG, as you can show concrete fruits of your work. Now, arguably this is not applicable to all companies and all domains, but I find it a great tool in my arsenal as I interact with a new customer, especially where there are other factors such as budgets, expectations, patience, sponsorship etc. are at play. One can show a quick win!
Another new Paradigm
Moving on, to another paradigm shift, in my view, is Data Product concept.
It starts with a Data Mesh paradigm created by Nextdata CEO, Zhamak Dehghani in her previous life as a Principal Consultant. This is expected to take on the traditional centralized data approaches. Already many leading tech companies are adopting the underlying concepts.
Data Product is like a boat in which all enterprise stakeholders want to get to a common destination; no more conflicting goals and metrics
One such concept is Data (as a) Product. It essentially talks about governing and streamlining the workflow between, and bringing together the producers and consumers of data. Today, these are two relatively different worlds managed and coordinated by a central Data Platform team.
In this pursuit, the role of a Data Governance tool and methodology is to bring together the Meta data and the collaboration on it starting from the business user, analyst, data engineer, down to application developer as needed. If Meta data required to drive this Data Product life cycle can be governed efficiently, as you can imagine, this will reduce the cycle times at the least. Not to mention the ownership for this decentralized team in the business outcome.
Some DG tools like Atlan are already doing this, taking the concept of Data Catalog beyond into this paradigm shift.
Before I leave, if you overlay these 2 paradigm shifts, I see these common points,
- Focus on a subset, a functional area, sub-domain, a tangible opportunity
- Bring value chain participants closer and,
- Use right Data Governance tool
Author: Lochan Narvekar; October 26, 2023
Please share your DG challenge or opportunities with me @ [email protected]
Lochan Narvekar is the Founder and Principal at PDMStrategy.com. He works with mid-to large corporations throughout US, including fortune 500 clients. He has published many articles on the broader topic of DG and MDM