As the world becomes increasingly familiar to working with more data at a much faster rate, I anticipate that real-time data analytics will be the next big thing in the world of data. Thanks to the accessibility of tools like Azure Data Explorer and KQL, companies of all sorts will be adopting real-time analytics in the coming months.
Aside from the technical aspects of real-time analytics, data quality and data governance will be crucial and more relevant. Nobody wants to consume data that is not accurate or not well defined. Imagine doing that in real-time, where the same data quality issue will be highly expanded and propagated across any platform or application. Things get worse if those applications are impacting the end customer.
This is where Microsoft Purview comes into place – to add a data governance layer on top of any data source, but more importantly over real-time data, like KQL databases. In return, it is also possible to use KQL to query Purview metadata and perform real-time analysis for data governance.
Some of the benefits of using Microsoft Purview to govern KQL databases as real-time data include:
- KQL editor can search queries in Microsoft Purview eDiscovery tools with autocompletion, suggestions, and error detection
- Microsoft Purview Data Map can be used to create a holistic, up-to-date map of your data landscape with automated data discovery, sensitive data classification, and end-to-end data lineage. This includes KQL databases
- Data Catalog allows interactive data lineage visualization to make company data easily discoverable by using familiar business and technical search terms, an enterprise-grade business glossary
- Microsoft Purview Data Sharing enables in-place data sharing in near real time and easy provisioning of data access
- Microsoft Purview Data Policy to enforce data security and compliance policies across the data estate using DevOps policies and data owner policies
How to Ingest KQL Databases into Purview for Data Governance
To ingest KQL databases into Purview for data governance, you need to follow these steps:
- Register a KQL database as a source in Microsoft Purview using the Azure portal or the REST API
- Scan and classify the data in a KQL database using the built-in and custom classifiers and Microsoft Information Protection sensitivity labels
- View and manage the metadata and lineage of the data in a KQL database using the Microsoft Purview Data Map and Data Catalog
Conclusion
Managing data is not an easy task, especially if is real-time data. Strong data governance becomes more relevant when data changes more frequently, and real-time data is ever-evolving.
Adding a strong level of data governance on top of real-time data will improve data quality and reduce data management costs by detecting and resolving data issues quickly, enhance data security and compliance, and minimize the risk of breaches, fines, and repetitional damage by enforcing data policies and monitoring data activities.
The post Data Governance in Real-Time with KQL and Microsoft Purview appeared first on Dynamics Communities.