Data Management Hub - Raw zone Data lake

[namrata01Apr]

2 customers that i am working on have asked for a centralized data lake ( only the raw zone) in the data management hub - Can this be included in the template?

[marvinbuss]

This is not in line with the principles of this.
The Data Management Zone is not here for data storage. The Data Management Zone is about governance and data management and not about integrating, storing or processing data. Please let us know why this can't be done in the Landing Zones and what is blocking.

[namrata01Apr]

This is a pattern considered by 2 banking customers (both are in the design phase) - where they want to land the raw data into the management zone -- to be owned by IT team and then the Domain Owners will pull data from the raw zone in the management zone to the Data Landing Zone for curation and transformation.

…

On Thu, Dec 16, 2021 at 5:52 PM Marvin Buss ***@***.***> wrote: This is not in line with the principles of this. The Data Management Zone is not here for data storage. The Data Management Zone is about governance and data management and not about integrating, storing or processing data. Please let us know why this can't be done in the Landing Zones and what is blocking. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#257 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQP3545NX6WLO46WG45WAL3URHKY3ANCNFSM5KF7AQQQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[marvinbuss]

Why not land the data in a Data Landing Zone? The data can still be picked up by Domain Owners once the data has landed there.

The Data Management Zone is solely for governance and data management purposes and not data storage or processing. This is recommended because of clear separation of duties and ownership in the data platform. Also, the data in the management zone would always have to land in the same region, as the Vnet and all the other data management zone services are tied to a specific region. Would this not be a problem in that scenario?

Also, how will cost be split across domains and businesses for these datasets? Will this the paid centrally? Usually, this is something that is supposed to be split across business groups within an organization.

[namrata01Apr]

This is my understanding from the customer today - Bank is going to use central india as primary and south India as DR . 1. they need to land all the raw data - as identifying domains etc may take time - hence they need an area to land all the raw data and the start thinking about how to segregate to the different domain - so as a temporary solution could be spin up a data landing zone for this — however again this could lead to them continuing to use the central landing zone and never come out of it — this was the same same pattern asked for by the Australian bank — Citing same reason that they want to have a centralised control on all the raw data 2. There will be common data that may be needed by all /some of the domains eg .. customer data etc — will not be specific to a single domain — is there a pattern for the common data ?

…

On Fri, 17 Dec 2021 at 3:01 PM, Marvin Buss ***@***.***> wrote: Why not land the data in a Data Landing Zone? The data can still be picked up by Domain Owners once the data has landed there. The Data Management Zone is solely for governance and data management purposes and not data storage or processing. This is recommended because of clear separation of duties and ownership in the data platform. Also, the data in the management zone would always have to land in the same region, as the Vnet and all the other data management zone services are tied to a specific region. Would this not be a problem in that scenario? Also, how will cost be split across domains and businesses for these datasets? Will this the paid centrally? Usually, this is something that is supposed to be split across business groups within an organization. — Reply to this email directly, view it on GitHub <#257 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQP35444PBR4PBLG55DKIZDURL7PRANCNFSM5KF7AQQQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[marvinbuss]

Hi @namrata01Apr,

1. Raw data
   Makes sense what you are describing. On the other hand, what keeps them away from staying with the intermediate solution of landing all the data in the management zone? In general, they should think about splitting the responsibility for these source systems and domains across multiple domain specific teams that are SMEs wrt the source systems and datasets. This will allow more quick and iterative development and a feedback cycle between producers and consumers. More on that here: https://github.com/Azure/data-management-zone/blob/main/docs/guidance/DataManagementAnalytics-SelfService.md

2. Common Data
   Some clients are creating a "Corp" landing zone where they land core corp capablities, but in general it does not matter in which landing zone these datasets are landed as we have designed the architecture with the concept of data sharing in mind. Data can easily be shared across landing zones on the networking as wall as IAM layer. Are there any other requirements that I am missing?

[marvinbuss]

I will move this to a discussion for now. Let's continue the discussion in the "Discussions" Tab.

[namrata01Apr]

@marvinbuss Do we have a reference implementation for the core service provider pattern as well? Or is the reference implementation limited to the harmonized mesh pattern

[marvinbuss]

Hi @namrata01Apr,
What do you mean with core service provider pattern?
Is this the name for the solution pattern, where the raw data lakes are deployed in the data management landing zone?

We do not have a deployment template for this pattern, as we are not recommending this option for the reasons mentioned above. We would recommend to rely on a data landing zone for landing the raw data.