Feature: DMLZ outputs in deploying DLZ via DevOps

[Hurvester]

Feature or Idea - What?

The "params.*.json" files seem to only include fixed values in the many IDs asked, could these be re-written as variables that use the actual outputs from the DMLZ that has been deployed earlier? Or how does this all work with Azure DevOps pipelines? I pasted two sample parameters:

The outputs set in the end of "main.json" can be found in Azure deployment history, yet the DLZ script doesn't seem to request those and instead uses the fixed values from the "params.*.json" file, which does not seem too automated? Or am I not seeing something?

Feature or Idea - Why?

We have a repo deploying the DMLZ and then we need those outputs in all the possible future DLZ deployments that are located in another repo, how can those be linked even after time has passed?

Code of Conduct

• I agree to follow this project's Code of Conduct

[marvinbuss]

Thanks @Hurvester for the question!

We are seeing that these two artifacts (Data Landing Zone and Data Management Landing Zone) usually have a different lifecycle and are also often managed by separate teams. Hence, they will land in different repositories on the customer side and will be deployed individually.

There is the possibility to merge the data landing zone and data management landing zone artifacts and deployments into one repository and CI/CD pipeline, if both of them have the same lifecycle on your side and if both of them are managed by the same team. However, even if this is the case today, I would not advise to do that. The responsibilities and ownership may also change over time or may change for a subset of Landing Zones. Separating the two artifacts later would be more tedious than just keeping them separate right from the beginning.

However, in general merging the two IaC artifacts is indeed possible and the proper outputs are already defined so that they can be passed between the two IaC templates. In fact, this is setup because we are already doing this for the All-in-one deployment that you can find here: https://github.com/Azure/data-management-zone/blob/main/docs/reference/dataManagementAnalytics.bicep
You can deploy this by clicking on this "Deploy To Azure" Button in the Readme file:

Please let me know if more clarity is required. If this answer was helpful and answers all your questions, then please flag this answer (Mark as answer).

[Hurvester]

Thanks for the quick answer!

The script in that link you provided, uses the DMLZ outputs in the further DLZ deployment as I was also planning to do. Yet, in that script, both LZs are deployed in the same execution? I'm trying to build this using the same approach you recommended. So, what if you wanted to deploy the DLZ, and any DLZs after that, as a totally separate entity -- and from separate repos -- than the original DMLZ?

In a way, the way I see it at least, the "DMLZvnetID" and some other ID parameters ("Purview" etc) for the DLZ should to be fetched from the Deployment History in Azure? Do the "references" still work similarly as they're used in that script you linked?

[marvinbuss]

Again, I want to repeat that I am not recommending you to do that, but if you want to merge the two, you can pass the parameters from one CI/CD step to another one within the CI/CD deployment pipeline and overwrite the default values that you have defined in the parameter file.

You can also deploy multiple landing zones, by specifying additional deployment steps in the CI/CD pipelines and just reference other parameter files for each individual deployment.

here is a sample of how this can be done:

data-landing-zone/.github/workflows/dataLandingZoneDeployment.yml

Lines 136 to 138 in f7fcb35

	-SubscriptionId "${{ steps.data_landing_zone_deployment001.outputs.mySqlServer001SubscriptionId }}" `
	-ResourceGroupName "${{ steps.data_landing_zone_deployment001.outputs.mySqlServer001ResourceGroupName }}" `
	-MySqlServerName "${{ steps.data_landing_zone_deployment001.outputs.mySqlServer001Name }}"

[Hurvester]

Oh yeah, now I see. Thanks!

[MiguelElGallo]

@marvinbuss
I totally agree with this: "We are seeing that these two artifacts (Data Landing Zone and Data Management Landing Zone)"
That means we have two repos, two pipelines.
But we need an "automated" way to pass the output of the Pipeline 1 (DMLZ) to the input of the other Pipeline 2. (DLZ).
Which is the recommend approach for this ?

[marvinbuss]

What is the problem with referencing this value as a parameter from the parameter file?

If you chain these artifacts one way or another, that means you will again end up with a common lifecycle for the Data Management Zone and the Data Landing Zones. I mentioned a few options above, that allow you to achieve what you are looking for (pass values from one deployment step to another).

To simplify the setup, the Data Management Zone deployment already provides the necessary information as outputs. You can pick the up from there to simplify the copy activity.