Trigger configuration in data factory changed to allow empty files

[erincon01]

The pipeline is wisely configured to wait 60 secs before processing the .csv files.
I have a few subscriptions where data exportation first creates an empty file, and after a few seconds the file is loaded.
Having the trigger to discard empty files, makes not to tranform the .csv to .parquet.

I have changed the trigger configuration to allow empty files.

It seems like when you builded the pipeline you had in mind that takes a while to export the data, but when building the deploy, the "allow empty files" optoin was not considered.

You might find this useful for next releases.

[flanakin]

@MSBrett thoughts?

[MSBrett]

Hi @erincon01

The delay is due to how triggers fire in ADLS. In newer builds we no longer have a wait for the CSV files. The trigger fires when the manifest.json file is created, waits a bit for things to quiesce and then starts processing all the files in the manifest per the concurrency settings on the for loop.

[erincon01]

@MSBrett , the trigger is not activated for json files.
it is configured to process .csv files only.

"blobPathBeginsWith": "/msexports/blobs/",
"blobPathEndsWith": ".csv",

[flanakin]

@erincon01 Brett is referring to a branch where he's working on the next release. He changed it from watching for CSVs to watch for the manifest file.

[erincon01]

Hi @MSBrett ,

I've deployed v0.2, and I see you have changed the trigger name to msexports_FileAdded.
I confirm that is subscribed to new files in the msexports folder as the older version.
It fires msexports_ExecuteETL pipeline. Older version fired: msexports_extract.

See that the trigger is disabled, and is configured to discard empty files.

I am not sure if I've deployed the correct version (deploy from here: https://microsoft.github.io/finops-toolkit/hubs).

{
"name": "msexports_FileAdded",
"properties": {
"annotations": [],
"runtimeState": "Stopped",
"pipelines": [
{
"pipelineReference": {
"referenceName": "msexports_ExecuteETL",
"type": "PipelineReference"
},
"parameters": {
"folderName": "@triggerBody().folderPath",
"fileName": "@triggerBody().fileName"
}
}
],
"type": "BlobEventsTrigger",
"typeProperties": {
"blobPathBeginsWith": "/msexports/blobs/",
"blobPathEndsWith": ".csv",
"ignoreEmptyBlobs": true,
"scope": "/subscriptions/XXXXXXXXXX/resourceGroups/rg-finops-v02/providers/Microsoft.Storage/storageAccounts/finopshubpoc432XXXXXXXX",
"events": [
"Microsoft.Storage.BlobCreated"
]
}
}
}

[erincon01]

@BMichaelJ , attached the two pipelines. it's the code i fixed for v.0.1.1.
Make sure you enable the trigger, and allow empty files.

if you cannot import the files, let me know, and I can share the json if needed.

msexports_ETL_ingestion.zip
msexports_ExecuteETL.zip

[BMichaelJ]

Thanks @erincon01
Just to verify, this should be included in hub 0.2?
That is where my errors appear now.
I am still working on getting a successful 0.2 greenfield installation up.

[erincon01]

Hi @BMichaelJ.

I am not sure if it should be included as I am not in the DevTeam, and I am not very familiar with github and PRs, but in my case, the changes I made in the pipeline just worked fine with my v.0.2 version.

I had to edit the Power Query transformations in regards the data filters (starDate and EndDate), but after that, all worked just fine.
If you read my comments 1-2 days ago regarding the x_*** columns, all is fixed after the FOCUS format in the exports.

[flanakin]

@BMichaelJ 0.2 had 2 bugs that have been fixed and will go out in the 0.2.1 release. There are a few other things blocking that, but you can get the release candidate from https://aka.ms/finops/hubs/rc.

@erincon01 Thanks for sharing! These changes make hubs 0.1.1 work with 0.2 Power BI reports and trigger based on the manifest? With the 0.2 bugs resolved, we shouldn't need the former. I would point anyone who needs 0.1.1 to use the 0.1.1 Power BI reports to avoid any confusion. And Brett already has a PR out for the triggering on the manifest change. That should go out in 0.3. Is there anything else in there?

I'm happy to setup some time to show you how to submit a PR, if you'd like 🙂 Contributions are always welcome!

[BMichaelJ]

@flanakin thanks for the rc link, we have the hub running and the reports working.

[erincon01]

I close this too. The trigger thing is fixed in the deploy you shared on friday.

[erincon01]

I'm happy to setup some time to show you how to submit a PR, if you'd like 🙂 Contributions are always welcome!

I'd be happy to learn how to contribute. However, I am new in github :)
meanwhile, docs, or videos would help to fill the gap.

Thanks.