Skip to content

Export Dialogflow conversation logs to BigQuery with masking PII using DLP API

License

Notifications You must be signed in to change notification settings

crosslibs/export-dialogflow-logs-to-bigquery

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

34 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Export Dialogflow conversations to BigQuery

Build Status Coverage Status

Export Dialogflow conversations to BigQuery after masking SPII data using DLP API

Solution Approach

Dialogflow Fulfillment mechanism is used to intercept all the conversations using a webhook (deployed as a GAE app). The conversations are logged asynchronously after masking the PII data using Google App Engine and DLP API to a PubSub topic. From there each conversation is then inserted into BigQuery using Dataflow job. (#streaming inserts to BQ)

Architecture

Export Conversations to BQ

Contributions Welcome

Please feel free to contribute to the code base by submitting a pull request.