Data engineering project using Azure Data Factory and Apache Airflow
Disaster Response Pipeline | Data Engineering
Spotify API, Airflow, Docker, AWS S3, Snowflake, dbt, localstack, Looker Studio
This project repo 📺 offers a robust solution meticulously crafted to efficiently manage, process, and analyze YouTube video data leveraging the power of AWS services. Whether you're diving into structured statistics or exploring the nuances of trending key metrics, this pipeline is engineered to handle it all with finesse.
Pipeline to automate the collection of board game and expansion data from BoardGameGeek's XML API2. Data is stored in Google Cloud Storage and BigQuery. Data is modelled using DBT in a star schema. (Terraform, GCP, Mage, Python, dbt)
Ayush @ Data Engineering Portfolio
Building Machine Learning and ETL Pipelines to categorize emergency messages based on the needs communicated by the sender
Two data frames of different kaggle cases of diesease cases and weather in Brazil. The project aims to clean the DFs and build a new one in order to analyse the correlation of dengue (serious disease transmited by mosquito), rain precipitation and temperature.
Analysis of NYC's citibike data. Technologies: Python , Prefect, dbt, Terraform , Looker data studio
Data Engineering Zoomcamp course assignments and notes.
A quick implementation of OCR Application with AWS Lambda.
IGTI Enhenheiro de Dados - Módulo 5 Desafio Final
Tiny Blocks to build large and complex data pipelines!
cryptocurrency ticker data pipeline
The Security Reference Architecture (SRA) implements typical security features as Terraform Templates that are deployed by most high-security organizations, and enforces controls for the largest risks that customers ask about most often.
Presentations/tutorials delivered by me at various conferences 👩🏽💻
Irish Property Price Register transformed into a data warehouse via an EtLT pipeline.
ETL Pipeline for Music Analysis
Add a description, image, and links to the data-engineering-pipeline topic page so that developers can more easily learn about it.
To associate your repository with the data-engineering-pipeline topic, visit your repo's landing page and select "manage topics."