Open Source Quick Start: Redshift support

[ebedford420]

No description provided.

[lexaneon]

hey @ebedford420 ,
could you please tell me where I can find more about this ticket? if I understood correctly this ticket about "how to populate Redshift from snowplow"? I'm currently working on this task (Snowplow -> Redshift) and I plan to write some notes how this works (for further support). Maybe if my doc will be useful I can adjust it and share with you for further publishing.

[ebedford420]

Hi @lexaneon ! You can already load to Redshift with Snowplow using the RDB loader. The docs will give you more information on how it works and how to implement.

This ticket is specifically to do with our open source quick start. Previously with our quick start you were only able to load to Postgres, or Elasticsearch, and so this ticket is related to adding Redshift support in our open source quick start (by having a module for the RDB loader). I hope that makes sense and let me know if this is of interest to you

[lexaneon]

@ebedford420 thanks for the reply,
yes I saw the page with RDB loader and I use it, just wanted to say that some information is missed there (for instance: list of parameters for streaming jobs, services which should be created on AWS side, credentials for services, terraform scripts for them). Maybe you already have this information in different documents, however, from my perspective, it will be tangible to have most of them in this document.
Btw could you please share why you decided to split loading to RDB for 2 jobs (enrich & load - separate java app)? From my perspective it can be done simpler - just with Kinesis Analytics (Apache Flink) and as result for streaming part will be achieved: scalability & simplicity.