-
Notifications
You must be signed in to change notification settings - Fork 79
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Does Kinesis connector support reading compressed payloads on stream? #105
Comments
Hello! I have been using this connector and I haven't seen a configuration for automatically unzipping the content. If you are expecting data from kinesis as GZIP compressed, you can use the following snippet to uncompress it in Python.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
We currently use the Qubole open source Kinesis connector in a Spark structured streaming application on AWS, to read payloads from the Kinesis stream, and generate Parquet files. Would it be possible to use Qubole Kinesis connector for reading compressed payloads (say, in GZip format), that have been posted to the stream using the same compression format as is recognized from the reading side (i.e. in terms of Kinesis Qubole source)? Thanks.
The text was updated successfully, but these errors were encountered: