Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting timeout exception for s3 source connector, has to do with tasks.file.status.storage.bootstrap.servers.. #578

Open
agavili opened this issue Nov 26, 2023 · 1 comment
Labels
question Further information is requested wontfix This will not be worked on

Comments

@agavili
Copy link

agavili commented Nov 26, 2023

Setup Details
I'm running FilePulse 2.13.0 (https://github.com/streamthoughts/kafka-connect-file-pulse/releases/download/v2.13.0/streamthoughts-kafka-connect-file-pulse-2.13.0.zip) on Mac

Concern
I'm trying to create a s3 source connector using filepulse and this connector is linked to a topic in IBM Cloud's Event Streams. I'm not entirely sure what to set tasks.file.status.storage.bootstrap.servers to. When I set it to localhost:9092, I get a timeout exception. The same occrs when I set the field to the cluster's internal listener: development-kafka-bootstrap.cp4i.svc:9093. If I comment out the field, I get a Failed to create shared StateBackingStore for group 'connect-file-pulse-amazon-s3-csv' error. What should I set it to? Is there any other configuration I'm missing?

Here's my connector's yaml:

apiVersion: eventstreams.ibm.com/v1beta2
kind: KafkaConnector
metadata:
  name: connect-file-pulse-amazon-s3-csv
  labels:
    eventstreams.ibm.com/cluster: my-connect-cluster
spec:
  class: io.streamthoughts.kafka.connect.filepulse.source.FilePulseSourceConnector
  tasksMax: 1
  config:
    topic: XXXX
    tasks.max: 1
    fs.listing.class: io.streamthoughts.kafka.connect.filepulse.fs.AmazonS3FileSystemListing
    fs.listing.interval.ms: 10000
    # fs.listing.filters: io.streamthoughts.kafka.connect.filepulse.scanner.local.filter.IgnoreHiddenFileListFilter
    file.filter.regex.pattern: .*\\.csv$

    fs.cleanup.policy.class: io.streamthoughts.kafka.connect.filepulse.fs.clean.LogCleanupPolicy 

    aws.access.key.id: XXXX
    aws.secret.access.key: XXXX
    aws.s3.region: us-east-2
    aws.s3.bucket.name: XXXX

    tasks.reader.class: io.streamthoughts.kafka.connect.filepulse.fs.reader.AmazonS3RowFileInputReader

    skip.headers: 1
    offset.attributes.string: uri

    filters: ParseLine
    filters.ParseLine.type: io.streamthoughts.kafka.connect.filepulse.filter.DelimitedRowFilter
    filters.ParseLine.extractColumnName: headers
    filters.ParseLine.trimColumn: true
    filters.ParseLine.separator: ;
    tasks.file.status.storage.bootstrap.servers: localhost:9092
    tasks.file.status.storage.topic: connect-file-pulse-status
    tasks.file.status.storage.topic.partitions: 10
    # tasks.file.status.storage.topic.replication.factor: 1
@agavili agavili added the question Further information is requested label Nov 26, 2023
Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the wontfix This will not be worked on label Feb 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

1 participant