Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: File closed twice #17549

Open
2 tasks done
candiduslynx opened this issue Apr 8, 2024 · 1 comment
Open
2 tasks done

bug: File closed twice #17549

candiduslynx opened this issue Apr 8, 2024 · 1 comment
Labels
area/plugin/destination/azblob area/plugin/destination/file area/plugin/destination/gcs area/plugin/destination/s3 kind/bug

Comments

@candiduslynx
Copy link
Contributor

candiduslynx commented Apr 8, 2024 •
edited

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

2024-04-08T10:51:14Z ERR error from StreamingBatchWriter error="handler failed on test_testdata_table: close test_testdata_table.parquet: file already closed" invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=file

Expected Behavior

No issue on the file writing

CloudQuery (redacted) config

kind: source
spec:
  name: test
  registry: "github"
  path: "cloudquery/test"
  version: "v4.1.0"
  tables:
  - "test_testdata_table"
  destinations:
  - file
---
kind: destination
spec:
  name: "file"
  path: "cloudquery/file"
  registry: "cloudquery"
  version: "v4.0.6"
  write_mode: "append"
  spec:
    path: ./{{TABLE}}.{{FORMAT}}
    format: "parquet"
    no_rotate: true

Steps To Reproduce

No response

CloudQuery (redacted) logs

Logs 
2024-04-08T10:51:07Z INF Loading spec(s) args=["./file.yaml","./test.yaml"] invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli
2024-04-08T10:51:07Z WRN registry: github is deprecated & will be removed in future releases invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 kind=source module=cli name=test
2024-04-08T10:51:09Z INF Plugin server listening address=/var/folders/xq/3kqvky950gs9yxllghq1cw2r0000gn/T/cq-qKwpeViRnMWwlcwG.sock invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli
2024-04-08T10:51:13Z INF Plugin server listening address=/var/folders/xq/3kqvky950gs9yxllghq1cw2r0000gn/T/cq-tGyJxGZzENJoiGrB.sock invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.076 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.176 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.036 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.136 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF Start sync destinations=["file (cloudquery/[email protected])"] invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli source="test@github (cloudquery/[email protected])" sync_time=2024-04-08T10:51:14Z
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.01 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.072 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.013 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.627 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.022 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetSpecSchema grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.082 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.109 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=6.309 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF Start fetching resources destinations=["file (cloudquery/[email protected])"] invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli source="test@github (cloudquery/[email protected])"
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.078 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF top level table resolver started client=Client#2 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=test-src table=test_testdata_table
2024-04-08T10:51:14Z INF table sync finished client=Client#2 errors=0 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=test-src resources=1 table=test_testdata_table
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=2.929 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=4.489 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z ERR error from StreamingBatchWriter error="handler failed on test_testdata_table: close test_testdata_table.parquet: file already closed" invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=file
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=18.937 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF started call grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.005 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF finished call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2024-04-08T13:51:14+03:00 grpc.time_ms=0.024 invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli peer.address= protocol=grpc
2024-04-08T10:51:14Z INF End sync destinations=["file (cloudquery/[email protected])"] invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli source="test@github (cloudquery/[email protected])" sync_time=2024-04-08T10:51:14Z
2024-04-08T10:51:14Z INF waiting for source plugin to terminate invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli
2024-04-08T10:51:14Z INF Got stop signal. Plugin server shutting down address=/var/folders/xq/3kqvky950gs9yxllghq1cw2r0000gn/T/cq-tGyJxGZzENJoiGrB.sock invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli signal=interrupt
2024-04-08T10:51:14Z INF waiting for source plugin to terminate invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli
2024-04-08T10:51:14Z INF Got stop signal. Plugin server shutting down address=/var/folders/xq/3kqvky950gs9yxllghq1cw2r0000gn/T/cq-qKwpeViRnMWwlcwG.sock invocation-id=0868a72c-2881-4179-b2ca-53f422c91a16 module=cli signal=interrupt

CloudQuery version

5.11.0

Additional Context

No response

Pull request (optional)

  • I can submit a pull request
@candiduslynx
Copy link
Contributor Author

I presume this goes for all filetypes-based plugins

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/plugin/destination/azblob area/plugin/destination/file area/plugin/destination/gcs area/plugin/destination/s3 kind/bug
Projects
Framework
Status: No status
Milestone
No milestone
Development

No branches or pull requests
1 participant
@candiduslynx