Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

field-id's missing in generated Avro files #353

Open
Tracked by #348
Fokko opened this issue Apr 25, 2024 · 2 comments
Open
Tracked by #348

field-id's missing in generated Avro files #353

Fokko opened this issue Apr 25, 2024 · 2 comments
Assignees

Comments

@Fokko
Copy link
Contributor

Fokko commented Apr 25, 2024

Looks like the field-id is missing in the Avro metadata generated by iceberg-rust. This should conform the spec: https://iceberg.apache.org/spec/#avro

avro-tools getschema snap-0-1-a0c0a37d-6828-47c6-80e8-64b0a013fe44.avro
24/04/25 14:58:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
{
  "type" : "record",
  "name" : "manifest_file",
  "fields" : [ {
    "name" : "manifest_path",
    "type" : "string"
  }, {
    "name" : "manifest_length",
    "type" : "long"
  }, {
    "name" : "partition_spec_id",
    "type" : "int"
  }, {
    "name" : "content",
    "type" : "int"
  }, {
    "name" : "sequence_number",
    "type" : "long"
  }, {
    "name" : "min_sequence_number",
    "type" : "long"
  }, {
    "name" : "added_snapshot_id",
    "type" : "long"
  }, {
    "name" : "added_data_files_count",
    "type" : "int"
  }, {
    "name" : "existing_data_files_count",
    "type" : "int"
  }, {
    "name" : "deleted_data_files_count",
    "type" : "int"
  }, {
    "name" : "added_rows_count",
    "type" : "long"
  }, {
    "name" : "existing_rows_count",
    "type" : "long"
  }, {
    "name" : "deleted_rows_count",
    "type" : "long"
  }, {
    "name" : "partitions",
    "type" : [ "null", {
      "type" : "array",
      "items" : {
        "type" : "record",
        "name" : "r508",
        "fields" : [ {
          "name" : "contains_null",
          "type" : "boolean"
        }, {
          "name" : "contains_nan",
          "type" : [ "null", "boolean" ],
          "default" : null
        }, {
          "name" : "lower_bound",
          "type" : [ "null", "bytes" ],
          "default" : null
        }, {
          "name" : "upper_bound",
          "type" : [ "null", "bytes" ],
          "default" : null
        } ]
      }
    } ],
    "default" : null
  }, {
    "name" : "key_metadata",
    "type" : [ "null", "bytes" ],
    "default" : null
  } ]
}
@s-akhtar-baig
Copy link
Contributor

@Fokko, can you please assign this to me? Thanks!

@Fokko
Copy link
Contributor Author

Fokko commented Apr 25, 2024

@s-akhtar-baig Certainly 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants