Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use google xml upload api for protected gcs buckets #1413

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Use google xml upload api for protected gcs buckets #1413

wants to merge 5 commits into from

Conversation

araji-collibra
Copy link

Database name

Wal-g provides support for many databases, please write down name of database you uses.
in this case postgres but gcs storage is the issue .

Pull request description

Allow pushing backups to protected GCS buckets.

Describe what this PR fix

Protected buckets do not allow chunk deletion so we switch to uploading the data using an xml API when that is the case.

Please provide steps to reproduce (if it's a bug)

set the GCS bucket to have any form of retention policy .

Please add config and wal-g stdout/stderr logs for debug purpose

no change to wal-g config

logs:

basebackups_005/base_000000010000000000000060/tar_partitions/pg_control.tar.lz4
uploadId = ABPnzm75msppsvmrENc240IFoiSM5_ZTDiFZ081CEwyosu_qcH-HfkOeqBG_obORvDMZSVw
Uploading part: 1 of uploadID: ABPnzm75msppsvmrENc240IFoiSM5_ZTDiFZ081CEwyosu_qcH-HfkOeqBG_obORvDMZSVw

PART: "b2bbecf877eb3510b72f57e8bb8c79b4", 1
Completing Upload
uploading completion summary:
 <CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"b2bbecf877eb3510b72f57e8bb8c79b4"</ETag></Part></CompleteMultipartUpload>

POST /basebackups_005/base_000000010000000000000060/tar_partitions/pg_control.tar.lz4?uploadId=ABPnzm75msppsvmrENc240IFoiSM5_ZTDiFZ081CEwyosu_qcH-HfkOeqBG_obORvDMZSVw HTTP/1.1
Host: pitr-d557ee04-9f13-5d75-8b03-3b55a8194880.storage.googleapis.com
Content-Length: 145
Content-Type: application/xml
Date: Mon, 05 Dec 2022 19:33:24 UTC

<CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"b2bbecf877eb3510b72f57e8bb8c79b4"</ETag></Part></CompleteMultipartUpload>
Response Code:200
Response body::
<?xml version='1.0' encoding='UTF-8'?><CompleteMultipartUploadResult xmlns='http://s3.amazonaws.com/doc/2006-03-01/'><Location>http://storage.googleapis.com/pitr-d557ee04-9f13-5d75-8b03-3b55a8194880/basebackups_005/base_000000010000000000000060/tar_partitions/pg_control.tar.lz4</Location><Bucket>-pitr-d557ee04-9f13-5d75-8b03-3b55a8194880</Bucket><Key>basebackups_005/base_000000010000000000000060/tar_partitions/pg_control.tar.lz4</Key><ETag>"341ba4c49e977a3177a7e2aea9b37386-1"</ETag></CompleteMultipartUploadResult>
completeUpload response:
 &{{http://s3.amazonaws.com/doc/2006-03-01/ CompleteMultipartUploadResult} http://storage.googleapis.com/pitr-d557ee04-9f13-5d75-8b03-3b55a8194880/basebackups_005/base_000000010000000000000060/tar_partitions/pg_control.tar.lz4 pitr-d557ee04-9f13-5d75-8b03-3b55a8194880 basebackups_005/base_000000010000000000000060/tar_partitions/pg_control.tar.lz4 "341ba4c49e977a3177a7e2aea9b37386-1"}

@araji-collibra araji-collibra requested a review from a team as a code owner January 11, 2023 16:01
@araji-collibra araji-collibra mentioned this pull request Jan 11, 2023
Open
@msvorad
Copy link

msvorad commented Mar 15, 2023

I've tried this build and it does not retry 502 error (at least this one seems retryable):

2023/03/14 17:48:05 expected status 2xx, got 502 Bad Gateway (<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 502 (Server Error)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>502.</b> <ins>That’s an error.</ins>
  <p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.  <ins>That’s all we know.</ins>
)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@araji-collibra @msvorad