Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: prevent failovers when disk space is exhausted #4404

Open
wants to merge 23 commits into
base: main
Choose a base branch
from

Conversation

leonardoce
Copy link
Contributor

@leonardoce leonardoce commented Apr 29, 2024

PostgreSQL will shut down cleanly when there is not enough disk space to store WAL files.

The operator did not recognize this condition and, since the primary failed, was performing a failover to the most advanced replica. This action will not fix the underlying issue.

Only a manual disk resize, initiated by the user, can ultimately lead to a fully working PostgreSQL cluster.

This patch makes the instance manager recognize this condition and report it to the operator. Upon detecting it, the operator will not trigger a switchover and set a phase describing the situation.

After the PVCs are resized, the cluster will restart working correctly.

Closes: #4521

@github-actions github-actions bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.21 release-1.22 release-1.23 labels Apr 29, 2024
Copy link
Contributor

❗ By default, the pull request is configured to backport to all release branches.

  • To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
  • To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

@leonardoce leonardoce force-pushed the dev/space branch 3 times, most recently from 4012706 to 126566d Compare May 13, 2024 09:54
@leonardoce leonardoce marked this pull request as ready for review May 13, 2024 10:02
@leonardoce leonardoce requested a review from a team as a code owner May 13, 2024 10:02
@leonardoce
Copy link
Contributor Author

I tested this using Longhorn in a Fedora VM, but any storage enforcing the PV capacity will do the trick.

To test the patch, you need to finish your WAL storage. To keep things easy, I used:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cluster-example
spec:
  instances: 1

  storage:
    size: 256Mi

And then:

CREATE TABLE storage_area (t text);

-- repeat the following query 20-30 times (you need to be fast!)
INSERT INTO storage_area (t) (select repeat('Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do', 5*1024*1024));

With the predefined WAL settings, you'll finish your WAL disk space before you finish the space for PGDATA.

@armru armru force-pushed the dev/space branch 2 times, most recently from 9b2cea6 to e625d8f Compare May 15, 2024 15:25
@armru
Copy link
Member

armru commented May 16, 2024

/test limit=local

Copy link
Contributor

@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/9110497781

@github-actions github-actions bot added the ok to merge 👌 This PR can be merged label May 16, 2024
Copy link
Collaborator

@jsilvela jsilvela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About to start adding documentation and going over the E2E, but left a few comments on the implementation bits.
IMO the "WALDisk" nomenclature could get confusing as it seems to imply there is a separate WAL volume, which may or may not be the case.

internal/cmd/manager/instance/run/lifecycle/run.go Outdated Show resolved Hide resolved
pkg/fileutils/directory.go Outdated Show resolved Hide resolved
pkg/fileutils/directory.go Outdated Show resolved Hide resolved
pkg/fileutils/directory.go Outdated Show resolved Hide resolved
pkg/management/postgres/instance.go Outdated Show resolved Hide resolved
pkg/management/postgres/instance.go Outdated Show resolved Hide resolved
pkg/utils/fencing.go Outdated Show resolved Hide resolved
@leonardoce leonardoce force-pushed the dev/space branch 2 times, most recently from 172cfe7 to 3cdc43a Compare May 21, 2024 09:47
Copy link
Collaborator

@jsilvela jsilvela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still think it's worth renaming the ensureSufficientDiskSpace method, but otherwise give this an enthusiastic 👍

fcanovai and others added 22 commits June 4, 2024 09:31
Add an e2e to test the recovery in case a primary runs out of disk
space.

Signed-off-by: Francesco Canovai <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Jaime Silvela <[email protected]>
Co-authored-by: Jaime Silvela <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Jaime Silvela <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Co-authored-by: Jaime Silvela <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Jaime Silvela <[email protected]>
Signed-off-by: Leonardo Cecchi <[email protected]>
Signed-off-by: Gabriele Bartolini <[email protected]>
@leonardoce
Copy link
Contributor Author

We removed the WALSpaceAvailable because we're worried that it could be misunderstood and used as a metric instead of proper monitoring infrastructure.

The operator sets this value from the exit code of the instance Pods. The instance Pods check whether they have enough WAL disk space before starting PG and after PostgreSQL exits with an error condition.
This is not a way to monitor disk space.

@leonardoce
Copy link
Contributor Author

@leonardoce leonardoce added do not backport This PR must not be backported - it will be in the next minor release and removed backport-requested ◀️ This pull request should be backported to all supported releases release-1.21 release-1.22 release-1.23 labels Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do not backport This PR must not be backported - it will be in the next minor release do not merge 🙅 This PR cannot be merged (yet) ok to merge 👌 This PR can be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature]: Graceful handling of WAL disk space exhaustion
5 participants