Performance issues with high memory consumption and CPU load for medium-to-large stacks

[hcharley]

Description of topic

I am dealing with RAM/CPU (memory/load) issues with my Node.JS / Typescript Automation API stack when it grows beyond a certain size, and this discussion is for those having similar issues with memory & CPU for any Pulumi user (not just those with my stack).

I am not creating an issue, because I don't have a ready-made reproduction, nor do I have code samples. Instead, I would like to begin to collate various issues/PRs that I've been reading that all touch on this issue.

Observed output

According to PM2, as soon as my Pulumi "program" is finished running (for a preview), there is a consumption of greater than 100% (~135%-145%) CPU and with an available memory of 24GB, it quickly goes up to 14GB and above. I used node options to increase the max-old-space-size to 75% of my machine's total RAM (32 GB).

PM2 logged:

terminate called after throwing an instance of 'std::bad_alloc'
std::bad_alloc
exited with code [0] via signal [SIGABRT]

When pausing the script shortly after a hang, the serializeProperty seems to be running according to the heap profile:

node_modules/.pnpm/@[email protected]/node_modules/@pulumi/pulumi/runtime/rpc.js

Modified behavior

I have tried selectively targeting single URNs, but even that still seems to trigger the issue.

Environment

I was using the latest Pulumi version, but I just switched the latest alpha to see if that would help. Here are my versions now:

    "@pulumi/gcp": "^6.20.0",
    "@pulumi/kubernetes": "^3.18.3",
    "@pulumi/postgresql": "^3.4.0",
    "@pulumi/pulumi": "^3.31.0-alpha.1650929512",
    "@pulumi/random": "^4.5.0",

pnpm 6.32.10

pulumi about
CLI          
Version      3.31.0-alpha.1650929512
Go Version   go1.17.9
Go Compiler  gc

Host     
OS       debian
Version  11.3
Arch     x86_64

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/abc
User           abc
Organizations  x, y, z

Ran inside a Github Codespace with 32 GB RAM, 16 cores, 128 storage

Hunches

• My best guess is that some "serialization" or "unmarshaling" process is going wild due to lots of resource dependencies.
• An issue with the RPC could be causing this issue, expanding the impact beyond just the NodeJS Automation API, to other languages such as Go, Python, etc.
• I have recorded log points showing that parent URNs can't be looked up
• It could be promise leaks or infinite promises being created
• I am thinking that the Kubernetes YAML directory output could help save me since I have a lot of Kubernetes resources, and each of them have a lot of metadata associated with them
• Could some of the metadata associated with Kubernetes resources be removed (like last-applied-configuration)?
  • See pulumi config --show-secrets should use batch-decrypt #11850  #8677 (comment)
• Could it be that high network transfer (uploads specifically) are causing a compounding/confounding issue leading to high memory and CPU consumption?
  • See pulumi up , high upload network needs (remote backend -> GCS) #9190 (comment)
  • Pulumi eats bandwidth (when using S3 backend) #6074
  • Support optional compression of remote state JSON #5469

Related issues & pull requests

Newest first:

• Go SDK does not properly return errors to client code #9472*
• Fix race condition in reading EngineEvents from tailed logs #9467*
• Multi-Language Component Cannot be a Parent of Another Resource - Causes Promise Leak #9457
• Pulumi bails with no error messages (exit code 255) #9386
• CLI Performance Issues when using stackReferences #9350
• Very slow pulumi up leads to expired token error #9348
• Kubernetes Operator's Memory usage is higher > 2 Gi pulumi-kubernetes-operator#262
• Node.js closure serialization doesn't capture method bodies #9168
• Debug print incoming protos #9119
• Correctly await outputs containing resources #9106
• Use SWC to transpile typescript projects #9049*
• Using azblob backend, pulumi stack ls runs quite slowly #8872
• [codegen/go] pulumiTypes.go is too large to compile #8950
• Fixes to KillChildren function #8709
• pulumi config --show-secrets should use batch-decrypt #11850  #8677
• tsconfig "module" conflict: project needs "esnext", pulumi needs "commonjs" #8415*
• Make ts-node and typescript peer dependencies #8199*
• pulumi {preview/up/...} is slow with medium sized TS stack #7682
• Initial support for (un)marshaling output values #7861
• [sdk/go] Marshal output values #7958
• Pulumi 4.0 Changes Tracking #7346*
• Document immutable ConfigMap/Secret behavior for our provider pulumi-kubernetes#1568
• Pulumi became so slow (~7x times) since version 2.16.2 #6197
• Pulumi eats bandwidth (when using S3 backend) #6074*
• Support optional compression of remote state JSON #5469*
• Preview/Up Slowness #5001
• Revisit ts-node version #4876*
• Improve stability and functionality of self-managed state backends #4605*
• Pulumi CPU/Memory issue. #3987
• Memory leak on windows #2514
• Improving dependencies and lifecycles of ComponentResources. #2484 (comment)*
• Fix a "hang" when tsconfig.json is not present in the CWD #1857 (comment)*
• Evaluation can lead to space leaks #215

* Not strictly specific to performance issues, but collating here for awareness.

Searches

• impact/performance Something is slower than expected
• impact/non-specific Unreliable, occurs occasionally
• Duplicate issues

[hcharley]

I think I may have found my issue.

Circular serialization of objects passed into a ComponentResource, despite not being included in the output or args, could be causing infinite serialization.

[hcharley]

I am going to try and use doNotCapture to suppress some of this data from serialization.

[hcharley]

I do not know how to pass a custom serialize function. And doNotCapture does not appear to work how I thought it would.

[hcharley]

Okay, so by pre-serializing an object I had been using to construct the containers field in a k8 deployment into a JSON string and back into a JS object, I was able to avoid infinite serialization.

TIL (and should have always been doing, as its clean!): always give your pulumi component resources a simple DTO to consume, never a method that uses a JS class instance that returns an array.

[XBeg9]

Team, in the latest releases of pulumi something really wrong 3.34+, getting Killed or OOM easily on a stack that was working fine before.

Thanks

[rshade]

are you on 3.35.3? Can you try this, we found an alias issue causing a lot of memory usage?

[vtereshyn]

I am using pulumi 3.48.0 and node 18.12.0. Also, I am using the latest version of Typescript with EcmaScript modules, and issue with OOM has become pretty common. @rshade

[rshade]

@vtereshyn can you open an issue on the pulumi repo with some details so we can try to recreate?

[vtereshyn]

@rshade done – #11476