Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic interface conversion error message on migrate from v4 to v5 #1987

Open
Rus1an31 opened this issue Apr 29, 2024 · 7 comments
Open

Panic interface conversion error message on migrate from v4 to v5 #1987

Rus1an31 opened this issue Apr 29, 2024 · 7 comments
Labels
awaiting-feedback impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec

Comments

@Rus1an31
Copy link

Rus1an31 commented Apr 29, 2024 •
edited by thomas11

What happened?

After changing code to upgrade pulumi/azure from v4 to v5, pulumi preview command failed with error message "panic: interface conversion: interface {} is nil, not []interface {}". Full stack trace:

Diagnostics:
azure:containerservice:KubernetesCluster (danfoss-webapik8s-dev):
error: transport is closing

pulumi:pulumi:Stack (webapik8s-dev):
panic: interface conversion: interface {} is nil, not []interface {}
goroutine 64 [running]:
github.com/hashicorp/terraform-provider-azurerm/internal/services/containers/migration.KubernetesClusterV1ToV2.UpgradeFunc.func1({0xc5b5d60, 0xc0000c3a18}, 0x5d16da0, {0xc001bc4090, 0x6f4f8a0})
/home/runner/go/pkg/mod/github.com/pulumi/[email protected]/internal/services/containers/migration/kubernetes_cluster.go:1231 +0x30a
github.com/hashicorp/terraform-provider-azurerm/internal/tf/pluginsdk.StateUpgrades.func1({0x6f4c298, 0xc00005a1c8}, 0x2, {0x5cce140, 0xc0016f8e00})
/home/runner/go/pkg/mod/github.com/pulumi/[email protected]/internal/tf/pluginsdk/state_upgrades.go:52 +0x64
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.UpgradeJSONState({0x6f4c298, 0xc00005a1c8}, 0x0, 0xc000fbbb60, 0xc000c32dd0, {0x5cce140, 0xc0016f8e00})
/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:424 +0xa7
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.upgradeResourceState(0xc000476730, 0xc0005d4cf0, 0xc001cb92b0)
/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/upgrade_state.go:44 +0x205
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Diff({0xc0016d2240}, {0x66e662a, 0x1a}, {0x6f515f0, 0xc001da1140}, {0x6ef3580, 0xc0015a0960})
/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/provider.go:95 +0xda
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Diff(0xc0016d2240, {0x6f4c308, 0xc000e80c30}, 0xc0009f01c0)
/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/provider.go:746 +0x58c
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0x6f4c308, 0xc000e80c30}, {0x62837e0, 0xc0009f01c0})
/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider.pb.go:2611 +0x78
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x6f4c308, 0xc000c02ae0}, {0x62837e0, 0xc0009f01c0}, 0xc00153dc40, 0xc001c88d80)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x425
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0x63a3000, 0xc0016d2240}, {0x6f4c308, 0xc000c02ae0}, 0xc001ba2de0, 0xc00177b840)
/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider.pb.go:2613 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002e6700, {0x6fa5778, 0xc0012ef200}, 0xc001ca6900, 0xc00176aea0, 0xc56e8e8, 0x0)
/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1292 +0xc6f
google.golang.org/grpc.(*Server).handleStream(0xc0002e6700, {0x6fa5778, 0xc0012ef200}, 0xc001ca6900, 0x0)
/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1617 +0xa2a
google.golang.org/grpc.(*Server).serveStreams.func1.2()
/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:940 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:938 +0x294

Example

Definition of cluster:

const cluster = new azure.containerservice.KubernetesCluster(
    resourceName,
    {
      name: resourceName,
      resourceGroupName: resourceGroup.name,
      linuxProfile: {
        adminUsername: 'some_admin_name',
        sshKey: {
          keyData: sshKey
        }
      },
      dnsPrefix: resourceName,
      defaultNodePool: {
        name: 'linux1',
        vmSize: 'Standard_B2s',
        nodeCount: 10,
        osDiskSizeGb: 100,
        enableNodePublicIp: false,
        type: 'VirtualMachineScaleSets',
        enableAutoScaling: false,
        orchestratorVersion: '1.28.5',
        upgradeSettings: {
          maxSurge: '1'
        }
      },
      kubernetesVersion: '1.28.5',
      roleBasedAccessControlEnabled: true,
      azureActiveDirectoryRoleBasedAccessControl: {
        managed: true,
        adminGroupObjectIds: [group_here]
      },
      httpApplicationRoutingEnabled: true,
      azurePolicyEnabled: stack !== 'sandbox',
      networkProfile: {
        networkPlugin: 'azure',
        loadBalancerSku: 'Standard'
      },
      servicePrincipal: {
        clientId: clientId!,
        clientSecret: clientSecret!
      }
    }

Output of pulumi about

CLI
Version 3.17.0
Go Version go1.17.2
Go Compiler gc

Plugins
NAME VERSION
azure 5.0.0
azuread 5.48.0
kubernetes 3.21.4
nodejs unknown

Host
OS ubuntu
Version 22.04
Arch x86_64

This project is written in nodejs (/opt/hostedtoolcache/node/18.15.0/x64/bin/node v18.15.0)

NAME VERSION
@pulumi/azure 5.0.0
@pulumi/kubernetes 3.21.4
@types/node 16.11.26
@pulumi/azuread 5.48.0
@pulumi/pulumi 3.89.0
typescript 5.4.5

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@Rus1an31 Rus1an31 added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Apr 29, 2024
@thomas11
Copy link
Contributor

Hi @Rus1an31, sorry you're running into this. We'll be looking into it as soon as possible.

In the meantime, would you be able to try again with a recent version of the Pulumi Azure Classic provider? Your current version 5.0.0 is about two years old and many fixes have gone into later versions.

@thomas11 thomas11 added impact/panic This bug represents a panic or unexpected crash p1 Bugs severe enough to be the next item assigned to an engineer and removed needs-triage Needs attention from the triage team labels Apr 30, 2024
@Rus1an31
Copy link
Author

Hi @thomas11
I tried with the last version 5.74.0. Unfortunately, the same error message.

@thomas11
Copy link
Contributor

Thanks for checking, @Rus1an31. Could you post the first few lines of the new stack trace to get accurate line numbers for debugging?

It would also be very helpful if you could attach the output of pulumi stack export. Secrets will not be shown.

@danielrbradley
Copy link
Member

What specific version of 4.x are you migrating from? If you're not already using it, upgrading to v4.42.0 first might possibly help to sort out the state.

@Rus1an31
Copy link
Author

Rus1an31 commented May 3, 2024 •
edited

Hi @thomas11
Lines of new stack trace:

2024-04-29T09:15:35.1943605Z pulumi:pulumi:Stack webapik8s-dev running
2024-04-29T09:15:46.8497799Z azure:core:ResourceGroup danfoss-webapik8s-dev-rg [diff: ~provider]
2024-04-29T09:15:46.8677220Z azure:network:PublicIp danfoss-webapik8s-devstaticIngressIP [diff: ~provider]
2024-04-29T09:15:46.8797116Z pulumi:pulumi:Stack webapik8s-dev running panic: interface conversion: interface {} is nil, not []interface {}
2024-04-29T09:15:46.8798611Z pulumi:pulumi:Stack webapik8s-dev running goroutine 64 [running]:
2024-04-29T09:15:46.8799956Z pulumi:pulumi:Stack webapik8s-dev running github.com/hashicorp/terraform-provider-azurerm/internal/services/containers/migration.KubernetesClusterV1ToV2.UpgradeFunc.func1({0xc5b5d60, 0xc0000c3a18}, 0x5d16da0, {0xc001bc4090, 0x6f4f8a0})
2024-04-29T09:15:46.8801022Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/github.com/pulumi/[email protected]/internal/services/containers/migration/kubernetes_cluster.go:1231 +0x30a
2024-04-29T09:15:46.8802065Z pulumi:pulumi:Stack webapik8s-dev running github.com/hashicorp/terraform-provider-azurerm/internal/tf/pluginsdk.StateUpgrades.func1({0x6f4c298, 0xc00005a1c8}, 0x2, {0x5cce140, 0xc0016f8e00})
2024-04-29T09:15:46.8802977Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/github.com/pulumi/[email protected]/internal/tf/pluginsdk/state_upgrades.go:52 +0x64
2024-04-29T09:15:46.8803921Z pulumi:pulumi:Stack webapik8s-dev running github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.UpgradeJSONState({0x6f4c298, 0xc00005a1c8}, 0x0, 0xc000fbbb60, 0xc000c32dd0, {0x5cce140, 0xc0016f8e00})
2024-04-29T09:15:46.8804831Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:424 +0xa7
2024-04-29T09:15:46.8805964Z pulumi:pulumi:Stack webapik8s-dev running github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.upgradeResourceState(0xc000476730, 0xc0005d4cf0, 0xc001cb92b0)
2024-04-29T09:15:46.8806775Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/upgrade_state.go:44 +0x205
2024-04-29T09:15:46.8807661Z pulumi:pulumi:Stack webapik8s-dev running github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Diff({0xc0016d2240}, {0x66e662a, 0x1a}, {0x6f515f0, 0xc001da1140}, {0x6ef3580, 0xc0015a0960})
2024-04-29T09:15:46.8808505Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/provider.go:95 +0xda
2024-04-29T09:15:46.8809305Z pulumi:pulumi:Stack webapik8s-dev running github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Diff(0xc0016d2240, {0x6f4c308, 0xc000e80c30}, 0xc0009f01c0)
2024-04-29T09:15:46.8810071Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/provider.go:746 +0x58c
2024-04-29T09:15:46.8811113Z pulumi:pulumi:Stack webapik8s-dev running github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0x6f4c308, 0xc000e80c30}, {0x62837e0, 0xc0009f01c0})
2024-04-29T09:15:46.8811862Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider.pb.go:2611 +0x78
2024-04-29T09:15:46.8812955Z pulumi:pulumi:Stack webapik8s-dev running github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x6f4c308, 0xc000c02ae0}, {0x62837e0, 0xc0009f01c0}, 0xc00153dc40, 0xc001c88d80)
2024-04-29T09:15:46.8813843Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x425
2024-04-29T09:15:46.8814711Z pulumi:pulumi:Stack webapik8s-dev running github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0x63a3000, 0xc0016d2240}, {0x6f4c308, 0xc000c02ae0}, 0xc001ba2de0, 0xc00177b840)
2024-04-29T09:15:46.8815815Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider.pb.go:2613 +0x138
2024-04-29T09:15:46.8816659Z pulumi:pulumi:Stack webapik8s-dev running google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002e6700, {0x6fa5778, 0xc0012ef200}, 0xc001ca6900, 0xc00176aea0, 0xc56e8e8, 0x0)
2024-04-29T09:15:46.8817531Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1292 +0xc6f
2024-04-29T09:15:46.8818265Z pulumi:pulumi:Stack webapik8s-dev running google.golang.org/grpc.(*Server).handleStream(0xc0002e6700, {0x6fa5778, 0xc0012ef200}, 0xc001ca6900, 0x0)
2024-04-29T09:15:46.8818938Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1617 +0xa2a
2024-04-29T09:15:46.8819549Z pulumi:pulumi:Stack webapik8s-dev running google.golang.org/grpc.(*Server).serveStreams.func1.2()
2024-04-29T09:15:46.8820156Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:940 +0x98
2024-04-29T09:15:46.8820767Z pulumi:pulumi:Stack webapik8s-dev running created by google.golang.org/grpc.(*Server).serveStreams.func1
2024-04-29T09:15:46.8821405Z pulumi:pulumi:Stack webapik8s-dev running /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:938 +0x294
2024-04-29T09:15:46.8849311Z azure:containerservice:KubernetesCluster danfoss-webapik8s-dev error: transport is closing
2024-04-29T09:15:46.9166617Z azure:containerservice:KubernetesCluster danfoss-webapik8s-dev 1 error
2024-04-29T09:15:46.9167275Z pulumi:pulumi:Stack webapik8s-dev 28 messages
2024-04-29T09:15:46.9167567Z
2024-04-29T09:15:46.9167779Z Diagnostics:
2024-04-29T09:15:46.9168199Z azure:containerservice:KubernetesCluster (danfoss-webapik8s-dev):
2024-04-29T09:15:46.9168544Z error: transport is closing
2024-04-29T09:15:46.9168774Z
2024-04-29T09:15:46.9169101Z pulumi:pulumi:Stack (webapik8s-dev):
2024-04-29T09:15:46.9169440Z panic: interface conversion: interface {} is nil, not []interface {}
2024-04-29T09:15:46.9169738Z goroutine 64 [running]:
2024-04-29T09:15:46.9170673Z github.com/hashicorp/terraform-provider-azurerm/internal/services/containers/migration.KubernetesClusterV1ToV2.UpgradeFunc.func1({0xc5b5d60, 0xc0000c3a18}, 0x5d16da0, {0xc001bc4090, 0x6f4f8a0})
2024-04-29T09:15:46.9171585Z /home/runner/go/pkg/mod/github.com/pulumi/[email protected]/internal/services/containers/migration/kubernetes_cluster.go:1231 +0x30a
2024-04-29T09:15:46.9172383Z github.com/hashicorp/terraform-provider-azurerm/internal/tf/pluginsdk.StateUpgrades.func1({0x6f4c298, 0xc00005a1c8}, 0x2, {0x5cce140, 0xc0016f8e00})
2024-04-29T09:15:46.9173145Z /home/runner/go/pkg/mod/github.com/pulumi/[email protected]/internal/tf/pluginsdk/state_upgrades.go:52 +0x64
2024-04-29T09:15:46.9173927Z github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.UpgradeJSONState({0x6f4c298, 0xc00005a1c8}, 0x0, 0xc000fbbb60, 0xc000c32dd0, {0x5cce140, 0xc0016f8e00})
2024-04-29T09:15:46.9174679Z /home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:424 +0xa7
2024-04-29T09:15:46.9175731Z github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.upgradeResourceState(0xc000476730, 0xc0005d4cf0, 0xc001cb92b0)
2024-04-29T09:15:46.9176388Z /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/upgrade_state.go:44 +0x205
2024-04-29T09:15:46.9177133Z github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Diff({0xc0016d2240}, {0x66e662a, 0x1a}, {0x6f515f0, 0xc001da1140}, {0x6ef3580, 0xc0015a0960})
2024-04-29T09:15:46.9177810Z /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/provider.go:95 +0xda
2024-04-29T09:15:46.9178467Z github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Diff(0xc0016d2240, {0x6f4c308, 0xc000e80c30}, 0xc0009f01c0)
2024-04-29T09:15:46.9179099Z /home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/provider.go:746 +0x58c
2024-04-29T09:15:46.9179601Z github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler.func1({0x6f4c308, 0xc000e80c30}, {0x62837e0, 0xc0009f01c0})
2024-04-29T09:15:46.9180172Z /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider.pb.go:2611 +0x78
2024-04-29T09:15:46.9180900Z github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x6f4c308, 0xc000c02ae0}, {0x62837e0, 0xc0009f01c0}, 0xc00153dc40, 0xc001c88d80)
2024-04-29T09:15:46.9181625Z /home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x425
2024-04-29T09:15:46.9182208Z github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Diff_Handler({0x63a3000, 0xc0016d2240}, {0x6f4c308, 0xc000c02ae0}, 0xc001ba2de0, 0xc00177b840)
2024-04-29T09:15:46.9182695Z /home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider.pb.go:2613 +0x138
2024-04-29T09:15:46.9183177Z google.golang.org/grpc.(*Server).processUnaryRPC(0xc0002e6700, {0x6fa5778, 0xc0012ef200}, 0xc001ca6900, 0xc00176aea0, 0xc56e8e8, 0x0)
2024-04-29T09:15:46.9183650Z /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1292 +0xc6f
2024-04-29T09:15:46.9184075Z google.golang.org/grpc.(*Server).handleStream(0xc0002e6700, {0x6fa5778, 0xc0012ef200}, 0xc001ca6900, 0x0)
2024-04-29T09:15:46.9184499Z /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1617 +0xa2a
2024-04-29T09:15:46.9184858Z google.golang.org/grpc.(*Server).serveStreams.func1.2()
2024-04-29T09:15:46.9185224Z /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:940 +0x98
2024-04-29T09:15:46.9185609Z created by google.golang.org/grpc.(*Server).serveStreams.func1
2024-04-29T09:15:46.9185983Z /home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:938 +0x294
2024-04-29T09:15:46.9186266Z
2024-04-29T09:15:46.9186369Z
2024-04-29T09:15:46.9945854Z warning: A new version of Pulumi is available. To upgrade from version '3.17.0' to '3.114.0', run
2024-04-29T09:15:46.9949898Z $ curl -sSL https://get.pulumi.com | sh
2024-04-29T09:15:46.9950547Z or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.
2024-04-29T09:15:47.0008454Z
2024-04-29T09:15:47.0080597Z ##[error]Bash exited with code '255'.

What about pulumi stack export it gave this error:
========================== Starting Command Output ===========================
/usr/bin/bash /home/vsts/work/_temp/ae396e25-6b9d-450c-84f3-0fcdc274d9f6.sh
error: read ".pulumi/meta.yaml": blob (key ".pulumi/meta.yaml") (code=Unknown): DefaultAzureCredential authentication failed
GET http://169.254.169.254/metadata/identity/oauth2/token

RESPONSE 400 Bad Request

{
"error": "invalid_request",
"error_description": "Identity not found"
}

@Rus1an31
Copy link
Author

Rus1an31 commented May 3, 2024

Hi @danielrbradley
After changes to upgrade to 4.42.0, it's trying to remove the whole cluster:

2024-04-30T14:55:12.3682164Z pulumi:pulumi:Stack webapik8s-dev running
2024-04-30T14:55:24.8036278Z azure:core:ResourceGroup webapik8s-dev-rg [diff: ~provider]
2024-04-30T14:55:24.9620731Z -- kubernetes:core/v1:ServiceAccount default/nginx-ingress2 delete original
2024-04-30T14:55:24.9654476Z -- kubernetes:policy/v1:PodDisruptionBudget default/nginx-ingress2-controller delete original
2024-04-30T14:55:24.9686638Z -- kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding nginx-ingress2 delete original
2024-04-30T14:55:24.9706723Z -- kubernetes:apps/v1:Deployment default/nginx-ingress2-controller delete original
2024-04-30T14:55:24.9832949Z azure:network:PublicIp webapik8s-devstaticIngressIP [diff: ~provider]
2024-04-30T14:55:24.9885011Z -- kubernetes:networking.k8s.io/v1:IngressClass nginx delete original
2024-04-30T14:55:24.9893305Z -- kubernetes:core/v1:Service default/nginx-ingress2-controller delete original
2024-04-30T14:55:24.9960596Z -- kubernetes:core/v1:Service default/nginx-ingress2-controller-admission delete original
2024-04-30T14:55:24.9989307Z -- kubernetes:core/v1:ConfigMap default/nginx-ingress2-controller delete original
2024-04-30T14:55:25.0001177Z -- kubernetes:core/v1:ServiceAccount default/nginx-ingress2-admission delete original
2024-04-30T14:55:25.0040559Z -- kubernetes:rbac.authorization.k8s.io/v1:ClusterRole nginx-ingress2 delete original
2024-04-30T14:55:25.0171257Z -- kubernetes:batch/v1:Job default/nginx-ingress2-admission-patch delete original
2024-04-30T14:55:25.0257397Z -- kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding nginx-ingress2-admission delete original
2024-04-30T14:55:25.0281117Z -- kubernetes:batch/v1:Job default/nginx-ingress2-admission-create delete original
2024-04-30T14:55:25.0308532Z -- kubernetes:admissionregistration.k8s.io/v1:ValidatingWebhookConfiguration nginx-ingress2-admission delete original
2024-04-30T14:55:25.0325993Z -- kubernetes:rbac.authorization.k8s.io/v1:ClusterRole nginx-ingress2-admission delete original
2024-04-30T14:55:25.0414338Z -- kubernetes:rbac.authorization.k8s.io/v1:RoleBinding default/nginx-ingress2-admission delete original
2024-04-30T14:55:25.0416920Z -- kubernetes:rbac.authorization.k8s.io/v1:Role default/nginx-ingress2-admission delete original
2024-04-30T14:55:25.0436387Z -- kubernetes:rbac.authorization.k8s.io/v1:RoleBinding default/nginx-ingress2 delete original
2024-04-30T14:55:25.0493463Z -- kubernetes:rbac.authorization.k8s.io/v1:Role default/nginx-ingress2 delete original
2024-04-30T14:55:25.0509046Z -- pulumi:providers:kubernetes k8s-provider delete original
2024-04-30T14:55:25.0511836Z -- azure:dns:ARecord nginx-ingress-dns-record delete original
2024-04-30T14:55:25.0524154Z -- azure:containerservice:KubernetesCluster webapik8s-dev delete original
2024-04-30T14:55:25.0525278Z -- azure:containerservice:KubernetesCluster webapik8s-dev delete original error: Preview failed: unable to delete resource "urn:pulumi:dev::webapik8s::azure:containerservice/kubernetesCluster:KubernetesCluster::webapik8s-dev"
2024-04-30T14:55:25.0526146Z pulumi:pulumi:Stack webapik8s-dev running error: preview failed
2024-04-30T14:55:25.0538385Z -- azure:containerservice:KubernetesCluster webapik8s-dev replace 1 error
2024-04-30T14:55:25.0540484Z pulumi:pulumi:Stack webapik8s-dev 1 error
2024-04-30T14:55:25.0540717Z
2024-04-30T14:55:25.0540940Z Diagnostics:
2024-04-30T14:55:25.0541227Z pulumi:pulumi:Stack (webapik8s-dev):
2024-04-30T14:55:25.0541466Z error: preview failed
2024-04-30T14:55:25.0541664Z
2024-04-30T14:55:25.0541991Z azure:containerservice:KubernetesCluster (webapik8s-dev):
2024-04-30T14:55:25.0552912Z error: Preview failed: unable to delete resource "urn:pulumi:dev::webapik8s::azure:containerservice/kubernetesCluster:KubernetesCluster::webapik8s-dev"
2024-04-30T14:55:25.0553485Z as it is currently marked for protection. To unprotect the resource, either remove the protect flag from the resource in your Pulumiprogram and run pulumi up or use the command:
2024-04-30T14:55:25.0554218Z pulumi state unprotect urn:pulumi:dev::webapik8s::azure:containerservice/kubernetesCluster:KubernetesCluster::webapik8s-dev
2024-04-30T14:55:25.0554559Z
2024-04-30T14:55:25.0554640Z

@danielrbradley danielrbradley removed p1 Bugs severe enough to be the next item assigned to an engineer awaiting-feedback labels May 7, 2024
@danielrbradley
Copy link
Member

Thanks for the extra info @Rus1an31

So far we can establish that:

  • Upgrading to v5.x causes a panic during the state migration (most likely from this line of code in the upstream provider)
  • Upgrading to 4.42.0 causes some kind of diff in the provider which has the knock-on effect of causing replacements in the resources being used.

A couple more requests so we can narrow down the issue:

  1. Please can you confirm which version of the provider you are upgrading from
  2. Please could you share the state for the KubernetesCluster resource (danfoss-webapik8s-dev) with any personal information redacted.

Ideally, what we need for an investigation is a complete reproduction of the issue in a standalone pulumi program. I.e.

  1. A minimal program which creates the resource, with explicit pinned version of the provider.
  2. A minimal change to make which triggers the error you're seeing.

My working hypothesis is that an earlier v4 version of the provider it was still using v1 of this resource, but possibly with a slightly different shape for the role_based_access_control property state - which was a single item rather than a list of items. This then is unhandled when upgrading and running the state migration as it's expecting a list.

One workaround for now is to edit the state of this resource using pulumi state export, making the change, then re-importing using pulumi state import in order to make this role_based_access_control property a list.

Without more information it's hard to know for sure, but this is likely a bug in the very old version which stored the state in the wrong format, was later corrected, but you're now running into.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting-feedback impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@thomas11 @danielrbradley @Rus1an31