ccl/backupccl: TestTenantBackupNemesis failed

[cockroach-teamcity]

ccl/backupccl.TestTenantBackupNemesis failed on master @ b185402d34768c898d63a61de3e585e4723fbe81:

Fatal error:

panic: test timed out after 14m57s
running tests:
	TestTenantBackupNemesis (13m44s)

Stack:

goroutine 636920 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

    tenant_backup_nemesis_test.go:548: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1715683988250921519.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:552: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:548: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1715683991074677106.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:595: backup-nemesis: CANCELED IMPORT INTO finished
    tenant_backup_nemesis_test.go:590: backup-nemesis: IMPORT INTO started
    tenant_backup_nemesis_test.go:595: backup-nemesis: IMPORT INTO finished
    tenant_backup_nemesis_test.go:590: backup-nemesis: CREATE TENANT started
    tenant_backup_nemesis_test.go:595: backup-nemesis: CREATE TENANT finished
    tenant_backup_nemesis_test.go:590: backup-nemesis: CREATE TENANT started
    tenant_backup_nemesis_test.go:595: backup-nemesis: CREATE TENANT finished
    tenant_backup_nemesis_test.go:590: backup-nemesis: DELETE TENANT started
    tenant_backup_nemesis_test.go:595: backup-nemesis: DELETE TENANT finished
    tenant_backup_nemesis_test.go:590: backup-nemesis: IMPORT INTO started
    tenant_backup_nemesis_test.go:595: backup-nemesis: IMPORT INTO finished
    tenant_backup_nemesis_test.go:590: backup-nemesis: REPLICATE TENANT started
    tenant_backup_nemesis_test.go:398: already hit replication limit
    tenant_backup_nemesis_test.go:595: backup-nemesis: REPLICATE TENANT finished
    tenant_backup_nemesis_test.go:590: backup-nemesis: CREATE TENANT started
    tenant_backup_nemesis_test.go:595: backup-nemesis: CREATE TENANT finished
    tenant_backup_nemesis_test.go:590: backup-nemesis: IMPORT INTO started
    tenant_backup_nemesis_test.go:595: backup-nemesis: IMPORT INTO finished
    tenant_backup_nemesis_test.go:590: backup-nemesis: RENAME TENANT started
    tenant_backup_nemesis_test.go:595: backup-nemesis: RENAME TENANT finished
    tenant_backup_nemesis_test.go:590: backup-nemesis: CANCELED IMPORT INTO started
    tenant_backup_nemesis_test.go:434: waiting for paused import job
    tenant_backup_nemesis_test.go:552: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:548: backup-nemesis: incremental backup started: BACKUP INTO LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1715683993965404177.0000000000 WITH include_all_secondary_tenants
    tenant_backup_nemesis_test.go:552: backup-nemesis: incremental backup finished
    tenant_backup_nemesis_test.go:595: backup-nemesis: CANCELED IMPORT INTO finished
    tenant_backup_nemesis_test.go:255: backup-nemesis: restoring tenant 10 into restored-tenant-10: RESTORE TENANT 10 FROM LATEST IN 'nodelocal://1/cluster-backup' AS OF SYSTEM TIME 1715683993965404177.0000000000 WITH virtual_cluster = '11', virtual_cluster_name = 'restored-tenant-10'
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank.bank at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."create_index_942894b9-7f12-4fd2-8a59-32faf7817076" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."create_unique_index_681dd1a2-93f9-457c-b3dd-342cc4d234ff" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_658f8f8e-aff5-4ac8-9dc5-d4b1667bdb66" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_f1b0d6b4-aed4-4692-a2a4-239d1b061c8f" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_609d5c35-90d5-4e4b-a161-4abb05641bce" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."create_unique_index_97aee0d7-fd59-4498-a88c-a35225e2f848" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."create_unique_index_a80ccc37-fd57-4f58-953b-9ea4fd73c3e3" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."create_unique_index_e2869380-37c4-4292-bd6a-33748a7469cb" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_cancel_30fe2906-b048-42f7-9bfc-73c0df1ac4df" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_cancel_b5a4a4b9-7fc4-4eca-b588-94fdd000f992" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_cancel_7102f675-809a-4487-a3f5-1753d6e3c645" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_e79f5412-c7d8-4c0f-a12c-35c49e935b22" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."create_unique_index_e324177d-a6fa-418c-9471-f65ef97dbc0f" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."create_unique_index_b67d29a4-d27c-4e5d-b624-ceb53af8336b" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_56332d4b-1d14-4c79-b382-5abacc6e7d47" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_cancel_06426e32-77c3-4f46-afcb-e5f91b688e1b" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_706940e5-9a1c-4b99-9243-8dbd03f98bca" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_8eb1215d-b3a6-4794-8d10-253fb965bb54" at 1715683993965404177.0000000000
    tenant_backup_nemesis_test.go:286: backup-nemesis: checking table bank."import_into_85f05b7d-9991-465f-8522-c97e846e4212" at 1715683993965404177.0000000000

Parameters:

• attempt=1
• run=9
• shard=12

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-38717

[msbutler]

Closing this out. I believe this was fixed by #124162