Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

H2 DB corruption prevents server startup #11276

Open
damvinod opened this issue Feb 2, 2023 · 2 comments
Open

H2 DB corruption prevents server startup #11276

damvinod opened this issue Feb 2, 2023 · 2 comments
Assignees
@chadlwilson
Labels
bug no stalebot Don't mark this stale.

Comments

@damvinod
Copy link

damvinod commented Feb 2, 2023

Issue Type
  • Bug Report
Summary

When we restarted the GoCD server as part of regular patching of our underlying EKS worker nodes, there were some problems migrating the h2 database, hence preventing GoCD server from starting up properly. In order to mitigate this db migration error, we had to manually export and import the db data by using the steps mentioned in the below link

https://gist.github.com/arvindsv/c7934bee26ce1c47e9d24f0a493d3c87

Environment
Basic environment details
  • Go Version: GoCD Version: 22.3.0-15301
  • JAVA Version: 17.0.5
  • OS: Linux(5.4.228-131.415.amzn2.x86_64)
  • Browser vendor and version (if relevant): Chrome/Firefox
Expected Results

GoCD server should restart successfully

Actual Results

ERROR [WrapperJarAppMain] DatabaseMigrator:65 - Unable to create database upgrade script for database. The problem was: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:

Possible Fix
Log snippets
"Running go-server..."
wrapper  | Failed to set JVM input handle to non blocking mode: Bad file descriptor (9)
wrapper  | Failed to set JVM input handle to close on JVM exit: Bad file descriptor (9)
"wrapper  | JVM exited while loading the application."
"wrapper  | Unable to set JVM's stdin: Bad file descriptor"
"wrapper  | --> Wrapper Started as Console"
�]0;go-server�wrapper  | Java Service Wrapper Standard Edition 64-bit 3.5.50
wrapper  |   Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
"wrapper  |     http://wrapper.tanukisoftware.com"
"wrapper  |   Licensed to ThoughtWorks for GoCD Server"
"wrapper  | "
"wrapper  | Launching a JVM..."
jvm 1    | wrapper  | --> Wrapper Started as Console
jvm 1    | �]0;go-server�wrapper  | Java Service Wrapper Standard Edition 64-bit 3.5.50
jvm 1    | wrapper  |   Copyright (C) 1999-2022 Tanuki Software, Ltd. All Rights Reserved.
jvm 1    | wrapper  |     http://wrapper.tanukisoftware.com
jvm 1    | wrapper  |   Licensed to ThoughtWorks for GoCD Server
jvm 1    | wrapper  |
jvm 1    | wrapper  | Launching a JVM...
"wrapper  | Launching a JVM..."
jvm 2    | -XX:CompressedClassSpaceSize=335544320 -XX:ConcGCThreads=2 -XX:G1ConcRefinementThreads=8 -XX:GCDrainStackTargetSize=64 -XX:InitialHeapSize=2147483648 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=4294967296 -XX:MaxMetaspaceSize=419430400 -XX:MinHeapSize=2147483648 -XX:+PrintCommandLineFlags -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC
jvm 2    | WrapperManager: Initializing...
jvm 2    | [Fri Jan 27 14:42:36 GMT 2023] Starting process:
jvm 2    | [Fri Jan 27 14:42:36 GMT 2023]   Working directory    : /go-working-dir
jvm 2    | [Fri Jan 27 14:42:36 GMT 2023]   Application arguments: []
jvm 2    | [Fri Jan 27 14:42:36 GMT 2023]            GoCD Version: 22.3.0-15301
jvm 2    | [Fri Jan 27 14:42:36 GMT 2023]            Java Version: 17.0.5
jvm 2    | [Fri Jan 27 14:42:36 GMT 2023]        Operating System: Linux(5.4.228-131.415.amzn2.x86_64)
jvm 2    | Could not find file `config/logback.xml'. Attempting to load from classpath.
jvm 2    | Using classpath resource `jar:onejar:lib/server-launcher-22.3.0-15301-main.jar!/config/logback.xml'.
jvm 2    | 2023-01-27 14:42:37,413 INFO  [WrapperJarAppMain] Jetty9Server:193 - Configuring Jetty using /go-working-dir/config/jetty.xml
jvm 2    | 2023-01-27 14:42:37,469 WARN  [WrapperJarAppMain] Server:357 - ErrorPageMapper not supported for Server level Error Handling
jvm 2    | 2023-01-27 14:42:37,539 WARN  [WrapperJarAppMain] AbstractHandler:96 - No Server set for ResourceHandler@54d62683{STOPPED}
jvm 2    | 2023-01-27 14:42:43,244 WARN  [WrapperJarAppMain] ConnectionManager:117 - The file config/db.properties specified by `go.db.config` does not exist.
jvm 2    | 2023-01-27 14:42:45,050 INFO  [WrapperJarAppMain] DatabaseMigrator:40 - Upgrading database, this might take a while depending on the size of the database.
jvm 2    | ************************************************************************
jvm 2    | 2023-01-27 14:42:45,051 INFO  [WrapperJarAppMain] DatabaseMigrator:49 - ************************************************************************
jvm 2    | WARNING: Shutting down your server at this point will lead to a database corruption. Please wait until the database upgrade completes.
jvm 2    | 2023-01-27 14:42:45,051 INFO  [WrapperJarAppMain] DatabaseMigrator:49 - WARNING: Shutting down your server at this point will lead to a database corruption. Please wait until the database upgrade completes.
jvm 2    | ************************************************************************
jvm 2    | 2023-01-27 14:42:45,051 INFO  [WrapperJarAppMain] DatabaseMigrator:49 - ************************************************************************
jvm 2    | Jan 27, 2023 2:42:45 PM liquibase.database
jvm 2    | INFO: Set default schema name to PUBLIC
jvm 2    | Jan 27, 2023 2:42:45 PM liquibase.lockservice
jvm 2    | INFO: Waiting for changelog lock....
jvm 2    | 2023-01-27 14:42:55,176 ERROR [WrapperJarAppMain] DatabaseMigrator:65 - Unable to create database upgrade script for database. The problem was: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2    | ROLLBACK [50000-200]. The cause was: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2    | ROLLBACK [50000-200]
jvm 2    | liquibase.exception.LockException: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2    | ROLLBACK [50000-200]
jvm 2    | 	at liquibase.lockservice.StandardLockService.acquireLock(StandardLockService.java:331)
jvm 2    | 	at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:249)
jvm 2    | 	at liquibase.Liquibase.lambda$update$1(Liquibase.java:239)
jvm 2    | 	at liquibase.Scope.lambda$child$0(Scope.java:180)
jvm 2    | 	at liquibase.Scope.child(Scope.java:189)
jvm 2    | 	at liquibase.Scope.child(Scope.java:179)
jvm 2    | 	at liquibase.Scope.child(Scope.java:158)
jvm 2    | 	at liquibase.Liquibase.runInScope(Liquibase.java:2447)
jvm 2    | 	at liquibase.Liquibase.update(Liquibase.java:236)
jvm 2    | 	at liquibase.Liquibase.update(Liquibase.java:221)
jvm 2    | 	at liquibase.Liquibase.update(Liquibase.java:207)
jvm 2    | 	at com.thoughtworks.go.server.database.migration.DatabaseMigrator.migrate(DatabaseMigrator.java:54)
jvm 2    | 	at com.thoughtworks.go.server.database.Database.getDataSource(Database.java:73)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2    | 	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1176)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1071)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
jvm 2    | 	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:211)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1134)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1196)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1098)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getSingletonFactoryBeanForTypeCheck(AbstractAutowireCapableBeanFactory.java:927)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryBean(AbstractAutowireCapableBeanFactory.java:804)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:573)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403)
jvm 2    | 	at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:206)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1263)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1097)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1196)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1098)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:756)
jvm 2    | 	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
jvm 2    | 	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
jvm 2    | 	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
jvm 2    | 	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
jvm 2    | 	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
jvm 2    | 	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
jvm 2    | 	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
jvm 2    | 	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
jvm 2    | 	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
jvm 2    | 	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
jvm 2    | 	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
jvm 2    | 	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
jvm 2    | 	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
jvm 2    | 	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
jvm 2    | 	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
jvm 2    | 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
jvm 2    | 	at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46)
jvm 2    | 	at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
jvm 2    | 	at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:517)
jvm 2    | 	at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:157)
jvm 2    | 	at com.thoughtworks.go.server.Jetty9Server.startHandlers(Jetty9Server.java:173)
jvm 2    | 	at liquibase.Liquibase.runInScope(Liquibase.java:2447)
jvm 2    | 	at liquibase.Liquibase.update(Liquibase.java:236)
jvm 2    | 	at liquibase.Liquibase.update(Liquibase.java:221)
jvm 2    | 	at liquibase.Liquibase.update(Liquibase.java:207)
jvm 2    | 	at com.thoughtworks.go.server.database.migration.DatabaseMigrator.migrate(DatabaseMigrator.java:54)
jvm 2    | 	at com.thoughtworks.go.server.database.Database.getDataSource(Database.java:73)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2    | 	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1176)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1071)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
jvm 2    | 	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:211)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1134)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1196)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1098)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getSingletonFactoryBeanForTypeCheck(AbstractAutowireCapableBeanFactory.java:927)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.getTypeForFactoryBean(AbstractAutowireCapableBeanFactory.java:804)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:573)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doGetBeanNamesForType(DefaultListableBeanFactory.java:432)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:403)
jvm 2    | 	at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:206)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1263)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1097)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1062)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
jvm 2    | 	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:189)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1196)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1098)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:511)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
jvm 2    | 	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
jvm 2    | 	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:756)
jvm 2    | 	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
jvm 2    | 	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
jvm 2    | 	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
jvm 2    | 	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
jvm 2    | 	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
jvm 2    | 	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
jvm 2    | 	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
jvm 2    | 	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
jvm 2    | 	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:765)
jvm 2    | 	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
jvm 2    | 	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
jvm 2    | 	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
jvm 2    | 	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
jvm 2    | 	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
jvm 2    | 	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
jvm 2    | 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
jvm 2    | 	at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46)
jvm 2    | 	at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
jvm 2    | 	at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:517)
jvm 2    | 	at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:157)
jvm 2    | 	at com.thoughtworks.go.server.Jetty9Server.startHandlers(Jetty9Server.java:173)
jvm 2    | 	at com.thoughtworks.go.server.Jetty9Server.start(Jetty9Server.java:129)
jvm 2    | 	at com.thoughtworks.go.server.GoServer.startServer(GoServer.java:62)
jvm 2    | 	at com.thoughtworks.go.server.GoServer.go(GoServer.java:54)
jvm 2    | 	at com.thoughtworks.go.server.util.GoLauncher.main(GoLauncher.java:42)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2    | 	at com.thoughtworks.gocd.Boot.run(Boot.java:90)
jvm 2    | 	at com.thoughtworks.gocd.Boot.main(Boot.java:56)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2    | 	at org.tanukisoftware.wrapper.WrapperJarApp.run(WrapperJarApp.java:451)
jvm 2    | 	at java.base/java.lang.Thread.run(Unknown Source)
jvm 2    | Caused by: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2    | ROLLBACK [50000-200]
jvm 2    | 	at liquibase.database.jvm.JdbcConnection.rollback(JdbcConnection.java:489)
jvm 2    | 	at liquibase.database.AbstractJdbcDatabase.rollback(AbstractJdbcDatabase.java:1172)
jvm 2    | 	at liquibase.lockservice.StandardLockService.acquireLock(StandardLockService.java:287)
jvm 2    | 	... 92 common frames omitted
jvm 2    | Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2    | ROLLBACK [50000-200]
jvm 2    | 	at org.h2.message.DbException.getJdbcSQLException(DbException.java:505)
jvm 2    | 	at org.h2.message.DbException.getJdbcSQLException(DbException.java:429)
jvm 2    | 	at org.h2.message.DbException.get(DbException.java:194)
jvm 2    | 	at org.h2.engine.Database.throwLastBackgroundException(Database.java:2221)
jvm 2    | 	at org.h2.engine.Database.commit(Database.java:2202)
jvm 2    | 	at org.h2.engine.Session.rollback(Session.java:846)
jvm 2    | 	at org.h2.command.dml.TransactionCommand.update(TransactionCommand.java:49)
jvm 2    | 	at org.h2.command.CommandContainer.update(CommandContainer.java:198)
jvm 2    | 	at org.h2.command.Command.executeUpdate(Command.java:251)
jvm 2    | 	at org.h2.jdbc.JdbcConnection.rollbackInternal(JdbcConnection.java:1530)
jvm 2    | 	at org.h2.jdbc.JdbcConnection.rollback(JdbcConnection.java:561)
jvm 2    | 	at org.apache.commons.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:781)
jvm 2    | 	at org.apache.commons.dbcp2.DelegatingConnection.rollback(DelegatingConnection.java:781)
jvm 2    | 	at liquibase.database.jvm.JdbcConnection.rollback(JdbcConnection.java:486)
jvm 2    | 	... 94 common frames omitted
jvm 2    | Caused by: org.h2.message.DbException: General error: "java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0" [50000-200]
jvm 2    | 	at org.h2.message.DbException.get(DbException.java:194)
jvm 2    | 	at org.h2.message.DbException.convert(DbException.java:347)
jvm 2    | 	at org.h2.mvstore.db.MVTableEngine$1.uncaughtException(MVTableEngine.java:93)
jvm 2    | 	at org.h2.mvstore.MVStore.handleException(MVStore.java:2877)
jvm 2    | 	at org.h2.mvstore.MVStore.writeInBackground(MVStore.java:2813)
jvm 2    | 	at org.h2.mvstore.MVStore$BackgroundWriterThread.run(MVStore.java:3290)
jvm 2    | Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0" [50000-200]
jvm 2    | 	at org.h2.message.DbException.getJdbcSQLException(DbException.java:505)
jvm 2    | 	at org.h2.message.DbException.getJdbcSQLException(DbException.java:429)
jvm 2    | 	... 6 common frames omitted
jvm 2    | Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
jvm 2    | 	at org.h2.mvstore.Page.getKey(Page.java:270)
jvm 2    | 	at org.h2.mvstore.MVMap.rewritePage(MVMap.java:773)
jvm 2    | 	at org.h2.mvstore.MVMap.rewrite(MVMap.java:734)
jvm 2    | 	at org.h2.mvstore.MVMap.rewrite(MVMap.java:710)
jvm 2    | 	at org.h2.mvstore.MVStore.compactRewrite(MVStore.java:2137)
jvm 2    | 	at org.h2.mvstore.MVStore.rewriteChunks(MVStore.java:2026)
jvm 2    | 	at org.h2.mvstore.MVStore.doMaintenance(MVStore.java:2844)
jvm 2    | 	at org.h2.mvstore.MVStore.writeInBackground(MVStore.java:2788)
jvm 2    | 	... 1 common frames omitted
jvm 2    | Unable to create database upgrade script for database. The problem was: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2    | ROLLBACK [50000-200]. The cause was: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2    | ROLLBACK [50000-200]
jvm 2    | liquibase.exception.LockException: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2    | ROLLBACK [50000-200]
jvm 2    | 	at liquibase.lockservice.StandardLockService.acquireLock(StandardLockService.java:331)
jvm 2    | 	at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:249)
jvm 2    | 	at liquibase.Liquibase.lambda$update$1(Liquibase.java:239)
jvm 2    | 	at liquibase.Scope.lambda$child$0(Scope.java:180)
jvm 2    | 	at liquibase.Scope.child(Scope.java:189)
jvm 2    | 	at com.thoughtworks.go.server.Jetty9Server.start(Jetty9Server.java:129)
jvm 2    | 	at com.thoughtworks.go.server.GoServer.startServer(GoServer.java:62)
jvm 2    | 	at com.thoughtworks.go.server.GoServer.go(GoServer.java:54)
jvm 2    | 	at com.thoughtworks.go.server.util.GoLauncher.main(GoLauncher.java:42)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2    | 	at com.thoughtworks.gocd.Boot.run(Boot.java:90)
jvm 2    | 	at com.thoughtworks.gocd.Boot.main(Boot.java:56)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 2    | 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 2    | 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
jvm 2    | 	at org.tanukisoftware.wrapper.WrapperJarApp.run(WrapperJarApp.java:451)
jvm 2    | 	at java.base/java.lang.Thread.run(Unknown Source)
jvm 2    | Caused by: liquibase.exception.DatabaseException: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2    | ROLLBACK [50000-200]
jvm 2    | 	at liquibase.database.jvm.JdbcConnection.rollback(JdbcConnection.java:489)
jvm 2    | 	at liquibase.database.AbstractJdbcDatabase.rollback(AbstractJdbcDatabase.java:1172)
jvm 2    | 	at liquibase.lockservice.StandardLockService.acquireLock(StandardLockService.java:287)
jvm 2    | 	... 92 common frames omitted
jvm 2    | Caused by: org.h2.jdbc.JdbcSQLNonTransientException: General error: "General error: ""java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0"" [50000-200]"; SQL statement:
jvm 2    | ROLLBACK [50000-200]
@chadlwilson
Copy link
Member

chadlwilson commented Feb 2, 2023 •
edited

Unfortunately, this points to H2 database corruption (which you have worked around with the export+import). If your H2 database has been corrupted somehow, there's not a lot GoCD can do to recover it automatically.

So the key here is to figure out what caused the corruption in the first place and whether there's anything GoCD can do about it. I would check

  • was the server otherwise working OK and had been restarted (on the earlier version) with no similar h2 stack traces or start-up errors earlier?
  • were there similar h2 stack traces in the logs for, say, a week or two prior?
  • if not, during the upgrade, did you have problems or did anything happen such as the pod being manually deleted while start, or Kubernetes or the Tanuki wrapper auto-restarting the JVM for some reason?

@chadlwilson chadlwilson added the bug label Feb 2, 2023
@chadlwilson chadlwilson changed the title H2 DB corrupt H2 DB corruption prevents server startup Feb 3, 2023
@stale
Copy link

stale bot commented May 21, 2023

This issue has been automatically marked as stale because it has not had activity in the last 90 days.
If you can still reproduce this error on the master branch using local development environment or on the latest GoCD Release, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

@stale stale bot added the stale label May 21, 2023
@chadlwilson chadlwilson self-assigned this May 24, 2023
@stale stale bot closed this as completed Aug 12, 2023
@chadlwilson chadlwilson reopened this Aug 13, 2023
@chadlwilson chadlwilson added no stalebot Don't mark this stale. and removed stale labels Jan 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chadlwilson chadlwilson

Labels
bug no stalebot Don't mark this stale.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chadlwilson @damvinod