HDDS-10295. Provide an "ozone repair" subcommand to update the snapshot info in transactionInfoTable #6533

DaveTeng0 · 2024-04-15T22:15:42Z

What changes were proposed in this pull request?

The issue found in HDDS-9342 caused the snapshot info in OM transactionInfoTable not get updated timely, so that OM restart failed at update ID check during raft log reapply.

The recover solution is to find the largest update ID, and update the snapshot info in transactionInfoTable with this it.

The task aims to provide such an CLI to update the table. Be noted, the largest update ID and its term currently should still need manual find.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10295

How was this patch tested?

Integration test

…info in transactionInfoTable

DaveTeng0 · 2024-04-15T22:17:38Z

cc. @ChenSammi @szetszwo @errose28 @adoroszlai

errose28 · 2024-04-16T17:27:49Z

The task aims to provide such an CLI to update the table. Be noted, the largest update ID and its term currently should still need manual find.

Since this is an offline CLI I think it should also support finding the largest updateID (even if it's slow) and doing the update. Maybe as two steps (one to find the largest ID, and another to update to that). Doing the repair incorrectly can result in some bad states and we should try to make the repair commands as safe as possible. @ChenSammi or @fapifta can probably confirm what the correct steps to do the repair are since I haven't actually manually repaired a DB from this bug myself. I think scanning the DB for largest update ID will give the correct number to set the transaction index to.

hemantk-12

Thanks @DaveTeng0 for the patch.

Overall looks good to me, left some cosmetic comments.

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/om/TransactionInfoRepair.java

...ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneRepairShell.java

DaveTeng0 · 2024-04-17T17:14:09Z

The task aims to provide such an CLI to update the table. Be noted, the largest update ID and its term currently should still need manual find.

Since this is an offline CLI I think it should also support finding the largest updateID (even if it's slow) and doing the update. Maybe as two steps (one to find the largest ID, and another to update to that). Doing the repair incorrectly can result in some bad states and we should try to make the repair commands as safe as possible. @ChenSammi or @fapifta can probably confirm what the correct steps to do the repair are since I haven't actually manually repaired a DB from this bug myself. I think scanning the DB for largest update ID will give the correct number to set the transaction index to.

hey @ChenSammi @szetszwo , I look at the previous jira https://issues.apache.org/jira/browse/HDDS-9342, but I'm still not sure what's the best way to retrieve the highest TermIndex, except checking om's log. I see that two maps of 'applyTransactionMap' and 'ratisTransactionMap' have been removed from om, which might contain that information. so do you know where we could retrieve that TermIndex information, other than looking at om's log?
Thanks!

szetszwo · 2024-04-17T20:02:56Z

... two maps of 'applyTransactionMap' and 'ratisTransactionMap' have been removed from om ...

@DaveTeng0 , since this is an offline CLI, there is no OM running and these two maps are not available even if there were not removed.

szetszwo · 2024-04-17T20:07:19Z

... except checking om's log. ...

I guess you mean OM raft log? It also cannot be used since the log entries may or may not be applied.

The correct way is to fine the highest index from RocksDB. This should be what @errose28 has suggested.

DaveTeng0 · 2024-04-17T20:19:58Z

... two maps of 'applyTransactionMap' and 'ratisTransactionMap' have been removed from om ...

@DaveTeng0 , since this is an offline CLI, there is no OM running and these two maps are not available even if there were not removed.

oh! that's right!

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/RocksDBUtils.java

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/om/TransactionInfoRepair.java

DaveTeng0 · 2024-04-21T22:46:17Z

... except checking om's log. ...

I guess you mean OM raft log? It also cannot be used since the log entries may or may not be applied.

The correct way is to fine the highest index from RocksDB. This should be what @errose28 has suggested.

created a jira to investigate how to parse all RocksDB files to get latest highest TermIndex of OM. HDDS-10730

hemantk-12

LGTM.

errose28

Thanks @DaveTeng0 added some comments for improved testing and usability.

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/om/TransactionInfoRepair.java

...ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneRepairShell.java

DaveTeng0 · 2024-05-14T18:23:02Z

Hello! if no further new comments, please feel free to merge! Thanks!

hemantk-12 · 2024-05-14T20:18:31Z

@DaveTeng0 TestTransactionInfoRepair tests are failing due to NPE. Can you please fix that?

hemantk-12

LGTM+1

hemantk-12 · 2024-05-28T22:11:42Z

@errose28 can you please take a look at the final PR?

errose28

Just a few more comments based on the latest iteration.

errose28 · 2024-05-29T21:42:53Z

...ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneRepairShell.java

+ @Test
+ public void testUpdateTransactionInfoTable() throws Exception {
+ CommandLine cmd = new CommandLine(new RDBRepair()).addSubcommand(new TransactionInfoRepair());
+ String dbPath = OMStorage.getOmDbDir(conf) + OM_KEY_PREFIX + OM_DB_NAME;


nit. This is a path on the local filesystem, so it should be constructed from a Path or File object. OM_KEY_PREFIX is for files in the Ozone filesystem.

makes sense! Changed to create a File object and retrieve its path. And changed to use pure "/" in test case instead.

errose28 · 2024-05-29T21:46:33Z

...ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneRepairShell.java

+
+ String cmdOut2 = scanTransactionInfoTable(dbPath);
+ assertThat(cmdOut2).contains(testTerm + "#" + testIndex);
+ cluster.getOzoneManager().restart();


Is the goal to make sure that the OM starts correctly after the repair? If so, we should use the same transaction update command to restore the old values, then do a metadata write operation on the cluster when it comes back up.

makes sense! updated!

...ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneRepairShell.java

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/TransactionInfoRepair.java

errose28 · 2024-05-29T22:09:28Z

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/TransactionInfoRepair.java

+ System.err.println(TRANSACTION_INFO_TABLE + " is not in a column family in DB for the given path.");
+ return null;


I think the command will still exit 0 in this case. If you throw something likeIllegalArgumentException the stack trace will be filtered out, the message printed, and the return code will be non-zero.

This can be tested in TestTransactionInfoRepair too.

definitely makes sense, my mistake and I should have chose to throw exception here instead! thanks for catching it, and will add verification of the error message in test cases.

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/TransactionInfoRepair.java

HDDS-10295. Provide a ozone repair subcommand to update the snapshot …

532014e

…info in transactionInfoTable

dombizita changed the title ~~Provide an ozone repair subcommand to update the snapshot info in transactionInfoTable~~ HDDS-10295. Provide an "ozone repair" subcommand to update the snapshot info in transactionInfoTable Apr 16, 2024

adoroszlai added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Apr 16, 2024

hemantk-12 reviewed Apr 16, 2024

View reviewed changes

move common util function into RocksDBUtils

83d333b

hemantk-12 reviewed Apr 19, 2024

View reviewed changes

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/RocksDBUtils.java Outdated Show resolved Hide resolved

hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/repair/om/TransactionInfoRepair.java Outdated Show resolved Hide resolved

update RocksDBUtils.getValue name, fix code alignment

fcc1e42

hemantk-12 reviewed Apr 23, 2024

View reviewed changes

errose28 reviewed Apr 24, 2024

View reviewed changes

DaveTeng0 added 5 commits April 29, 2024 14:07

add transactionInfoTable unit test

0ef0eaf

remove debug log in test

cbd7375

add license

0fabe4f

fix findbugs of method has no side effect

6d953f0

fix integration test

3c22b19

DaveTeng0 added 3 commits May 15, 2024 11:31

fix NPE in TestTransactionInfoRepair

e249a3c

remove unused import

d204fff

fix TestOzoneRepairShell

2cb514f

hemantk-12 approved these changes May 28, 2024

View reviewed changes

errose28 reviewed May 29, 2024

View reviewed changes

change to throw exception after printing error message

71b15a8

DaveTeng0 added 3 commits June 4, 2024 14:48

fix checkstyle

48f6f56

update command in TestOzoneRepairShell

989e8a8

update command in TestOzoneRepairShell

8f3929f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-10295. Provide an "ozone repair" subcommand to update the snapshot info in transactionInfoTable #6533

HDDS-10295. Provide an "ozone repair" subcommand to update the snapshot info in transactionInfoTable #6533

DaveTeng0 commented Apr 15, 2024

DaveTeng0 commented Apr 15, 2024

errose28 commented Apr 16, 2024

hemantk-12 left a comment

DaveTeng0 commented Apr 17, 2024 •

edited

szetszwo commented Apr 17, 2024

szetszwo commented Apr 17, 2024 •

edited

DaveTeng0 commented Apr 17, 2024

DaveTeng0 commented Apr 21, 2024

hemantk-12 left a comment

errose28 left a comment

DaveTeng0 commented May 14, 2024

hemantk-12 commented May 14, 2024 •

edited

hemantk-12 left a comment

hemantk-12 commented May 28, 2024

errose28 left a comment

errose28 May 29, 2024

DaveTeng0 Jun 4, 2024

errose28 May 29, 2024

DaveTeng0 Jun 4, 2024

errose28 May 29, 2024

errose28 May 29, 2024

DaveTeng0 Jun 4, 2024 •

edited

		System.err.println(TRANSACTION_INFO_TABLE + " is not in a column family in DB for the given path.");
		return null;

HDDS-10295. Provide an "ozone repair" subcommand to update the snapshot info in transactionInfoTable #6533

Are you sure you want to change the base?

HDDS-10295. Provide an "ozone repair" subcommand to update the snapshot info in transactionInfoTable #6533

Conversation

DaveTeng0 commented Apr 15, 2024

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

DaveTeng0 commented Apr 15, 2024

errose28 commented Apr 16, 2024

hemantk-12 left a comment

Choose a reason for hiding this comment

DaveTeng0 commented Apr 17, 2024 • edited

szetszwo commented Apr 17, 2024

szetszwo commented Apr 17, 2024 • edited

DaveTeng0 commented Apr 17, 2024

DaveTeng0 commented Apr 21, 2024

hemantk-12 left a comment

Choose a reason for hiding this comment

errose28 left a comment

Choose a reason for hiding this comment

DaveTeng0 commented May 14, 2024

hemantk-12 commented May 14, 2024 • edited

hemantk-12 left a comment

Choose a reason for hiding this comment

hemantk-12 commented May 28, 2024

errose28 left a comment

Choose a reason for hiding this comment

errose28 May 29, 2024

Choose a reason for hiding this comment

DaveTeng0 Jun 4, 2024

Choose a reason for hiding this comment

errose28 May 29, 2024

Choose a reason for hiding this comment

DaveTeng0 Jun 4, 2024

Choose a reason for hiding this comment

errose28 May 29, 2024

Choose a reason for hiding this comment

errose28 May 29, 2024

Choose a reason for hiding this comment

DaveTeng0 Jun 4, 2024 • edited

Choose a reason for hiding this comment

DaveTeng0 commented Apr 17, 2024 •

edited

szetszwo commented Apr 17, 2024 •

edited

hemantk-12 commented May 14, 2024 •

edited

DaveTeng0 Jun 4, 2024 •

edited