NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable #7544

ravinarayansingh · 2023-07-31T03:28:36Z

Summary

NiFi 11858

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Apache NiFi Jira issue created

Pull Request Tracking

Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

Pull Request based on current revision of the main branch
Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

Build completed using mvn clean install -P contrib-check
- JDK 17

Licensing

New dependencies are compatible with the Apache License 2.0 according to the License Policy
New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

Documentation formatting appears as expected in rendered files

…abaseTable processor

…tegy

exceptionfactory

@ravinarayansingh Thanks for the contribution. There are still some static code analysis issues. You can test locally by running the following command:

./mvnw clean install -Pcontrib-check -am -pl :nifi-standard-processors

ravinarayansingh · 2023-08-01T17:03:16Z

Hi @exceptionfactory ,

following is the rat report:

Summary

Generated at: 2023-08-01T08:57:42-07:00

Notes: 24
Binaries: 5
Archives: 3
Standards: 27

Apache Licensed: 13
Generated Documents: 0

JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.

14 Unknown Licenses

Files with unapproved licenses:

nifi-h2/nifi-h2-database/nifi-h2-database.iml
nifi-h2/nifi-h2-database/target/.plxarc
nifi-h2/nifi-h2-database/target/maven-archiver/pom.properties
nifi-h2/nifi-h2-database-migrator/nifi-h2-database-migrator.iml
nifi-h2/nifi-h2-database-migrator/target/.plxarc
nifi-h2/nifi-h2-database-migrator/target/maven-archiver/pom.properties
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/inputFiles.lst
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/groovy-tests/createdFiles.lst
nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/groovy-tests/inputFiles.lst
nifi-h2/nifi-h2.iml
nifi-h2/target/.plxarc

Archives:

nifi-h2/nifi-h2-database/target/nifi-h2-database-1.18.0-SNAPSHOT.jar
nifi-h2/nifi-h2-database/target/original-nifi-h2-database-1.18.0-SNAPSHOT.jar
nifi-h2/nifi-h2-database-migrator/target/nifi-h2-database-migrator-1.18.0-SNAPSHOT.jar

Files with Apache License headers will be marked AL
Binary files (which do not require any license headers) will be marked B
Compressed archives will be marked A
Notices, licenses etc. will be marked N
AL .asf.yaml
AL .github/PULL_REQUEST_TEMPLATE.md
AL .github/workflows/ci-workflow.yml
AL .github/workflows/stale.yml
AL .github/workflows/system-tests.yml
AL .mvn/wrapper/maven-wrapper.properties
AL checkstyle.xml
B jffi10666034361742842335.dll
N KEYS
N LICENSE
AL mvnw
AL mvnw.cmd
AL nifi-dependency-check-maven/suppressions.xml
!????? nifi-h2/nifi-h2-database/nifi-h2-database.iml
!????? nifi-h2/nifi-h2-database/target/.plxarc
N nifi-h2/nifi-h2-database/target/classes/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database/target/classes/META-INF/LICENSE
N nifi-h2/nifi-h2-database/target/classes/META-INF/NOTICE
!????? nifi-h2/nifi-h2-database/target/maven-archiver/pom.properties
N nifi-h2/nifi-h2-database/target/maven-shared-archive-resources/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database/target/maven-shared-archive-resources/META-INF/LICENSE
N nifi-h2/nifi-h2-database/target/maven-shared-archive-resources/META-INF/NOTICE
A nifi-h2/nifi-h2-database/target/nifi-h2-database-1.18.0-SNAPSHOT.jar
A nifi-h2/nifi-h2-database/target/original-nifi-h2-database-1.18.0-SNAPSHOT.jar
N nifi-h2/nifi-h2-database/target/test-classes/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database/target/test-classes/META-INF/LICENSE
N nifi-h2/nifi-h2-database/target/test-classes/META-INF/NOTICE
!????? nifi-h2/nifi-h2-database-migrator/nifi-h2-database-migrator.iml
!????? nifi-h2/nifi-h2-database-migrator/target/.plxarc
N nifi-h2/nifi-h2-database-migrator/target/classes/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database-migrator/target/classes/META-INF/LICENSE
N nifi-h2/nifi-h2-database-migrator/target/classes/META-INF/NOTICE
B nifi-h2/nifi-h2-database-migrator/target/classes/org/apache/nifi/h2/database/migration/H2DatabaseMigrator.class
B nifi-h2/nifi-h2-database-migrator/target/classes/org/apache/nifi/h2/database/migration/H2DatabaseUpdater.class
!????? nifi-h2/nifi-h2-database-migrator/target/maven-archiver/pom.properties
N nifi-h2/nifi-h2-database-migrator/target/maven-shared-archive-resources/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database-migrator/target/maven-shared-archive-resources/META-INF/LICENSE
N nifi-h2/nifi-h2-database-migrator/target/maven-shared-archive-resources/META-INF/NOTICE
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/compile/default-compile/createdFiles.lst
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/compile/default-compile/inputFiles.lst
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/createdFiles.lst
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/default-testCompile/inputFiles.lst
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/groovy-tests/createdFiles.lst
!????? nifi-h2/nifi-h2-database-migrator/target/maven-status/maven-compiler-plugin/testCompile/groovy-tests/inputFiles.lst
A nifi-h2/nifi-h2-database-migrator/target/nifi-h2-database-migrator-1.18.0-SNAPSHOT.jar
N nifi-h2/nifi-h2-database-migrator/target/test-classes/META-INF/DEPENDENCIES
N nifi-h2/nifi-h2-database-migrator/target/test-classes/META-INF/LICENSE
N nifi-h2/nifi-h2-database-migrator/target/test-classes/META-INF/NOTICE
B nifi-h2/nifi-h2-database-migrator/target/test-classes/nifi-flow-audit.mv.db
B nifi-h2/nifi-h2-database-migrator/target/test-classes/org/apache/nifi/h2/database/migration/TestH2DatabaseUpdater.class
!????? nifi-h2/nifi-h2.iml
!????? nifi-h2/target/.plxarc
N nifi-h2/target/maven-shared-archive-resources/META-INF/DEPENDENCIES
N nifi-h2/target/maven-shared-archive-resources/META-INF/LICENSE
N nifi-h2/target/maven-shared-archive-resources/META-INF/NOTICE
N NOTICE
AL pom.xml
AL README.md
AL SECURITY.md

let me know if i have to change anything
Thanks

exceptionfactory · 2023-08-01T17:09:28Z

@ravinarayansingh The Static Analysis build lists the particular problem:

https://github.com/apache/nifi/actions/runs/5719909353/job/15509139654?pr=7544#step:5:5763

It looks like those nifi-h2 directories are leftover on your local system from the support branch, so they could be removed, but that will not impact the pull request build.

ravinarayansingh · 2023-08-01T19:45:12Z

Hi @exceptionfactory
I have made the require changes
Thanks

MikeThomsen · 2023-08-12T11:18:53Z

@ravinarayansingh you have a merge conflict.

…NIFI-11858 � Conflicts: � nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/UpdateDatabaseTable.java � nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TableSchema.java

ravinarayansingh · 2023-08-12T15:55:24Z

@MikeThomsen

resolved

ravinarayansingh · 2023-09-13T23:44:46Z

Hi @exceptionfactory ,
when this feature will be available in NiFi
Thanks

exceptionfactory · 2023-09-16T15:07:11Z

Hi @exceptionfactory , when this feature will be available in NiFi Thanks

@ravinarayansingh These changes still need substantive review, along with a number of other pull requests, but thanks for addressing the initial feedback so far.

exceptionfactory

Thanks for your patience on this pull request review @ravinarayansingh. Reviewing the implementation details, the general concept of a configurable column normalization strategy is helpful.

As a general recommendation, it looks like it would be helpful to make ColumnNameNormalizer an interface, with getNormalizedName(String columnName) as the interface method. Although the current implementation is simple enough, providing separate implementations would remove the need for the conditional logic. Given that this method will be called for every column, there seems to be value in having narrowly defined implementation classes.

ravinarayansingh · 2023-11-12T03:37:53Z

Thanks for your patience on this pull request review @ravinarayansingh. Reviewing the implementation details, the general concept of a configurable column normalization strategy is helpful.

As a general recommendation, it looks like it would be helpful to make ColumnNameNormalizer an interface, with getNormalizedName(String columnName) as the interface method. Although the current implementation is simple enough, providing separate implementations would remove the need for the conditional logic. Given that this method will be called for every column, there seems to be value in having narrowly defined implementation classes.

Hi @exceptionfactory
thanks for suggestion i will make required changes

…NIFI-11858 � Conflicts: � nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java

ravinarayansingh · 2023-11-24T07:56:00Z

Hi @exceptionfactory ,

I have made the required changes, please have a look
Thanks

exceptionfactory

Thanks for the updates @ravinarayansingh. The general approach looks good, but there are a few more naming recommendations. In particular, any regular expression pattern should be parsed to a Pattern object for more efficient processing in the normalizer implementation.

...standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java

...rd-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizer.java

...essors/src/main/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizerUtility.java

...ard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java

…/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java Co-authored-by: David Handermann <[email protected]>

…NIFI-11858

ravinarayansingh · 2024-03-19T14:06:52Z

Hi @dan-s1 and @exceptionfactory
please have look and let me know if any change required

exceptionfactory

@ravinarayansingh Thanks for your work and patience on this pull request.

Unfortunately I may not have been clear in previous comments regarding the interface-based approached for the ColumnNameNormalizer. I did not intend for the ColumnNameNormalizer to be implemented by each Processor, but instead, the ColumnNameNormalizer should have separate implementations for each strategy.

If it would be helpful, I could follow up with a commit that implements the approach I described.

…NIFI-11858 # Conflicts: # nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java

ravinarayansingh · 2024-04-05T09:00:46Z

@ravinarayansingh Thanks for your work and patience on this pull request.

Unfortunately I may not have been clear in previous comments regarding the interface-based approached for the ColumnNameNormalizer. I did not intend for the ColumnNameNormalizer to be implemented by each Processor, but instead, the ColumnNameNormalizer should have separate implementations for each strategy.

If it would be helpful, I could follow up with a commit that implements the approach I described.

Hi @exceptionfactory
I have made the required changes please have a look

exceptionfactory · 2024-04-06T21:55:31Z

Thanks for the updates @ravinarayansingh, the ColumnNameNormalizer with individual implementations matches what I had in mind. I appreciate your efforts, I will take a closer look soon.

mattyb149 · 2024-04-17T16:24:02Z

This needs a rebase against the latest main, then I will take a look also, thanks!

…NIFI-11858

ravinarayansingh · 2024-04-25T06:50:27Z

Hi @mattyb149
Build is failing but i think that is not related to this PR
please let me know if i have to change anything form my side

…NIFI-11858 # Conflicts: # nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizer.java # nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/ColumnNameNormalizerFactory.java # nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java # nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/PatternNormalizer.java # nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/RemoveAllSpecialCharNormalizer.java # nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/RemoveSpaceNormalizer.java # nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/db/impl/RemoveUnderscoreNormalizer.java

jrsteinebrey

I just noticed this PR today. I would like at least 24 hours more to review this and see if I have feedback on the code. Right now I am giving some wording feedback.

...ard-processors/src/main/java/org/apache/nifi/processors/standard/db/TranslationStrategy.java

…NIFI-11858 # Conflicts: # nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/PutDatabaseRecordTest.java

ravinarayansingh · 2024-05-22T16:06:09Z

Hi @jrsteinebrey
I have updated the code as per your suggestion, have a look

jrsteinebrey · 2024-05-23T18:29:01Z

Thanks for the changes you made. Some other files in this PR now have checkstyle violations:

Warning: src/main/java/org/apache/nifi/processors/standard/db/TableSchema.java:[155,47] (whitespace) WhitespaceAround: '!=' is not followed by whitespace.
Warning: src/main/java/org/apache/nifi/processors/standard/db/TableSchema.java:[155,47] (whitespace) WhitespaceAround: '!=' is not preceded with whitespace.
Warning: src/test/java/org/apache/nifi/processors/standard/db/impl/TestOracleDatabaseAdapter.java:[124,122] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
Warning: src/test/java/org/apache/nifi/processors/standard/db/impl/TestOracle12DatabaseAdapter.java:[166,122] (whitespace) WhitespaceAfter: ',' is not followed by whitespace.
Warning: src/test/java/org/apache/nifi/processors/standard/PutDatabaseRecordTest.java:[1832,11] (whitespace) WhitespaceAfter: 'catch' is not followed by whitespace.
Warning: src/test/java/org/apache/nifi/processors/standard/PutDatabaseRecordTest.java:[1832,11] (whitespace) WhitespaceAround: 'catch' is not followed by whitespace.

mattyb149 · 2024-05-24T19:32:06Z

The checkstyle rules were recently made more stringent, looks like this needs another rebase and please run your Maven build with the contrib-check profile activated.

ravinarayansingh added 6 commits July 29, 2023 19:54

Improved column name normalization in PutDatabaseRecord processor

ae11896

Improved column name normalization in PutDatabaseRecord processor

c3de9ee

Improved column name normalization in PutDatabaseRecord and UpdateDat…

09bc153

…abaseTable processor

updated import statement with single import

d1f0a37

fixed import statement and line length

d01b51f

added the license.txt to the ColumnNameNormalizer and TranslationStra…

20cc866

…tegy

exceptionfactory requested changes Aug 1, 2023

View reviewed changes

mattyb149 changed the title ~~NiFi 11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor~~ NIFI-11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor Aug 1, 2023

added the license text to ColumnNameNormalizerTest.java

41ebb89

ravinarayansingh added 2 commits August 12, 2023 08:38

conflict resolved

1a90ed5

fixed code style

a1dc504

ravinarayansingh requested a review from exceptionfactory August 16, 2023 14:11

exceptionfactory requested changes Nov 11, 2023

View reviewed changes

exceptionfactory changed the title ~~NIFI-11858 Improve column name normalization in PutDatabaseRecord and UpdateDatabaseTable processor~~ NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable Nov 16, 2023

ravinarayansingh added 2 commits November 23, 2023 22:09

changed ColumnNameNormalizer from class to interface

5e97150

Merge branch 'main' of https://github.com/ravinarayansingh/nifi into …

0c13d21

…NIFI-11858 � Conflicts: � nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java

added method argument details

095d9bf

exceptionfactory requested changes Dec 2, 2023

View reviewed changes

ravinarayansingh and others added 2 commits December 3, 2023 20:53

Update nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors…

5a61651

…/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java Co-authored-by: David Handermann <[email protected]>

Update nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors…

5547f94

…/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java Co-authored-by: David Handermann <[email protected]>

ravinarayansingh added 3 commits February 4, 2024 04:16

Merge branch 'main' of https://github.com/ravinarayansingh/nifi into …

69df3af

…NIFI-11858

fixed TestOracleDatabaseAdapter and TestOracle12DatabaseAdapter unittest

fc9a206

remove trailing space

4eb9ccd

ravinarayansingh requested review from exceptionfactory and dan-s1 February 29, 2024 20:03

exceptionfactory requested changes Mar 26, 2024

View reviewed changes

ravinarayansingh added 3 commits April 5, 2024 00:56

added separate implementation for each translation strategy

55cac5c

Merge branch 'main' of https://github.com/ravinarayansingh/nifi into …

7048c36

…NIFI-11858 # Conflicts: # nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutDatabaseRecord.java

added licence text

f58c12a

fixed style

3afa54d

mattyb149 self-requested a review April 17, 2024 16:24

ravinarayansingh added 2 commits April 18, 2024 02:16

rebase

bee7351

Merge branch 'main' of https://github.com/ravinarayansingh/nifi into …

7ac8d25

…NIFI-11858

ravinarayansingh added 2 commits April 26, 2024 12:09

added log api dependency

f536355

ravinarayansingh requested a review from exceptionfactory April 30, 2024 23:52

jrsteinebrey suggested changes May 16, 2024

View reviewed changes

ravinarayansingh added 2 commits May 22, 2024 08:50

Merge branch 'main' of https://github.com/ravinarayansingh/nifi into …

7433b84

…NIFI-11858 # Conflicts: # nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/PutDatabaseRecordTest.java

updated descriptions

017d41d

ravinarayansingh added 2 commits May 24, 2024 14:07

fixed styles

48cd6c4

fixed styles

0d4d0e6

ravinarayansingh requested a review from jrsteinebrey June 2, 2024 02:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable #7544

NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable #7544

ravinarayansingh commented Jul 31, 2023

exceptionfactory left a comment

ravinarayansingh commented Aug 1, 2023

exceptionfactory commented Aug 1, 2023

ravinarayansingh commented Aug 1, 2023

MikeThomsen commented Aug 12, 2023

ravinarayansingh commented Aug 12, 2023 •

edited

ravinarayansingh commented Sep 13, 2023

exceptionfactory commented Sep 16, 2023

exceptionfactory left a comment

ravinarayansingh commented Nov 12, 2023

ravinarayansingh commented Nov 24, 2023

exceptionfactory left a comment

ravinarayansingh commented Mar 19, 2024

exceptionfactory left a comment

ravinarayansingh commented Apr 5, 2024

exceptionfactory commented Apr 6, 2024

mattyb149 commented Apr 17, 2024

ravinarayansingh commented Apr 25, 2024 •

edited

jrsteinebrey left a comment

ravinarayansingh commented May 22, 2024 •

edited

jrsteinebrey commented May 23, 2024

mattyb149 commented May 24, 2024

NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable #7544

Are you sure you want to change the base?

NIFI-11858 Configurable Column Name Normalization in PutDatabaseRecord and UpdateDatabaseTable #7544

Conversation

ravinarayansingh commented Jul 31, 2023

Summary

Tracking

Issue Tracking

Pull Request Tracking

Pull Request Formatting

Verification

Build

Licensing

Documentation

exceptionfactory left a comment

Choose a reason for hiding this comment

ravinarayansingh commented Aug 1, 2023

Summary

exceptionfactory commented Aug 1, 2023

ravinarayansingh commented Aug 1, 2023

MikeThomsen commented Aug 12, 2023

ravinarayansingh commented Aug 12, 2023 • edited

ravinarayansingh commented Sep 13, 2023

exceptionfactory commented Sep 16, 2023

exceptionfactory left a comment

Choose a reason for hiding this comment

ravinarayansingh commented Nov 12, 2023

ravinarayansingh commented Nov 24, 2023

exceptionfactory left a comment

Choose a reason for hiding this comment

ravinarayansingh commented Mar 19, 2024

exceptionfactory left a comment

Choose a reason for hiding this comment

ravinarayansingh commented Apr 5, 2024

exceptionfactory commented Apr 6, 2024

mattyb149 commented Apr 17, 2024

ravinarayansingh commented Apr 25, 2024 • edited

jrsteinebrey left a comment

Choose a reason for hiding this comment

ravinarayansingh commented May 22, 2024 • edited

jrsteinebrey commented May 23, 2024

mattyb149 commented May 24, 2024

ravinarayansingh commented Aug 12, 2023 •

edited

ravinarayansingh commented Apr 25, 2024 •

edited

ravinarayansingh commented May 22, 2024 •

edited