NIFI-12674 Modified ValidateCSV to make the schema optional if a head… #8362
Conversation
|
Any reason for not using ValidateRecord processor if the only requirement is to confirm that the data is valid CSV without any specific constraint? |
pvillard31
changed the title
|
@pvillard31 The ValidateRecord processor splits the input FlowFile into 2 Flowfiles, one for valid and another for invalid records. With the change to ValidateCSV, the whole file will be routed to either valid or invalid. |
.../nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateCsv.java
Outdated
Show resolved
Hide resolved
.../nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateCsv.java
Outdated
Show resolved
Hide resolved
.../nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ValidateCsv.java
Outdated
Show resolved
Hide resolved
|
@exceptionfactory Can you please restart the failed job? It does not seem related to the changes. Thanks! |
|
Any updates on reviewing this change? |
|
There are merge conflicts that need to be resolved |
|
@mattyb149 I've rebased against main. Thank You |
…er is provided. Added validate on attribute option.
| if (schema != null) { | ||
| this.parseSchema(schema); | ||
| } else if (!headerProp.asBoolean()) { | ||
| throw(new Exception("Schema cannot be empty if header is false.")); |
There was a problem hiding this comment.
Rather than throwing an exception, a ValidationResult should be added to a list to be returned at the end of the method.
…er is provided. Added validate on attribute option.
Summary
NIFI-12674
Made the schema optional for the ValidateCSV processor if a header is provided. In this case, only the structure of the CSV will validated, using the header to determine how many fields each line should have.
Additionally, a new validation strategy was implemented, Validate on Attribute. This works similar to the way ValidateXML works in which the value of a given attribute of a FlowFile will be treated as the contents of a CSV file. Validation will be done on that attribute and not on the content of the FlowFile.
I would also like to note that I made the stream a variable that can be assigned to either the FlowFile content or the value of the attribute. In doing this, I removed the need for an inner method, and thus all of the variables that previously needed to be Atomic References could now be regular variables.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking
Pull Request Tracking
NIFI-00000NIFI-00000Pull Request Formatting
mainbranchVerification
Please indicate the verification steps performed prior to pull request creation.
Build
mvn clean install -P contrib-checkLicensing
LICENSEandNOTICEfilesDocumentation