Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NIFI-7085 add flowFile batching to ConsumeJMS and PublishJMS #8584

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

mosermw
Copy link
Member

@mosermw mosermw commented Mar 29, 2024

Summary

NIFI-7085

This PR adds batching to PublishJMS and ConsumeJMS in the form of a new property. ConsumeJMS can read more than one message from the JMS broker per triggered task. PublishJMS can publish more than one flowFile to the JMS broker per triggered task. This helps get higher message throughput to JMS in busy NiFi environments.

The SupportsBatching annotation was considered, but it is not appropriate for processors that send/receive data from an external source.

The new Maximum Batch Size property has a default value of 1, to match old behavior. ConsumeJMS in record processing mode used to have a 10,000 max batch size which is replaced by this property value. ConsumeJMS will migrate the new Maximum Batch Size property to 10000 if it was not set and it was in record processing mode.

Fixed an bug in JMSConsumer:consumeMessageSet where it could read, but not acknowledge or commit, an extra message when batchCounter = MAX_MESSAGES_PER_FLOW_FILE

Fixed JMSPublisherConsumerIT integration test which didn't account for when consumeMessage() did not read a message.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

Pull Request Tracking

  • Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
  • Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

  • Pull Request based on current revision of the main branch
  • Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

  • Build completed using mvn clean install -P contrib-check
    • JDK 21

Licensing

  • [n/a] New dependencies are compatible with the Apache License 2.0 according to the License Policy
  • [n/a] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

  • Documentation formatting appears as expected in rendered files

@mosermw
Copy link
Member Author

mosermw commented May 13, 2024

Rebased to account for nifi-nar-bundles moving to nifi-extension-bundles. Removed the JMSPublisherConsumerIT testMultipleThreads integration test fixes since that test itself was removed.

This MR may not seem like an appealing change, but ConsumeJMS already supported batching messages when using a Record Reader/Writer. This MR promotes batching to be available in all configurations. And as a reminder this MR fixes an bug in JMSConsumer consumeMessageSet() where it could read then drop an extra message when batchCounter reaches MAX_MESSAGES_PER_FLOW_FILE.

@mosermw
Copy link
Member Author

mosermw commented Jun 4, 2024

Rebased onto main to resolve latest conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant