Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Excessive Queue job logged when use with batch import of large amount of data (#846) #884

Closed
kenfai opened this issue Dec 21, 2023 · 3 comments

Comments

@kenfai
Copy link

kenfai commented Dec 21, 2023

Q A
Bug? no(?)
New Feature? yes
Framework Laravel
Framework version 8.x
Package version 13.6.3
PHP version 7.4.x

Actual Behaviour

When using Supervisor to manage process queues and job logging, and Laravel Excel package to import large amount of data to the database, a high amount of logging been produced from the Queue Auditing:

[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:32][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processing: OwenIt\Auditing\Listeners\ProcessDispatchAudit
[2023-12-21 14:36:33][] Processed:  OwenIt\Auditing\Listeners\ProcessDispatchAudit

I believe this issue happened due to a recently added Queue Auditing feature from PR:#846.

Expected Behaviour

I understand this may not be related to Laravel-Auditing library, but is there any way to disable logging for the above audit listener class by default?

This excessive logging is polluting the actual Queue job processes in the Supervisor's stdout_logfile file, making it hard to read.

Steps to Reproduce

  1. Setup Supervisor to manage Queue processes in Laravel: https://laravel.com/docs/8.x/queues#supervisor-configuration

  2. Create an Laravel-Excel Import class: https://docs.laravel-excel.com/3.1/imports/

  3. Upload an excel file with data to be imported to the database.

  4. Monitor or read the Supervisor stdout_logfile file to see the logs.

Possible Solutions

Allow audit Listener to be set to 'silent' mode from config/audit.php?

@parallels999
Copy link

parallels999 commented Dec 21, 2023

Did you try audit.queue.enable => false on config/audit.php??

'enable' => true,

@kenfai
Copy link
Author

kenfai commented Dec 22, 2023

Did you try audit.queue.enable => false on config/audit.php??

'enable' => true,

Okay I have just applied the settings. Didn't realise when upgrading.

The feature is also causing Out of Memory issue as such:

[2023-12-22 11:33:07] production.ERROR: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 90181632 bytes) {"userId":500,"exception":"[object] (Symfony\\Component\\ErrorHandler\\Error\\FatalError(code: 0): Allowed memory size of 1073741824 bytes exhausted (tried to allocate 90181632 bytes) at /var/www/html/vendor/laravel/framework/src/Illuminate/Queue/Queue.php:158)

Will see if this fixes the issues.

Thanks!

@angeljqv
Copy link
Contributor

Log Levels: Laravel supports various log levels such as emergency, alert, critical, error, warning, notice, info, and debug. Use appropriate log levels to differentiate between different types of log entries. Avoid excessive logging to prevent performance issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants