New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"RuntimeError: Attempt to pop from an empty stack" is raised when running models fit in parallel with threads. #2620
Comments
Same problem when i'm trying Optuna with n_jobs=-1 [W 2024-04-09 17:36:10,609] Trial 99 failed with parameters: {'iterations': 1831, 'learning_rate': 0.08854663531454893, 'subsample': 0.8477806412881673, 'depth': 6, 'border_count': 182, 'l2_leaf_reg': 60.04725887937673, 'colsample_bylevel': 0.768423720172805} because of the following error: RuntimeError('Attempt to pop from an empty stack'). It happens in the end of a process |
We close the issue when the fix is committed to the master branch. New CatBoost release 1.2.5 includes the fix. |
I installed CatBoost 1.2.5, but I still have this error when running with multiple threads. I'm getting similar errors to the ones posted above. |
Then please describe in maximum detail (CPU, GPU, OS, python version, how exactly do you install catboost) how to reproduce it. I'm trying the example in this ticket and it works fine with CatBoost 1.2.5. |
Running with njobs = 4 [W 2024-04-22 16:55:15,244] Trial 21 failed with parameters: {'objective': 'CrossEntropy', 'colsample_bylevel': 0.01556629158209212, 'depth': 10, 'bootstrap_type': 'Bernoulli', 'l2_reg': 0.4445241346856548, 'score': 'Cosine', 'grow_pol': 'Depthwise', 'samp_freq': 'PerTree', 'subsample_bern': 0.10263351289964752} because of the following error: RuntimeError('Attempt to pop from an empty stack'). File ~/miniconda3/lib/python3.9/site-packages/catboost/core.py:2433, in _fit(self, X, y, cat_features, text_features, embedding_features, pairs, sample_weight, group_id, group_weight, subgroup_id, pairs_weight, baseline, use_best_model, eval_set, verbose, logging_level, plot, plot_file, column_description, verbose_eval, metric_period, silent, early_stopping_rounds, save_snapshot, snapshot_file, snapshot_interval, init_model, callbacks, log_cout, log_cerr) File ~/miniconda3/lib/python3.9/contextlib.py:126, in _GeneratorContextManager.exit(self, typ, value, traceback) File ~/miniconda3/lib/python3.9/site-packages/catboost/core.py:186, in log_fixup(log_cout, log_cerr) File ~/miniconda3/lib/python3.9/site-packages/catboost/core.py:165, in pop(self) RuntimeError: Attempt to pop from an empty stack catboost version: 1.2.5 No GPU, Python 3.9.17, installed via pip install catboost==1.2.5 |
Problem: "RuntimeError: Attempt to pop from an empty stack" is raised when running models fit in parallel with threads.
Similar to: #1855
The custom logger stack is not thread safe and raises the RuntimeError.
This does not happen in version 1.2.2, the code finishes successfully.
How to reproduce:
catboost version: 1.2.3
Operating System: Amazon Linux 2023.4.20240319
CPU: Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 4
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
GPU: none
The text was updated successfully, but these errors were encountered: