Indexing - performance warning - full index can result in a large number of pairs

[gajghaten]

Using warnings.filterwarnings("ignore") does not disable this warning.

I use this function on a dataframe much bigger than the one in the example above and that results in a bunch of these warnings being displayed on the screen.

Could I please get help in disabling them? There are not many resources online on how to disable these kinds of warnings.

[rohitgarud]

@gajghaten I think the package uses logging to display warnings. Try the following code

import logging
logging.getLogger("recordlinkage").setLevel(logging.ERROR)

[gajghaten]

@rohitgarud Thanks! That worked!

The following helps too!

logger = logging.getLogger('recordlinkage')
logger.disabled = True

# Your code

logger.disabled = False

[rohitgarud]

@gajghaten Glad to help. But I think you should absolutely never use full index and always use blocking or sorted index. Full index will give (n choose 2 i.e. n*(n-1)/2) pairs, which increases quadratically with the number of records and slows down the record linkage or deduplication process significantly