RARBG Import - Incomplete Torent List?

[uninvitedguest]

I ran the RARBG import as described on the bitmagnet website, however after completion I found that the tracker lists 1,106,737 entries while the rarbg_db.sqlite database contains 2,843,691 entries.

Is this the expected outcome? There were no errors indicated during the import process.

[Send8213]

Did you keep the part of the example code that filtered out a number of categories:

  where
    cat not like '%_720' and
    cat not like '%_SD' and
    cat not like 'software%' and
    cat != 'xxx' and

That probably would have excluded quite a few from the import.

[uninvitedguest]

Ah, thank you. That would do it.

Are you aware of the process to wipe the previous import and start fresh? Or would a new import add only unique entries?

[Send8213]

Looks like it will be fine from what I can see. I tried reimporting the first few from that and didn't end up with duplicates and there's no torrents with more than two sources (rarbg and dht) in torrents_torrent_sources when grouped by info_hash. I was originally worried it would duplicate sources like "Source: RARBG, RARBG" but that does not seem to be the case. Part of me now wants to write a script to import everything from my torrent client into bitmagnet.

As for what you can do, you could just edit the sql and remove the "not"s and change the "and"s to "or"s (except the last that's "and true"), and it should only import the things that match those categories, inverting the original query.

You can also always make a backup of the various tables with pg_dump just in case (and I recommend making a backup once you've finished the import and let the classifier churn through everything, it may take awhile).