Import time

[henryiii]

I've noticed that something that changed recently has caused the import time to skyrocket. (This could even just be in my branch). Running this in Python 3.7 with python -X importtime shows a whopping 10+ seconds devoted to particle.particle.literals!

[henryiii]

Loading the nuclei table takes 10 seconds. @eduardo-rodrigues we need to address this before releasing. You could load the entire numpy library almost 100 times in this timeframe. Particle was designed to load a few hundred particles in a table, not 6,500. (though the speed per object is rather dismal...)

[henryiii]

Ran py-spy and the flame graph showed that all the time was in in the search for an existing particle. I've moved the internal cls._table to a set instead of a list and used MyPy to help me fix all the necessary changes. (Particle was always designed to be hashable via PDGID number). It now runs in 0.1 second instead of 10 seconds and worked the first time (after the MyPy assistance).

[eduardo-rodrigues]

@henryiii, these timing issues you are seeing is something I had sorted out a while back, see https://github.com/scikit-hep/particle/pull/231/commits and 55486ea specifically. I'm very surprised you are seeing the same kind of issues again. It was all working fine. I have no idea what happened in between the developments. Good anyway that you find another solution! I will look closer at the updates today ...

[eduardo-rodrigues]

Just confirming - on my Windows 3.7.7 local installation all goes fast. Hmm …

[eduardo-rodrigues]

Loading the nuclei table takes 10 seconds. @eduardo-rodrigues we need to address this before releasing. You could load the entire numpy library almost 100 times in this timeframe. Particle was designed to load a few hundred particles in a table, not 6,500. (though the speed per object is rather dismal...)

The issue with timing was indeed something I saw when adding information on nuclei, hence the discussion at the time since trouble for information that >95% of our users will not care about was a no-go. Indeed there are many isotopes added …

BTW, I'm working on a complete rewrite of the data files, which will be much easier to maintain and rather comprehensive. This relates also to #118 and will be much nicer. News as soon as I find more time to add the new data per categories - quarks, leptons, light mesons, charm baryons, etc.; you get it.

[eduardo-rodrigues]

Ran py-spy and the flame graph showed that all the time was in in the search for an existing particle. I've moved the internal cls._table to a set instead of a list and used MyPy to help me fix all the necessary changes. (Particle was always designed to be hashable via PDGID number). It now runs in 0.1 second instead of 10 seconds and worked the first time (after the MyPy assistance).

I have not played with this so if you have some commands to teach me and give it a go, I would be very interested. Thanks a lot!

[henryiii]

The problem was that I had to remove the file chaining that was being done, and that removed your speed hack. I might be able to restore the speed boost there too, and get even more! (Greedy cackling). I will check. The set nicely speeds up user code too, while the old hack only helped the original file loading.

[eduardo-rodrigues]

Agreed that using set is neat :-).

Yep, we may be able to be even faster ...

[henryiii]

Set checking is order 1; I can even drop the initial check for append entirely. (If you have multiple particles with the same PDGID, the last one will always be present). There's no performance difference for turning it off at all.