Optimize time to add constraints JuMP 0.19.0

[Thuener]

The benchmark code presented in JuMP-dev 2019(https://www.youtube.com/watch?v=MLunP5cdRBI):
https://gist.github.com/Thuener/5fd30bda29a84afb126cb5b723574eba

########### Vectorized - WithoutDirect ###########
Memory consumption 1111 Mb and Time 20.904308405
Time to solve model 5.079105861
Memory consumption 1118 Mb and Time 20.27646884
Time to solve model 5.080772802
########### Scalar 1 - WithoutDirect ###########
Memory consumption 997 Mb and Time 12.158498048
Time to solve model 5.005659555
Memory consumption 992 Mb and Time 12.178707011
Time to solve model 5.056525515
########### Scalar 2 - WithoutDirect ###########
Memory consumption 854 Mb and Time 13.162695245
Time to solve model 5.002032299
Memory consumption 1065 Mb and Time 12.788777522
Time to solve model 4.613660219
########### Low-level API ###########
Memory consumption 304 Mb and Time 2.538083196
Time to solve model 4.572178837
Memory consumption 367 Mb and Time 2.641781411
Time to solve model 4.766848752
########### Vectorized - WithDirect ###########
Memory consumption 587 Mb and Time 12.18446959
Time to solve model 5.053543398
Memory consumption 482 Mb and Time 12.490777377
Time to solve model 5.095189864
########### Scalar 1 - WithDirect ###########
Memory consumption 405 Mb and Time 5.850087326
Time to solve model 4.78109017
Memory consumption 456 Mb and Time 5.842091906
Time to solve model 5.22244529
########### Scalar 2 - WithDirect ###########
Memory consumption 440 Mb and Time 6.361990639
Time to solve model 5.057280697
Memory consumption 440 Mb and Time 6.51641373
Time to solve model 4.437310264

[Thuener]

One of the problems is the ordered dictionary. There is an issue open in Julia requesting a function to update values in-place JuliaLang/julia#31199
@joaquimg

[joaquimg]

Another bottleneck is canonicalize on MOI.
We might want to implement a special canonicalize in LQOI as suggested by @mlubin in JuMPdev 2019.

[Thuener]

We use ProfileView to understand the bottlenecks in the Direct Mode. I couldn't share the data but I did some nice graphs in paint 😉
I just include the code in the gist and did:

using Profile
Profile.clear()
@profile test_const(3; direct = true)
using ProfileView
ProfileView.view()

[ndinsmore]

@Thuener Could you re-run your benchmark making sure that the following PR is applied to the version MOI you are using. jump-dev/MathOptInterface.jl#696, which ensures that when copying the caching optimizer to the solver it uses add_constraints rather than the singular version. Best to use the caching optimizer, because I know that with Clp I have seen 30x improvements in copy time.

[ndinsmore]

A few notes julia v1.2 will have the method map!(f,values(dict)) which was added in JuliaLang/julia#31223. I have opened a PR to add that functionality to OrderedDict and LittleDict JuliaCollections/OrderedCollections.jl#22. So we could make use of it and it should improve the perf.

That being said the code says that for Dicts smaller that 30 elements LittleDIct should out perform OrderedDict.

[joaquimg]

I know we want to avoid dependencies, but we should keep these fast dictionaries in mind:
https://github.com/andyferris/Dictionaries.jl

[joaquimg]

I have opened a PR to add that functionality to OrderedDict and LittleDict JuliaCollections/OrderedCollections.jl#22. So we could make use of it and it should improve the perf.

JuliaCollections/OrderedCollections.jl#22 was replaced by JuliaCollections/OrderedCollections.jl#41
and merged

[odow]

See #2750 for an improvement on the dictionary front. There's also #1939 to track the plural issue. We've also introduced direct_model, removed LQOI, and entirely rewritten the CPLEX wrapper since this issue was opened.

Overall, I think performance of JuMP as it stands is acceptable.

For the same reason as #1403, I'm going to close this in favor of more general performance issues such as #2735 and #42 (a 2-digit issue!) to track on-going performance. Ideally, we'll set up something that makes it easy to benchmark JuMP, and then keep attacking that. We can also open issues if specific performance problems are identified.