actions/python installs too many requirements

[mdbenito]

In the "setup python" github action, both dev and docs requirements are always installed. But this action is used e.g. in the build docs stage in the main workflow, where dev requirements shouldn't be necessary.

[AnesBenmerzoug]

That takes about 40s though. How much time would we shave off that?

[mdbenito]

Half? But that accrues over hundreds of runs and I don't see the point of wasting build minutes without reason.

[AnesBenmerzoug]

I did a bit of benchmarking and it turns out that installing the documentation dependencies takes as much time as if not more than the main dependencies (not including any optional ones like PyTorch for example). You're right, we should optimize that.

	Docs dependencies	Main dependencies
Without Cache	real 1m0,479s user 0m48,623s sys 0m4,046s	real 0m52,664s user 0m41,056s sys 0m2,824s
With Cache	real 0m41,236s user 0m35,469s sys 0m2,846s	real 0m46,728s user 0m42,487s sys 0m3,201s