[ENH] Automate generation of DRIVER_TRANSLATE and driver colors

[psychemedia]

Proposed new feature or change:

The DRIVER_TRANSLATE lookup in plotting.py is currently defined explicitly:

Fast-F1/fastf1/plotting.py

Line 134 in f935c19

DRIVER_TRANSLATE = {'LEC': 'charles leclerc', 'SAI': 'carlos sainz',

Would it make more sense to generate this automatically, eg from a lookup via the ergast API?

For example:

from fastf1.ergast import base_url, _parse_ergast, _parse_json_response, _headers
from fastf1.api import Cache

# Generalise the ergast parser a bit more
def _parse_ergast(data, table="RaceTable:Races"):
    """Parse specified table from ergast data."""
    data_table = data['MRData']
    for t in table.split(":"):
        data_table = data_table[t]
    return data_table

# --- Lookup drivers ---
#http://ergast.com/api/f1/drivers
#http://ergast.com/api/f1/2010/drivers
#http://ergast.com/api/f1/2010/2/drivers
def fetch_driver(year, gp=None):
    """Get driver information from ergast API by year and optionally by race."""
    if gp is None:
        url = f"{base_url}/{year}/drivers.json"
    else:
        url = f"{base_url}/{year}/{gp}/drivers.json"
    return _parse_ergast(_parse_json_response(
        Cache.requests_get(url, headers=_headers)),
                         "DriverTable:Drivers"
    )

from unidecode import unidecode

def driver_translate(year, gp=None):
    """Get a driver translate lookup dict via ergast API."""
    def _n(d):
        #Normalise driver name
        return unidecode(f'{d["givenName"]} {d["familyName"]}').lower()

    drivers = fetch_driver(year, gp)
    _DRIVER_TRANSLATE = {d["code"]: _n(d) for d in drivers}
    return _DRIVER_TRANSLATE

DRIVER_TRANSLATE = driver_translate(2022)
# Note that we get *guanyu zhou* rather than the current *zhou guanyu*

[theOehrly]

Yes, this certainly needs to be improved at some point. These values should not be defined like this. And it should be possible to get them for any season. So your approach through Ergast is reasonable. But we also need teams names and colors. This should all be unified. And I want to phase out the access through attributes in fastf1.plotting as it is now.
So we need an approach that deals with all of this.

[theOehrly]

@lmontrieux, you noted this problem in #340. There's is an API endpoint that offers the necessary information, at least partially. If you like to take a shot at this, the TODO for implementing that would approximately be:

• update api.driver_info to support at least team colors as well (other driver information could be added, but is not currently required). Note that there is an alternative API endpoint using the url .../DriverList.json instead of .../DriverList.jsonStream that is way more straightforward to parse, but then we'd lose support for recorded live timing data which has the json stream format.
• add some sort of color modification algorithm that creates secondary colors from the team colors
• the color data then needs to be accessible through Session, this needs to be implemented
• a deprecation warning needs to be added to the old TEAM_COLORS and DRIVER_COLORS dictionaries

This is all fairly self-contained and has minimal interaction with existing code, so it should not be too difficult. Still, it is a bit of work.

[theOehrly]

Talking about colour selection. It looks like the team colour == the first driver's colour (in some cases we could debate who's the first and second driver until we're all blue in the face but let's not do that now 😄 ). However I wonder how the second driver colours are selected? Is it simply "something that looks a bit lighter", or is it a more precise process to it?

Regarding your question from the other issue. Yes, the first driver gets the team color. The second driver gets a color that is a bit lighter. There was nothing precise to this process so far. First and second driver are either defined by the order that the API returns them in or by the order in the driver names dictionary in FastF1. Whoever needs to merge the PR (me) shall not debate until we're all blue in the face about driver preferences 😁.

[lmontrieux]

I'll give this a shot - bit busy these days so no promises on timelines, but my idea for driver colour is to derive them from the team color. I'm thinking of converting the team color to HSL, and then change the lightness value by x% if one direction for the second driver. A third driver? Double the lightness change. I'm not sure there ever was a fourth driver for any given team in a season, but with 15% increments we can support at least 4 drivers. Assuming the colours look sufficiently different then, I haven't checked yet

[lmontrieux]

Mmmmh no that doesn't work well. I tried making a few manual changes, and for example the 2nd and 3rd Aston Martin drivers are getting colours very similar to the Mercedes ones. I'll think of something else.

[Casper-Guo]

I played around with using LAB color space but ran into similar issues. I looked at all the current color pairs in different encodings and some pairs vary much more than others (e.g. the Aston colors are much more different than the Mercedes colors). I think it is at least safe to say that the current colors are hand-picked rather than generated algorithmically

[theOehrly]

@Casper-Guo They are absolutely hand-picked. Even by different people. And they don't need to stay exactly the same.

[theOehrly]

@Casper-Guo @lmontrieux I've been playing around with this as well, but I currently don't really have enough spare time, so this got stuck. I thought, I'd just share my current results here.

These are some test results I have so far.

Click to show

I currently prefer the "monochromatic" colours the most.

I also calculated a "perceived monochrome difference" for each of those colour pairs.

# For Figure 4 - Monochromatic
#900000 Perceived brightness difference: 0.06333333333333332
#2b4562 Perceived brightness difference: -0.201318431372549
#0090ff Perceived brightness difference: 0.1424074509803922
#006f62 Perceived brightness difference: -0.23524941176470593
#dc0000 Perceived brightness difference: 0.06333333333333335
#ffffff Perceived brightness difference: 0.3019607843137255
#ff8700 Perceived brightness difference: 0.17919176470588238
#00d2be Perceived brightness difference: 0.23553215686274498
#0600ef Perceived brightness difference: 0.023438039215686282
#005aff Perceived brightness difference: 0.09751960784313729

You can see that this varies greatly. My current idea is that it might be possible to normalize the change of the colour value to the perceived monochrome difference. For example, always have a perceived monochrome difference of 0.15. Currently, some colours have a larger change while others have a smaller one.

Maybe someone is interested in playing around with this some more.

Here's a gist of the script:

https://gist.github.com/theOehrly/672d998955cd481c109d3ccbe27dd9a9

[Casper-Guo]

Interested in seeing what those 4 schemes each generate for Riccardo since we'll need to add him very soon

[theOehrly]

@Casper-Guo you mean, because that necessitates a third color?

I have been wondering about whether this automatic color palette generation isn't just a huge waste of time. Given that I don't really know what I'm doing... color science isn't my thing really.
Maybe a more efficient approach would be to simply define a team color palette with three or maybe four colors manually. And then driver colors are just taken from there.

[Casper-Guo]

you mean, because that necessitates a third color?

That would be consistent with what was done for Drugovich. I didn't notice the new color for Riccardo is so similar to the old color for De Vries. The three Aston drivers have distinct enough colors. This is a bit problematic because I think people will definitely try to visualize these three drivers together in terms of race pace, qualifying gap etc. That is why I was interested in what the four algorithms will come up with. My preference is between brightened and monochromatic btw.

Maybe a more efficient approach would be to simply define a team color palette with three or maybe four colors manually

The most ideal scenario is being able to get driver colors for any season, in which case this approach is certainly not feasible. It would be nice to just be able to do that for all the seasons where we can easily get team colors for drivers via API (I'm not too familiar with the specific endpoints). In that case, defining all the palettes is doable but of course, that's not the best long-term solution since we have essentially been doing the same thing.

[theOehrly]

I won't have time to properly revisit this before three or four weeks from now. So if you or anybody else wants to work on this, feel free to do so.

A proper approach with manually generated team color palettes would require color palettes for every season. In that case it might very well be feasible, I'd say?

[Casper-Guo]

I think I might have gotten confused. If we are only going to provide driver colors from 2018 onwards then the number of palettes is 100% manageable.

I thought the goal was to get driver colors for any season where Ergast can provide team colors tho? If this is the case then that's a ton of palettes that need to be designed.

[theOehrly]

Ergast doesn't provide team colors though (correct me if I'm wrong). And the F1 livetiming API provides data for 2018 and onwards only.

So if we want colors for older seasons, we'd need to find those manually first. I guess at that point, you could also just generate the whole palette and manually. Or maybe semi-automatic. Use something like the above script and review them afterwards with some manual corrections if necessary.

I'm probably going to spin up my own API backend anyway. The potential use cases are starting to get more and more. Then we could just add them gradually as well. And I'd like to make all the data open for collaboration (if technically feasible). So people can contribute data.

[theOehrly]

I've had some more thoughts about this. But no real work or testing done about feasibility.

This problem gets more complicated, the longer I think of it. What do you do with drivers switching teams during the season? For visualizations of one session, not a problem. For a visualization over the whole season, it is a problem.

I might try to model this as an optimization problem in LUV colour space (or another colour space). Basically, given 10 base colours (the team colours), generate N*10 colours that have the greatest perceived difference between them.
Also, it might be possible to implement an approach that returns a stable plot style per driver instead of only providing colours, a user-configurable combination of colour, marker, line style or hatch variation.

Maybe I also need to do some literature research. Someone else might have solved a problem like this one already.

I joked with some friends about hiring a working student for this problem recently. Only issue being, I can't pay them 😂

[Casper-Guo]

I don't quite understand how CIE color space works and how the difference is being measured here so I don't have anything to offer regarding the parameters.

But just based on the image, the Alfa, Ferrari, and McLaren colors look identical to me.

I will also try to do some research about human-perceived visual differences.

[theOehrly]

A short update here. I had taken a deep dive into the rabbit hole that is colour science in December. But in the end, no matter how much "science" you throw at the problem, finding 20 or more colours that can be distinguished while they are also recognizably similar within a team is simply a bad idea.

Therefore, the plan forward is the following:

1. Unique driver colours will be deprecated and removed in 2025.
2. There will be two colour maps, with one colour for each team. The default colour map will make use of secondary team colours for some teams, when beneficial (for example: pink for Alpine, yellow for Red Bull). The second colour map will provide the "official" team colours, which may be very similar between some teams.
3. Colours will be added for the 2018 to 2024 season.
4. A new function to generate driver styles for plots by varying colour, linestyles, markers and similar (highly configurable, but with easy to use defaults) will be added.

Additionally, the driver-team relationship, abbreviations, ... will be automatically loaded/generated from the F1 live timing API. This will require that all new plotting functions get a reference to the session as a required argument. As a benefit, this will be able to handle multiple seasons, as well as drivers switching teams during a season or any case of "Hulkenback".

About the not so great things, I really wanted to get this done before the season starts. Simply because it is nicer to not make this change in the middle of the season. And also, because the current system of hard coding all values only for the current season basically requires a breaking change at the start of the season.
But I have been held up a lot with compatibility fixes for dependencies at the start of this year. And I'm currently really swamped with other stuff. I have been really trying to get this done over the last week. And I'm frustratingly close. But I'm way past the "I enjoy working on this" point while trying to somehow get this done between everything else. I have more or less no time at all to work on FastF1 in the next two weeks. I might be somehow able to finish this in a mostly working state today. But I'm not going to drop a fairly major change barely tested and badly documented and then be gone for two weeks.

Therefore, the usual release that updates team colours and stuff will come before testing. It'll break compatibility with the 2023 season regarding team colours. The nice and shiny new features will likely come within the next month.