makefile: switch to using upstream dtv-scan-tables

[chewitt]

This PR switches the codebase to use the upstream https://git.linuxtv.org/dtv-scan-tables.git/ scan tables repo instead of using our own fork of someone else's fork of the upstream repo. The goals for this change are:

• Reduce downstream maintenance effort for Tvheadend
• Redirect and encourage user contributions upstream for the benefit of all Linux DTV users
• Promote usage of the upstream source to projects/distros that bundle/package Tvheadend

Sending changes to the linux-tv mailing list instead of creating a pull-request to our repo fork may/will require users to learn a new git trick but "it's not rocket science" and I will author a HOWTO guide on that topic for the README.md of our repo before we archive it, along with a future new-forum FAQ post or wiki article. I'm also happy to coach anyone on the process over IRC chat, and will do some outreach to authors of pending PRs submitted to our repo to encourage them to resubmit their changes upstream.

ping to @oliv3r for awareness

[Flole998]

I'm wondering what the "correct" repo is, our fork is based on https://github.com/crazycat69/dtv-scan-tables

[chewitt]

https://git.linuxtv.org/dtv-scan-tables.git is the root upstream source, but the linux-media list isn't always quick to merge the patches submitted. I'm in the process of reaching out to the maintainer(s) to have a chat.

[Flole998]

I've tried multiple re-runs and this appears to break some builds.

[chewitt]

I've done some testing and it fails when trying to clone the repo:

Cloning into '/home/chewitt/work/tvheadend/tvheadend/support/../data/dvb-scan'...
fatal: unable to access 'https://git.linuxtv.org/dtv-scan-tables.git/': gnutls_handshake() failed: Handshake failed

The probable cause is a cipher mismatch due to Trusty/Jessie being ancient. I'm looking for a workaround.

[Saentist]

With speed of updates/commits to all repos above,
is more easy to update tvheadend/dtv-scan-tables repo,
from any public repos and make PR to Linuxtv repo.
Linuxtv.org repo is not updated from one year.

Who knows maybe in feature TVHeadend repo will be main repo ;)
Guess more people can commit to GitHub then Linuxtv.
If both repos have same content, with one will prefer to use?

[Flole998]

If you find a maintainer for the tvheadend/dtv-scan-tables repo that would work aswell. Even though that maintainer should really consider maintaining the upstream repo instead.

[chewitt]

The workaround in Trusty is to perform the general apt-get install of the runner, then add the Xenial deb repo and install only curl which adds an updated gnutls needed for git commands to suceed. I suspect something similar is needed for Jessie .. am looking into that now.

[chewitt]

Jessie cannot authenticate the letsencrypt cert used by https://git.linuxtv.org. Setting git config --global http.sslVerify false is a workaround but it would be better to update the ca list; annoyingly update-ca-certificates doesn't work (as Jessie isn't maintained I guess) so something else will be needed.

[chewitt]

Forcing sudo update-ca-certificates --fresh seems to work on Jessie (avoiding the need to disable verification). The same trick doesn't work on Trusty as the openssl version is too old to handle the updated letsencrypt chain, and since we compile against openssl we can't use the Xenial update trick so disabling verification is required.

[CvH]

you can also download the files directly from the site
https://linuxtv.org/downloads/dtv-scan-tables/dtv-scan-tables-2022-04-30-57ed29822750.tar.bz2
there is also a latest
https://linuxtv.org/downloads/dtv-scan-tables/dtv-scan-tables-LATEST.tar.bz2

no need to clone the files if this is easier, but no idea if this still works at prehistoric distributions

btw its just a matter of time till you can't download from github either with this EOL distros

[Saentist]

If you find a maintainer for the tvheadend/dtv-scan-tables repo that would work aswell. Even though that maintainer should really consider maintaining the upstream repo instead.

Make me maintainer then.
What is this fear, I can't understand.

[Flole998]

You should apply to linux-media to be a maintainer, we want to get rid of our fork, and that requires a well-maintained upstream repo.

[Saentist]

You should apply to linux-media to be a maintainer, we want to get rid of our fork, and that requires a well-maintained upstream repo.

Read again may previous post #1594 (comment)
Sometimes you have to take responsibilities and not run away from them.
Passing the responsibility to someone else is also not responsible.
Procrastination and procrastination are also bad manners.

[Flole998]

@Saentist You've had multiple warnings in the past about unacceptable behavior from me and from various other people. Accusing me (or others) of procrastinating is rude and unacceptable, so you have been blocked for 3 days from this repo. You can see this as a chance to reflect your behavior, and as a sign that I am not procrastinating.

[chewitt]

Updating certs on Jessie works but this requires admin rights (su to root or use sudo) and the base Debian Jessie image in the container doesn't contain sudo and we can't use interactive su commands in the script. You can install sudo, but that also requires interactive commnands (su to root) so it's not possible without changing the Docker to contain sudo. I haven't looked into that, but disabling sslVerify (same as Trusty) avoids the need. With the workarounds, all builds pass.

[chewitt]

I've moved the workarounds into CI to keep getmuxlist simple. In testing the cert refresh (alone) didn't work on Jessie so we're back to skipping verification (as with Trusty). It's not a desirable scenario but these OS are coming up on 9-years old so some hoop-jumping is to be expected. Everything builds (all green) now.

[Flole998]

I'll wait just a little longer before I merge this until they've done a little more merging at the LinuxTV repo, I think there's still a lot to do until they catch up....

[chewitt]

Upstream responded to my suggestion: https://lore.kernel.org/linux-media/[email protected]/ and I'm preparing to send some initial patches to the linux-media list based on the large diff from our current repo. I've added a MERGE BLOCKED label to the PR until things are more aligned.

[oliv3r]

With speed of updates/commits to all repos above, is more easy to update tvheadend/dtv-scan-tables repo, from any public repos and make PR to Linuxtv repo. Linuxtv.org repo is not updated from one year.

Who knows maybe in feature TVHeadend repo will be main repo ;) Guess more people can commit to GitHub then Linuxtv. If both repos have same content, with one will prefer to use?

I'm actually the official dtc-scan-tables repo maintainer :) and I've used to track github dtv repo's and pulled (manually!) in changes from repo's at some point, as nobody was willing to contribute really.

The 'nail in the coffin' to me, was that when tvheadend (one of the main consumers of these tables) decided it was a better idea to host the dtv-scan-tables themselves. This now meant, nothing was moving upstream anymore. Oh and the two repo's where different so that a different merge wasn't even possible anymore.

As for 'the repo is dead' (there was a cleanup done last week of some dead transponders), is because nobody is contributing, so no commits, no changes. (Though I admit, in part that is due to the imo horrible e-mail patch workflow). I wouldn't mind re-setting up a repo (probably on gitlab) to make contributions easier of course.

[oliv3r]

I'll wait just a little longer before I merge this until they've done a little more merging at the LinuxTV repo, I think there's still a lot to do until they catch up....

what do you suggest we do to move forward?

[oliv3r]

Upstream responded to my suggestion: https://lore.kernel.org/linux-media/[email protected]/ and I'm preparing to send some initial patches to the linux-media list based on the large diff from our current repo. I've added a MERGE BLOCKED label to the PR until things are more aligned.

Btw, it usually helps if you CC the maintainers (there's a nother one besides me :p)

[chewitt]

I'm used to the ./get_maintainers.pl workflow for patch submission but that depends on a MAINTAINERS file which doesn't exist for the scan tables repo so I started by emailing the linux-media list targetting Mauro after Hans flagged him to me via the #linux-media IRC channel on OFTC. I did also send you some direct messages (from my @kodi address) as I spotted your name as the upstream repo creator and figured you're the same person, but I assume messages ended up in a Junk folder as you haven't replied. At least we made contact now :)

See this thread for Mauro's response: https://lore.kernel.org/linux-media/[email protected]/

I plan to start with a patchset for DVB-C/T items we have that don't exist upstream. As there's no pre-existing content to compare with that should be simple to review and merge. Dealing with modified items probably requires chronological comparison which will be more opaque. Worst case I'm fine to get 'most' of the changes upstream and switch. Breaking something will cause users to report issues which can be fixed.

Mauro mentioned some scripts he has for grabbing ISDB-T/DVB-S changes. I'd like to see those added and adapted for automation. If the repo is regularly updated we can perhaps think about providing a deb/rpm package. This would also simplify buildsystem dependencies.

Also see this for something I plan to host on our repo once things are in-sync: chewitt/dtv-scan-tables@a342c75 .. because users will continue to send changes directly to us out of habit, ignorance, or laziness. It would be nicer to have a proper form with mandatory input items but GitHub doesn't support them for PRs the same as they do for issues.

[oliv3r]

a Junk folder as you haven't replied. At least we made contact now :)

Quite likely :) Or, I just missed it. Apologies!!

See this thread for Mauro's response: https://lore.kernel.org/linux-media/[email protected]/

I saw that thread (I think you linked it above) and I didn't see my name in the CC :p hence why.

Also, iirc the linux-media isn't even the right list, as the proper list is the 'dtv-utils' or 'v4l' or ... I think it was 'linuxtv'!! or something other :)

Mauro mentioned some scripts he has for grabbing ISDB-T/DVB-S changes. I'd like to see those added and adapted for automation.

I agree, I wonder what his 'source of truth' is though. I once made a script to get some stuff from kingofsat i think it was years ago. But how 'accurate' is kingofsat ...

It would be nicer to have a proper form with mandatory input items but GitHub doesn't support them for PRs the same as they do for issues.

It's nice that people still contribute to the tables. I wonder how feasable it is to redirect them to a webpage. I once thought of crating a simple dumb web 1.0 style <form> where they can fill in their details, and it would use the gitlab API to trigger a MR. But, time, effort (mostly time though) :S