Port to Python 3

[BorreSanders]

Just installed Ubuntu 20.04 in which support for Python 2.x and pip has officially been dropped. Would be great to be able to continue to use tvgrabpyAPI on Python 3.

[kyl416]

Yeah, up until Thursday the python2 versions of python-requests and python-urllib3 were available on the repository for the release candidate. While the release notes did say that python2 would no longer be installed by default and moved to universe, unfortunately they neglected to mention that they will also be removing some of those key python2 packages from the respository.

Until we get a python3 versions of DataTree and tvgrabpyAPI working, here's the .deb packages for python-requests and its dependency python-urllib3 that were available in the release candidate. The rest of the python2 dependencies like python-tz and python-openssl are still available in the repository:
python-requests.zip

Also, if you previously installed it with "python setup.py", you'll have to remove the build folder and redo the install with "python2 setup.py"

[hikavdh]

This is work in progress. Most of the code is compatible, but this requires a lot of testing.

[meijjaa]

Would it be possible to create a Python 3 branch for testing? Anyways, thanks for the hard work!

[meijjaa]

Hi,

Tried to do some work as I miss my EPG data 😉
https://github.com/meijjaa/DataTree/tree/python3
https://github.com/meijjaa/tvgrabpyAPI/tree/python3

Currently the resulting output is this, not sure yet why it craps out with that last message:
./tv_grab_nl3.py
tv_grab_py_API does not yet support Pyton 3 or higher.
Expect errors while we proceed
Loaded the English texts file

[kyl416]

If you need guide data now and don't have another machine with a working python2 installation to use, install the deb packages for python-requests and python-urllib3 from the zip file I attached in post 2.

Since you're on Linux and know how to create your own git fork and commit to it, I'm guessing you already know how to do that, but for anyone else, extract them to a folder, open a terminal and navigate to that folder, and run the following:

sudo dpkg -i python-urllib3_1.25.8-1_all.deb
sudo dpkg -i python-requests_2.22.0-2build1_all.deb

If you get an error message about missing dependencies, you'll need to install them first:

sudo apt-get install python2 python2-dev python-tz python-certifi python-chardet python-idna python-cryptography python-socks python-six python-ipaddress ca-certificates

If you previously installed tvgrabpyAPI with "python setup.py install", you'll have to remove the build folder and redo the install with "python2 setup.py install" since Focal also removed the /usr/bin/python symlink.

[meijjaa]

Hi Kyl,
Thanks for the info! Getting the data isn't really a problem but this seemed like a nice challenge, I didn't realize what I was getting myself into though, this is quite a project 😄 interesting to see the challenges of converting huge Python 2 projects to Python 3.

I pushed my latest try with below log results, hope it helps you guys.

./tv_grab_nl3.py --configure
tv_grab_py_API does not yet support Pyton 3 or higher.
Expect errors while we proceed
Loaded the English texts file
Using config file: /home/jjm/.xmltv/tv_grab_nl3_py.conf
  Downloading tv_grab_API.json...
  Downloading tv_grab_nl.json...
Creating config file: /home/jjm/.xmltv/tv_grab_nl3_py.conf
Verifying the database

An unexpected error has occured:
Traceback (most recent call last):
  File "/opt/tvgrabpyAPI/tvgrabpyAPI/tv_grab_config.py", line 144, in grabber_main
    x = config.validate_commandline()
  File "/opt/tvgrabpyAPI/tvgrabpyAPI/tv_grab_config.py", line 775, in validate_commandline
    if not self.write_config(True):
  File "/opt/tvgrabpyAPI/tvgrabpyAPI/tv_grab_config.py", line 3181, in write_config
    chan_list[str(g)].sort(key=lambda channel: (channel['chan_string']))
TypeError: '<' not supported between instances of 'NoneType' and 'str'

If you want assistence, please attach your configuration and log files!
     /home/jjm/.xmltv/tv_grab_nl3_py.conf
     /home/jjm/.xmltv/tv_grab_nl3_py.log
Waiting for all remaining threads to finish
Waiting for all remaining threads to finish

./tv_grab_nl3.py
tv_grab_py_API does not yet support Pyton 3 or higher.
Expect errors while we proceed
Loaded the English texts file
Using config file: /home/jjm/.xmltv/tv_grab_nl3_py.conf
  Downloading tv_grab_API.json...
  Downloading tv_grab_nl.json...
Start time of this run: 2020-04-28 01:02
Verifying the database
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE tv SYSTEM "xmltv.dtd">
<tv generator-info-name="tv_grab_nl3_py (API-Version: 1.0.10-p20200209-beta)" generator-info-url="https://github.com/tvgrabbers/tvgrabnlpy">
</tv>

Execution complete.

Fetch statistics for 0 programms on 0 channels:
 Start time: 2020-04-28 01:02
   End time: 2020-04-28 01:02
   Duration: 0:00:05.242207
     2 page(s) fetched, of which 0 failed and 0 returned no data
     0 cache hits
     0 succesful ttvdb.com lookups
     0    failed ttvdb.com lookups
  Time/page: 2.6211035 seconds

     0 page(s) fetched from theTVDB.com
     0 failure(s) on theTVDB.com
...

Waiting for all remaining threads to finish

[kyl416]

Yeah, as you saw it's a lot more complicated than just changing references of "python2" to "python3". They changed the capitilazation of several libraries like "Queue" to "queue", and they also changed the parameters of many API calls so some segments of code might need a complete rewrite to reproduce the python2 behavior.

This was one of the big reasons why the python team kept python2 alive for an additional 5 years. The original plan was to drop support in 2015, but the changes were just too complex and many key projects weren't ready for python3. Even major projects like GIMP are still using python2 code for some of their plugins. i.e. on Focal if you install gimp-plugin-repository you'll get a bunch of errors in the log because Focal no longer ships gimp-python, and if you install python-is-python3 to create a symlink to python3, you'll get crashes when those plugins attempt to load with python3.

[meijjaa]

Yep they made it a serious refactoring job, luckily they have the "2to3" script to help out with a lot of the boring stuff but it's not perfect.

But I have good news, I got my fork working. Be warned I'm not a Python expert so I'm sure some things could be done better! 😃

Python 3 branches (I'll send a pull request if you like):
https://github.com/meijjaa/DataTree/tree/python3
https://github.com/meijjaa/tvgrabpyAPI/tree/python3

Just sudo ./setup install as usual for both projects and all should be fine with Ubuntu 20.04.

I tested with TvHeadend and manual:
tv_grab_nl3.py --configure
tv_grab_nl3.py

[hikavdh]

I am working on it, but as I had some mayor updates standing on standby, I'm including the jump to python3 into those. Wait 1 or 2 weeks and I'll publish an alfa for both DataTreeGrab and tvgrabpyAPI.

[meijjaa]

Hi Hikavdh, thanks for the update. Looking forward to it 👍

[rsenden]

Hi, any updates on this?

[udovdh]

Op Fedora 32 soortgelijke issues als ik een nieuwere versie uit git wil installeren.
python2-requests is er alleen voor Fedora 31 als ik de stup wil laten draaien.
Python3 is dan ook erg belangrijk voor deze situatie.

[udovdh]

Even worse than writing in English for a product that has NL in the name:
The grabber completely doesn't work under Fedora 32 because of the python 3 thing:

(...)
2020-10-23 14:30:24.629452 I [168106/168127] SystemManager mythsystemunix.cpp:262:run Starting process manager
2020-10-23 14:30:24.629564 I [168106/168129] SystemIOHandlerR mythsystemunix.cpp:82:run Starting IO manager (read)
2020-10-23 14:30:24.632875 I [168106/168130] SystemIOHandlerW mythsystemunix.cpp:82:run Starting IO manager (write)
2020-10-23 14:30:24.632969 I [168106/168128] SystemSignalManager mythsystemunix.cpp:488:run Starting process signal handler
2020-10-23 14:30:24.783522 E [168106/168106] CoreContext filldata.cpp:361:Run tv_grab_nl3.py --capabilities failed or we timed out waiting. You may need to upgrade your xmltv grabber
2020-10-23 14:30:24.783636 E [168106/168106] CoreContext filldata.cpp:663:Run Grabbing XMLTV data using tv_grab_nl3.py is not supported. You may need to upgrade to the latest version of XMLTV.
2020-10-23 14:30:24.804114 N [168106/168106] CoreContext main.cpp:409:main Data fetching complete.
2020-10-23 14:30:24.804154 I [168106/168106] CoreContext main.cpp:417:main Adjusting program database end times.
(cut)

So this must be a grave situation.

[hikavdh]

Before I issue an upgrade I do want to incorporate everything I have been developing on a parallel thread. With the present worldwide situation I have a hard time focusing and concentrating on finish this job, so please have patients.
On the bright side running the 2to3 script on the code seems to give a good running result, most of the code has been written and prepared with python3 in view.
Please have patience and you'll get more then you had.

By the way, one of our developers is American. He does understand dutch, but give him some courtesy