Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Ultimate Way of Browsing Channels #1320

Open
wants to merge 14 commits into
base: master
Choose a base branch
from

Conversation

emrakyz
Copy link
Contributor

@emrakyz emrakyz commented May 2, 2023

I forgot to add after the change from Bash to Dash I also simplified this: ~/.local/share/channels.txt looks like below:
Luke Smith=https://www.youtube.com/@LukeSmithxyz/videos
Mental Outlaw=https://www.youtube.com/@MentalOutlaw/videos

~/.local/share/categories.txt:
Tech=Luke Smith|Mental Outlaw

The script works extremely fast except the first time to update the whole data. It takes about 2 minutes to update the whole database with 80 different channels. This is parallel though, the biggest channel determines the total time. You can set a cronjob for this. It's not a heavy work for the PC. It justs fetch the text data with yt-dlp. Video example below:

This script is a sophisticated and ingenious tool designed to streamline your YouTube experience by organizing and managing your favorite YouTube channels, allowing you to browse and watch videos directly within the script without ever visiting its website. You can assign the channels with various categories such as "Tech", "Science", "Sports", etc. The videos can be played using the 'mpv' media player. Moreover, the script allows you to sort videos based on view count or duration; download videos; and even maintain a "Watch Later" list. If you combine this script with "SponsorBlock" lua script created for "mpv", then you will have the ultimate experience. SponsorBlock removes all sponsored segments in a video including intros, outros or similar unnecessary parts. It's normally a browser extension but is also available for "mpv".

No browsers, accounts, distractions, crappy algorithm and recommendations, advertisements, sponsors, intros, outros, fillers or empty spaces. We eliminate them all.

Video Example:

yt-browser.webm

Required Programs: dmenu | mpv | jq | yt-dlp

FEATURES

  • Browse all videos from all channels you set at the same time. You can filter titles through dmenu.
  • Browse a channel's videos.
  • Select a channel either from the main menu or inside a Category.
  • Watch, Download or Put videos in a "Watch Later List".
  • Sort videos by view or duration. The default sort is upload date. The only problem is, we can't have the exact upload date, so we can't apply much more advanced filtering. It can be done but it makes fetching the data for the first time too slow.
  • The menus have a complex loop system. It always continues where you left off. The script doesn't close itself when you make a selection. So you don't have to run the script over and over again and get to where you left off. You can also press Escape to return to a prior menu.
  • You won't see the URLs or any unnecessary things inside dmenu. Just the titles.

JUSTIFICATION

This script is incredibly beneficial for those who seek a minimalist and focused approach to consuming content on YouTube. By providing a CLI-based interface (dmenu), the script reduces distractions and clutter that are commonly encountered on the Youtube website. It allows users to personalize their content consumption and manage channels more effectively. The script is also remarkably efficient and easy to navigate, providing a user-friendly experience that saves time and promotes productivity.

The script is organized into functions that each perform a specific task, such as updating channel data, retrieving video titles, playing videos, downloading videos, adding videos to the watch later list, and browsing all channels. These functions are called by the main script to provide the user with various options for navigating and interacting with the videos.

The script makes use of various Bash features such as associative arrays, shell redirection and piping, to simplify and streamline the code. It also uses conditionals and loops to handle different user input and error cases. Overall, this script is a powerful and flexible tool for browsing, watching, organizing YouTube channels, and it provides a great example of Bash usage to automate and streamline complex tasks.

Why Does The Script Use "Bash"? (It works with Dash now)

  • Associative arrays: Dash only supports indexed arrays, not associative arrays. The script uses associative arrays for both CHANNELS and CATEGORIES.

  • Indirect parameter expansion: The ${!array[@]} notation, which is used to get the keys of an associative array, is a Bash-specific feature.

  • declare command: The declare command is not available in Dash. In this script, it is used to define associative arrays.

  • To make the script compatible with Dash, you would need to replace the associative arrays with a compatible data structure or use separate indexed arrays for keys and values. You would also need to replace or modify the parts of the script that use indirect parameter expansion or the declare command. However, doing so might make the script more complex and harder to maintain.

DETAILED EXPLANATION

  • The script begins by defining two associative arrays, CHANNELS and CATEGORIES, which store the YouTube channel names along with their respective URLs and categories. It then sets the directories for storing data and videos, and creates them if they do not already exist.
  • The 'update_data' function updates the metadata for a given channel, while the 'update_all_channels' function updates metadata for all channels. The metadata includes the video title, URL, view count, and duration, which are extracted using 'yt-dlp' and 'jq' utilities.
  • The 'get_videos' function retrieves the video titles from the metadata of a given channel, sorted by the specified criteria. The 'video_url' function returns the URL of a video based on its title and channel name. The 'play_video' and 'download_video' functions use 'mpv' and 'yt-dlp', respectively, to play or download a video given its title and channel name.
  • The 'add_to_watch_later' function appends the video title and channel name to a watch later list, while the 'play_watch_later' and 'delete_from_watch_later' functions play a video from the list or remove it, respectively.
  • The 'get_all_videos' function retrieves all video titles from the metadata of all channels, sorted by the specified criteria. The 'browse_all_channels' function lets you browse through all channels and select a video to watch, download, or add to the watch later list.
  • The main part of the script first prompts the user to update the database of channels. If the user chooses to do so, the 'update_all_channels' function is called. The script then presents the user with options to browse all channels, browse channels by category, or browse the watch later list. The script loops through these options until the user decides to exit.

The script works extremely fast except the first time to update the whole data. It takes about 2 minutes to update the whole database with 80 different channels. You can set a cronjob for this. It's not a heavy work for the PC. It justs fetch the text data with yt-dlp. Video example below:

This script is a sophisticated and ingenious tool designed to streamline your YouTube experience by organizing and managing your favorite YouTube channels, allowing you to browse and watch videos directly within the script without ever visiting its website. You can assign the channels inside various categories such as "Tech", "Science", "Sports", etc. The videos can be played using the 'mpv' media player. Moreover, the script allows you to sort videos based on view count or duration; download videos; and even maintain a "Watch Later" list. If you combine this script with "SponsorBlock" lua script created for "mpv", then you will have the ultimate experience. SponsorBlock removes all sponsored segments in a video including intros, outros or similar unnecessary parts. It's normally a browser extension but is also available for "mpv".

No browsers, accounts, distractions, crappy algorithm and recommendations, advertisements, sponsors, intros, outros, fillers or empty spaces. We eliminate them all.

Required Programs: dmenu | mpv | jq | yt-dlp

FEATURES
1. Browse all videos from all channels you set at the same time. You can filter titles through dmenu.
2. Browse a channel's videos.
3. Select a channel either from the main menu or inside a Category.
4. Watch, Download or Put videos in a "Watch Later List".
5. Sort videos by view or duration. The default sort is upload date. The only problem is, we can't have the exact upload date, so we can't apply much more advanced filtering. It can be done but it makes fetching the data for the first time too slow.
6. The menus have a complex loop system. It always continues where you left off. The script doesn't close itself when you make a selection. So you don't have to run the script over and over again and get to where you left off. You can also press Escape to return to a prior menu.
7. You won't see the URLs or any unnecessary things inside dmenu. Just the titles.



JUSTIFICATION
This script is incredibly beneficial for those who seek a minimalist and focused approach to consuming content on YouTube. By providing a CLI-based interface (dmenu), the script reduces distractions and clutter that are commonly encountered on the Youtube website. It allows users to personalize their content consumption and manage channels more effectively. The script is also remarkably efficient and easy to navigate, providing a user-friendly experience that saves time and promotes productivity.

The script is organized into functions that each perform a specific task, such as updating channel data, retrieving video titles, playing videos, downloading videos, adding videos to the watch later list, and browsing all channels. These functions are called by the main script to provide the user with various options for navigating and interacting with the videos.

The script makes use of various Bash features such as associative arrays, shell redirection and piping, to simplify and streamline the code. It also uses conditionals and loops to handle different user input and error cases. Overall, this script is a powerful and flexible tool for browsing, watching, organizing YouTube channels, and it provides a great example of Bash usage to automate and streamline complex tasks.

DETAILED EXPLANATION
- The script begins by defining two associative arrays, CHANNELS and CATEGORIES, which store the YouTube channel names along with their respective URLs and categories. It then sets the directories for storing data and videos, and creates them if they do not already exist.
- The 'update_data' function updates the metadata for a given channel, while the 'update_all_channels' function updates metadata for all channels. The metadata includes the video title, URL, view count, and duration, which are extracted using 'yt-dlp' and 'jq' utilities.
- The 'get_videos' function retrieves the video titles from the metadata of a given channel, sorted by the specified criteria (if any). The 'video_url' function returns the URL of a video based on its title and channel name. The 'play_video' and 'download_video' functions use 'mpv' and 'yt-dlp', respectively, to play or download a video given its title and channel name.
- The 'add_to_watch_later' function appends the video title and channel name to a watch later list, while the 'play_watch_later' and 'delete_from_watch_later' functions play a video from the list or remove it, respectively.
- The 'get_all_videos' function retrieves all video titles from the metadata of all channels, sorted by the specified criteria. The 'browse_all_channels' function lets you browse through all channels and select a video to watch, download, or add to the watch later list.
- The main part of the script first prompts the user to update the database of channels. If the user chooses to do so, the 'update_all_channels' function is called. The script then presents the user with options to browse all channels, browse channels by category, or browse the watch later list. The script loops through these options until the user decides to exit.
@TheYellowArchitect
Copy link
Contributor

TheYellowArchitect commented Nov 18, 2023

I first ran channel-refresh (as implied)
cannot open /home/USER/.local/share/channels.txt: No such file

I suggest a simple check, then creating it if it doesn't exist.
Same goes for categories.txt (I got same error as above)

I used this for yt-browser which I see is more recent than the above commits
https://github.com/LukeSmithxyz/voidrice/blob/96b997e8cf92134e5f8f7188a3b96ec300523b74/.local/bin/yt-browser

On pressing all videos on dmenu, it showed on dmenu "Choose a Video or enter @@sv or @@sd" awaiting input and showed the following output on the terminal:
sort: cannot read: /home/USER/.cache/youtube_channels/all_videos.tsv: No such file or directory

Anyway, cool scripts, the concepts got me hyped enough to try all 3 of them hahaha
While I think the url bookmarks script cannot be fixed (serious technical problems), this and the one with searching the hard drive should take a day to fix, and if fixed I would definitely give them a try myself, as I think they are a great addition to anyone's initial setup (I would definitely use them even if they were denied pull request)

@emrakyz
Copy link
Contributor Author

emrakyz commented Nov 18, 2023

@TheYellowArchitect

Haha :) I know that Luke probably won't merge all of these but I think his viewers are interested in things like this in general so I thought it would have been a good idea to share some of my scripts here. That's probably why he doesn't close the PRs even though he didn't accept some of these.

All of the pull request scripts work since I and some other friends consistently use them. I can help you fix the problems.

The script works like this:

You need yt-dlp to download videos and their metadata. You need jq because yt-dlp gives a JSON output so it's easy to manipulate the metadata with using the lightweight and fast jq. It's installed as a default in some distros but not for some others.

Then you need to have file named: ~/.local/share/categories.txt:

In that file we create categories like below:
Tech=Luke Smith|Brodie Robertson|Mental Outlaw
Art=Great Art Explained|The Art Assignment|TATE

And we need channel URLs in the file named: "~/.local/share/channels.txt" and it's formatted as below:
Luke Smith=https://www.youtube.com/@LukeSmithxyz/videos
Mental Outlaw=https://www.youtube.com/@MentalOutlaw/videos

So when you run "channelrefresh" script at this point, you download the metadata of the channels inside (channels.txt) with their name and the URL. I generally put this script to boot time so every time I boot my PC, I get notified as to which channels get updated.

So when channelrefresh operation is finished, you can run yt-browser script and see your channels and categories there.

I hope this helps. If not, please contact again and I am sure we can make it work.

By the way bookmark script is an extremely simple script and works as expected without a problem. It's impossible that it has serious technical problems. It just sends the URL to the browser as below:
$ firefox example.com

and if your URL has search in it it just adds your keyword at the end:
$ firefox example.com/search?=yourkeyword

So it's nothing crazy.

For the adding, removing and editing we use very basic sed commands. So there is no way that this script can have serious technical problems :) It's not a rocket science.

I can agree that it can be improved but for now adding more features than necessary is pointless. I once added a feature to use it as a URLBAR and it had a function to go to any URL even if it's not saved but then I thought it was pointless because we can just open the browser for it or use Vim extensions such as vimium-ff or vimvixen.

@emrakyz
Copy link
Contributor Author

emrakyz commented Nov 18, 2023

@TheYellowArchitect
By the way, please reach out if you need help. I think you can easily make all of the scripts here work without a problem. We can also make improvements if needed.

EDIT: I have added robust error handling and improved the way it waits for the network. Please use the latest version.

@TheYellowArchitect
Copy link
Contributor

TheYellowArchitect commented Nov 19, 2023

The script works like this:

You need yt-dlp to download videos and their metadata. You need jq because yt-dlp gives a JSON output so it's easy to manipulate the metadata with using the lightweight and fast jq. It's installed as a default in some distros but not for some others.

Then you need to have file named: ~/.local/share/categories.txt:

In that file we create categories like below: Tech=Luke Smith|Brodie Robertson|Mental Outlaw Art=Great Art Explained|The Art Assignment|TATE

And we need channel URLs in the file named: "~/.local/share/channels.txt" and it's formatted as below: Luke Smith=https://www.youtube.com/@LukeSmithxyz/videos Mental Outlaw=https://www.youtube.com/@MentalOutlaw/videos

So when you run "channelrefresh" script at this point, you download the metadata of the channels inside (channels.txt) with their name and the URL. I generally put this script to boot time so every time I boot my PC, I get notified as to which channels get updated.

So when channelrefresh operation is finished, you can run yt-browser script and see your channels and categories there.

I hope this helps. If not, please contact again and I am sure we can make it work.

It works. Damn, it's amazing to watch youtube via mpv at any time (even added SponsorBlock you suggested). Thank you very very much. I didn't even use the latest version you uploaded, it seems like I simply lacked the proper data at channels.txt and didn't have the package jq installed.

Man this is incredible. I hope this gets merged. But I would like to mention minor suggestions which are so minor that they can be ignored since the functionality is solid:

  1. Comments at the top of both bins (line 2 or 3), explaining the content format for channels.txt and categories.txt, and to run channelrefresh beforehand.
  2. While check of categories.txt and channels.txt is good, add check for channels.txt containing anything within, and also check for lists.txt and create it if it doesn't exist (what format is its content?)
  3. Check for installation of jq via ifinstalled jq || { echo >&2 "'jq is required but it's not installed."; exit 1; }
  4. For faster typing name it ybrowser so it won't conflict with yt and pressing Tab for yt-dlp
  5. Once you download a video and it finishes, it returns you to the same video with the prompts (Watch/Download/AddToList). Watch opens it via remote network mpv which is needless since it's locally downloaded, Download is needless since it's already downloaded. It should instead return you to the videos menu once downloading finishes. Or even better, on pressing downloading, open a new process (and move it to window 9) and detach it from this script. Try this in your terminal $TERMINAL >/dev/null 2>&1 for an example

I will definitely make use of this, will probably make channelrefresh run after boot like you do. This also reminds me of the setup to newsboat (with the urls text file)

Whether this gets merged or not (it's just 2 bin files, it would be a shame not to), I also suggest making it its own (AUR) package, for these benefits:

  1. Automatic dependency installation (so a user just installs "ybrowser" without caring about the dependencies)
  2. Its own config file under .config like newsboat instead of ~/.share/*.txt
  3. Its own man and --help (how to add lists? What should the content of lists be?)

Cheers and thank you :)

@TheYellowArchitect
Copy link
Contributor

Edited/Updated the above text*

@emrakyz
Copy link
Contributor Author

emrakyz commented Nov 19, 2023

I am glad you liked the script :) Your suggestions are pretty valid. They can easily be done. Please feel free to add anything so we can think about it.

I will probably add your suggestions soon.

I still suggest you to use the latest versions though. I added good improvements. You can add the script to boottime and get notified immediately. Since computers can't connect to the internet in a second, the script also waits for the network connection properly now by pinging the 9.9.9.9 domain (Quad9 DNS: A reliable, safe DNS provider).

@emrakyz
Copy link
Contributor Author

emrakyz commented Nov 19, 2023

@TheYellowArchitect

I have added all of your suggestions and made them even better. Do you have anything more to add?

  • The script checks the data file if it's empty, notifies you that you need to run the "channelrefresh" script first.
  • When you run the "channelrefresh" script, it notifies you if you don't have anything on your "channels.txt" file.
  • If you have channels but the formatting of "channels.txt" file is wrong, it notifies you again regarding the formatting.
  • It automatically installs the dependencies if not installed already: yt-dlp, mpv and jq
  • I have also added some comments explaining the formatting of categories and channels.
  • The script waits for the proper network connection so if you add it to boot-time, it doesn't fail because you had no internet.

Thanks for your testing and suggestions. Again, if you have any more suggestions, feel free to mention.

@TheYellowArchitect
Copy link
Contributor

Aside of the nitpick above, all the changes are great and much-needed. I have no more suggestions, you basically implemented them all in a single day! 🚀

@emrakyz
Copy link
Contributor Author

emrakyz commented Nov 19, 2023

@TheYellowArchitect

I share these scripts for users on other distros. Explicit functions and variables help people easily change the features without specifically modifying lots of different places on scripts.

For example below function:

DMENU() {
    rofi -dmenu -i -l $1 -p "$2"
}

You can easily change it to rofi -dmenu instead of using dmenu and there won't be any need to change any part of the code.

It's also same for INSTALLER variable. It can be emerge $pkg or pacman -S --noconfirm $pkg or any other package manager based on the specific user's distro. Some users (including me) can also prefer the simpler and more minimal "doas" instead of sudo. So INSTALLER variable at the top can be distinctive because it's one of the first lines. Otherwise, it can disappear in the code below for the most general users. That's why I added an external variable instead of directly using.

It's better to be spooky and transparent instead of hiding things :) Since the same users can also see that the installer variable is just a pacman command.

@TheYellowArchitect
Copy link
Contributor

TheYellowArchitect commented Nov 19, 2023

I share these scripts for users on other distros

You can easily change it to rofi -dmenu instead of using dmenu and there won't be any need to change any part of the code.

Sure, but this repo (Artix+LARBS) doesn't have rofi by default. You already have the installation script in the same script, just include rofi too, or straight up remove it for this specific pull request.

It's also same for INSTALLER variable. It can be emerge $pkg or pacman -S --noconfirm $pkg or any other package manager based on the specific user's distro. Some users (including me) can also prefer the simpler and more minimal "doas" instead of sudo. So INSTALLER variable at the top can be distinctive because it's one of the first lines. Otherwise, it can disappear in the code below for the most general users. That's why I added an external variable instead of directly using.

Same argument as above. This repo is meant for Artix/Arch, so it's sudo pacman. Anyway, no big deal, I am only making suggestions to make it more likely to be merged, it works great anyway and the code is good.

Regardless, well done. This with newsboat is quite the combo for watching youtube without youtube :P

@emrakyz
Copy link
Contributor Author

emrakyz commented Nov 20, 2023

@TheYellowArchitect

Of course. I already use sudo pacman and dmenu for the PR on this repo but this repo and Luke's channel is popular enough so other people can also see and use the scripts. In fact I share the PR links with people in general so there are also non-larbs users looking at these. So the main aim is to create fast, minimal, portable, extensible and easily modifiable scripts.

It's also a good practice that your scripts are mostly POSIX compliant and easily portable for users on on other systems such as BSD. That's why I also use /bin/sh instead of Bash along with other advantages of Dash.

In my opinion the best thing about the script is that even though it's very minimal (6.8kb) it has features that native YouTube client doesn't have. For example you can't sort by duration on YouTube or you can't put subscribed channels into separate categories and you can't search among your subscriptions. We also do all of these without a browser and an account and SponsorBlock is the icing on the cake :)

This script can be easily be integrated with fzf so it can also fetch thumbnails and do YouTube search with Python integration or even with curl maybe but I think it's beyond its purpose. The idea here is not creating a different YouTube client but an alternative way to change the habits.

FYI sponsorblock.lua config file has a snippet below, make sure to select all categories. Not all of these are activated as a default:

    -- Categories to fetch
    categories = "sponsor,intro,outro,interaction,selfpromo,filler,preview,music_offtopic",

    -- Categories to skip automatically
    skip_categories = "sponsor,intro,outro,interaction,selfpromo,filler,preview,music_offtopic",

@TheYellowArchitect
Copy link
Contributor

FYI sponsorblock.lua config file has a snippet below, make sure to select all categories. Not all of these are activated as a default:

You just keep on giving!
Thank you very much.

@ZaxonXP
Copy link
Contributor

ZaxonXP commented Dec 13, 2023

I have to admit that is nice concept to watch videos using dmenu/rofi and data taken uding yt-dlp. :)

Myself I use RRS reader (newsboat) for channels subscription and some script to get the older videos (then using nvim to manage/play them with mpv).

@emrakyz
Copy link
Contributor Author

emrakyz commented Dec 13, 2023

@ZaxonXP

Yes I also used Newsboat for that before.

The main advantage of this method is that you can categorize channels and easily filter categories, channels, or all videos from all channels. You can search for keywords, sort by views or duration (duration sorting is not even possible on native YouTube). You can create and manage custom playlists with this and all of these are completely modular so you don't actually use the 334 lines of the script but you only interact with the functions you need. So, it's minimal, clean and fast :)

The other advantage is that I get notified with every boot as below when a channel has new videos:

image

@ZaxonXP
Copy link
Contributor

ZaxonXP commented Dec 13, 2023

Usually I stick to the creator channel and for that I find my method better. :) But I am different person, so I have a different needs. ;)

But your script give me better ideas how to retreive the content using yt-dlp and jq commands.

@ZaxonXP
Copy link
Contributor

ZaxonXP commented Dec 13, 2023

I update my scripts and I found strange that I am getting different results between my approach (using youtube-viewer) and yours (using yt-dlp).

Here is how I am getting the list using youtube-viewer:

#!/bin/bash

# script will extract the youtube list using user ID
#
# this is workaround as the youtube-viewer does not support paging
# from the commend line anymore.

[ $# -ne 2 ] && (
    echo -e "Get the Youtube videos from the provided channel ID\n\n";
    echo -e "$0 <channelID> <output_file>\n\n";
    exit 0;
)
[ $# -eq 2 ] && (
    st -g 200x200 -o "$2_tmp" youtube-viewer --no-use-colors --custom-layout=' *TIME* # *ID* # *PUBLISHED* # *TITLE*' -uv=$1 ;
    cat -n "$2_tmp" | grep '#' | sort -uk2 | sort -n | cut -f2- > "$2"
    rm -vi "$2_tmp"
)

yt_playlist_align.pl "$2"

In the yt_playlist_align.pl I replace # sign, align the columns (duration colum) and fix publish dates:

#!/usr/bin/perl
# Align YT playlist file

my (@data, @one, @two, @three, @four, $max);
$max = 0;

my $file = open(IN, "<", $ARGV[0]) or die "Cannot open file: $!\n";

while (<IN>) {
    m/(.+?)#([^#]*)#([^#]*)#(.*)$/;
    my $len = length($1);
    push(@data, $len);
    push(@one, $1);
    push(@two, $2);
    my $date = $3;
    if ($date =~ m/\d{8}/) {
        $date = sprintf("%s-%s-%s", substr($date, 0,4), substr($date,5,2), substr($date,7,2) );
    }
    push(@three, $date);
    push(@four, $4);
    $max = $len if ($len > $max);
}
close(IN);

my $file = open(OUT, ">", $ARGV[0]) or die "Cannot open file for writting: $!\n";

for ( my $i=0; $i<$#data; $i++ ) {
    my $str = " " x ($max - $data[$i]); 
    printf(OUT "%s%s|%s|%s |%s\n", $str, $one[$i], $two[$i], $three[$i], $four[$i]);
}
close(OUT);

So I ran the command like this:
new_get_list.sh UCwpHKudUkP5tNgmMdexB3ow MP1.txt

Then I run another script using your approach:

#!/bin/bash

# script will extract the youtube list using user ID
#
# this is workaround as the youtube-viewer does not support paging
# from the commend line anymore.

[ $# -ne 2 ] && (
    echo -e "Get the Youtube videos from the provided channel ID\n\n";
    echo -e "$0 <channelName> <output_file>\n\n";
    exit 0;
)
[ $# -eq 2 ] && (
    #st -g 200x200 -o "$2_tmp" youtube-viewer --no-use-colors --custom-layout=' *TIME* | *ID* | *PUBLISHED* | *TITLE*' -uv=$1 ;
    channel_url=https://www.youtube.com/@${1}/videos
    yt-dlp -j --flat-playlist --skip-download --extractor-args youtubetab:approximate_date "$channel_url" | jq -r '[.duration_string, .id, .upload_date, .title] | @tsv' | sed 's/\t/ # /g' > "${2}"
    yt_playlist_align.pl "$2"
)

So I run this command like this:
new_get_list2.sh MentourPilot MP2.txt

Then I compared the line count:

wc -l MP*
  640 MP1.txt
  581 MP2.txt

So I wrote small comparison script:

#!/usr/bin/perl

use warnings;
use strict;

use File::Slurp;
use String::Util qw(trim);

my @file1 = read_file("MP1.txt", chomp => 1);
my @file2 = read_file("MP2.txt", chomp => 1);

my (%d1, %d2);

for ( my $i=0; $i < $#file1; $i++ ) {
    $file1[$i] =~ m/[^\|]+\|([^\|]+)\|.*$/;
    my $x = trim($1); 
    $d1{$x} = $i ;
}

for ( my $i=0; $i < $#file2; $i++ ) {
    $file2[$i] =~ m/[^\|]+\|([^\|]+)\|.*$/;
    my $x = trim($1); 
    $d2{$x} = 1;
}

foreach my $k (keys %d1) {
    if (! exists $d2{$k} ) {
        print "F1: $file1[$d1{$k}]\n";
    }
}

foreach my $k (keys %d2) {
    if (! exists $d1{$k} ) {
        print "F2: $file2[$d2{$k}]\n";
    }
}

This showed me the 59 records difference:

F1:     35:15 | 35DtVI71FC0 | 16 Jul 2017  | Sunday Live 16/07/2017
F1:  01:10:51 | SKEuCEKe_w0 | 01 Oct 2020  | Spontaneous Aviation LIVE with Mentour Pilot
F1:     01:00 | NVOsrd_MUDs | 28 Feb 2023  | How do Pilots Use CHECKLISTS?! #shorts
F1:     00:54 | laAhpuEVDrY | 31 Dec 2022  | Bugs in the Aircraft Fuel! #shorts
F1:     00:58 | ULKqQWoJCpE | 15 Dec 2022  | TITANIC of the Skies?! - Air France 447 #shorts
F1:     37:33 | XJwUk5HH4KI | 12 Mar 2019  | MAX Chaos - Live answers
F1:     00:50 | cilFmWrBykg | 30 May 2023  | USA Spying on the USSR?! #shorts
F1:     01:00 | v2vo336ZCMA | 18 May 2023  | Nervous Flyer? Check this out!!  #shorts
F1:     05:54 | tz8bUDDTl_s | 12 Jun 2018  | Can a pilot fly more than one type?
F1:  01:39:10 | CneKup3OAcQ | 20 Sep 2020  | Aviation chat with Mentour Pilot!
F1:     00:53 | PmrzGBcXhG0 | 07 Nov 2022  | B777 TOO LOW on approach! #shorts
F1:  01:37:27 | Nd9fZbI7hnk | 10 Feb 2022  | The ONE MILLION LIVE!
F1:  01:02:53 | 5Kur6BgliSs | 09 Sep 2016  | 3rd Friday night stream with Mentour.
F1:     39:43 | 1pk2nkoU7_k | 27 Jan 2017  | Friday night live!
F1:     36:33 | r5wnUZlbZaw | 06 Dec 2016  | Livestream with Mentour!
F1:     58:12 | -8kxs4KXau0 | 08 Jul 2018  | 💥Cosmic radiation and other Live questions 💥
F1:  01:05:11 | kejYputHShU | 08 Nov 2020  | Come chat with Mentour Pilot
F1:  01:07:06 | KVyTCZyPoXM | 03 Jun 2020  | "Aviation Talk" with Mentour Pilot!
F1:  01:16:46 | UHpb-fJWreE | 18 Mar 2020  | Live-stream with Mentour Pilot! Ask your Aviation questions now!
F1:     44:27 | RMb1Y0Tsf48 | 28 Sep 2016  | Livestream with Mentour!
F1:     40:05 | i7K_wkplJb8 | 04 Jan 2017  | 2 000 000 Views Livestream!!
F1:  01:12:36 | PstqrMXudAI | 27 May 2020  | Aviation talk with Mentour Pilot!
F1:     00:45 | tO1MyNuajiw | 14 Sep 2021  | So, how DID the flying go this week?!
F1:     31:32 | 9xDatboUlO4 | 18 Feb 2017  | Saturday Night Live!
F1:  01:40:30 | 4vwETgqdiD0 | 19 Apr 2020  | LIVE with Mentour & MentourWIFE!
F1:  01:06:43 | NjgCjO1S5-4 | 06 May 2020  | Mentour Pilot speaks to Ben the Dispatcher - LIVE
F1:     57:29 | ofZpNCtnNzQ | 18 May 2020  | Live with Mentour Pilot! Lets hang out and talk aviation!
F1:     57:38 | -8BEtMj6-lo | 15 Jul 2018  | Live chat with Mentour Pilot! Ask your questions now.
F1:  01:08:14 | -9t8JTDAiI0 | 13 May 2020  | LIVE with Mentour and helicopter Pilot Luana!
F1:  01:12:27 | 5-_ptvKOH6I | 15 Mar 2020  | Live-stream with Mentour Pilot! Ask your Aviation questions now!
F1:     06:16 | fT4bh720s_M | 05 Jun 2018  | Live from Paris Beauvais
F1:     00:49 | QK54I2qoRRM | 08 Sep 2021  | About going flying again…
F1:     00:25 | pF0_8J3ceEI | 03 Aug 2021  | A #shorts peak into the Boeing 737 wheel well bay!
F1:  01:35:41 | L85YsebOOdg | 26 Apr 2020  | Mentour Pilot and Sam Chui - LIVE!
F1:     59:01 | IparTd9dKt4 | 20 Jul 2015  | Pilot live feed - CRM and much more!
F1:  01:31:16 | YAD2VCOz92w | 03 May 2020  | Swayne Martin and Mentour Pilot LIVE!
F1:  01:32:07 | WG3JnDLkiQY | 02 Apr 2020  | Aviation Live-stream with Mentour Pilot
F1:     33:22 | 6wgaL16H9gg | 14 Jan 2017  | Spontaneous saturday evening live!
F1:     09:05 | GG5ah39GZl0 | 25 Jul 2018  | Slots, what are they? Stuck in Memmingen
F1:     00:48 | Z7_6GX2VUn4 | 23 Jan 2023  | Airplane Out of CONTROL! #shorts
F1:  01:25:16 | zDNh4G-_j8A | 06 Sep 2020  | Aviation Talk with Mentour!
F1:     55:16 | LVoOGM-SL6Q | 17 Jan 2021  | Aviation talk with Mentour Pilot
F1:     09:05 | lybBJhq3dv0 | 23 Jul 2018  | Technical delay and slots
F1:  01:26:05 | WeULPiZEgNI | 29 Mar 2020  | Live-stream with Mentour Pilot! Ask your Aviation questions now!
F1:  01:30:11 | Fvfc6LLj_ww | 24 May 2020  | Mentour Pilot and Juan from Blancolirio about current events!
F1:  01:10:46 | vc4Ut28qdyE | 09 Apr 2020  | Aviation Live-stream with Mentour Pilot and Blancolirio!
F1:  01:11:11 | SVNhVVypmSo | 10 Jun 2020  | Aviation Talk with Mentour pilot!
F1:  01:41:59 | IVJX8IaGsEM | 12 Apr 2020  | LIVE with Mentour, FlightChops and Mikey from Plane-Savers!
F1:  01:27:50 | JHsuPr_tjS4 | 10 May 2020  | Come ask your Questions to Mentour! Good old fashion LIVE
F1:     34:35 | 1sZfeFJ9n0I | 28 Nov 2018  | LionAir B737 MAX Interim report released, What does it mean?
F1:  01:25:41 | -SJ8AzvQWFs | 14 Jun 2020  | Aviation talk with Pilot Bambi and Mentour Pilot
F1:  01:17:12 | YHva90aEodg | 22 Mar 2020  | Live-stream with Mentour Pilot! Ask your Aviation questions now!
F1:  01:40:25 | _NKOUEKsfPw | 15 Apr 2020  | Live with Mentour Pilot and DutchPilotgirl!
F1:  01:21:19 | YVYJz-lssdU | 22 Apr 2020  | LIVE with Mentour Pilot and PilotMaria!
F1:  01:25:31 | Y5Z2m0lr0No | 09 Aug 2020  | Aviation talk with Mentour pilot!
F1:  01:18:37 | JMKrqj_wHdI | 16 Dec 2020  | Wednesday LIVE with Mentour Pilot
F1:     32:18 | 3M51jlzWEB0 | 20 Nov 2019  | Mentour Pilot FUNDING update
F1:  01:04:05 | 77uioWlOXEc | 11 Mar 2020  | Live-stream with Mentour Pilot! Ask your Aviation questions now!
F1:  01:44:28 | fbMzPjpGIPo | 29 Apr 2020  | C.W Lemoine and Mentour Pilot - LIVE!!

I wonder why geting list of videos using the channel name vs using channel ID gives such difference?
Do you know how to get list of videos using yt-dlp and the channel ID?

Channel ID I got using youtube-viewer and :i command on the video (like :i1 for video 1 form the result list).

Kind regards,
Piotr

@ZaxonXP
Copy link
Contributor

ZaxonXP commented Dec 13, 2023

OK. I found that if you use the following url to get the list of videos, then you will get a complete list:
channel_url=https://www.youtube.com/channel/${1}
where ${1} is the channel ID.

I am not sure how you can get channel ID easily using yt-dlp.

@ZaxonXP
Copy link
Contributor

ZaxonXP commented Dec 13, 2023

OK. Found it. You can use .playlist_id in the jq to get the channel ID. With that you can get the complete list of videos from the channel.

@emrakyz
Copy link
Contributor Author

emrakyz commented Dec 14, 2023

@ZaxonXP

For these types of things you can check the yt-dlp community. Some of the data is not default. For example, in order to get video dates, you need to use this argument with yt-dlp: yt-dlp --extractor-args youtubetab:approximate_date.

You can try the yt-dlp command individually to see the output then try piping it into jq.

To be honest I have tried to remove jq too and use native text processing tools such as sed but it's nearly impossible because the structure of the website can change but jq as a native json processor can properly extract the related data in a faster and safer way even when the structure changes.

You can extract any json data you want with jq.

jq -r '[.title, .url, .view_count, .duration, .upload_date] | @tsv'

For example the above command is pretty intuitive and gives us some necessary data then it converts it into a tab separated value so we can use cut to use those. We can cut the first field to see the titles, second; to open the video, and others to sort the videos properly.

This message is a little bit late so I couldn't help with your actual question but I am glad that you found this inspirational :) I can still try helping if you come with anything new though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants