Python script to scrape trending projects on Git

Python/Git_Trending_Repositories/Readme.md

@@ -0,0 +1,40 @@
+### Scrape Git Trending respositories
+
+This script is scrapes details about Git trending repositories displayed here https://github.com/trending.
+
+The data will be exported in a .csv file.
+
+### Setup
+
+1. Create a Virtual Environment.
+2. Install the requirements by using `pip3 install -r requiremnts.txt`
+3. Hurray.! You're ready to use the script.
+
+### Running a file
+
+`python git-trending-repository-scraper.py`
+
+### Running this script as a cron job
+
+The syntax is:
+
+`mm hh * * * <location where python3 is installed> <location of the python script>`
+
+This will execute the cron job everyday at a particular hour.
+
+`0 12 * * * /usr/bin/python3 /home/user/Rotten-Scripts/Python/Git_Trending_Repositories/git-trending-repository-scraper.py`
+
+Adding the above command in the `crontab` will run the script at 12:00 am every day.
+
+`.csv` file will be generated in the directory where the script file is.
+
+The only downside of the cron job is, to install all the requirements globally.
+
+
+## Output
+
+![Output Pic](https://i.imgur.com/oJyFuwP.png)
+
+## Author(s)
+
+Hi I'm [Jagruti Tiwari](https://github.com/Jagrutiti/) author of this script.

Python/Git_Trending_Repositories/git-trending-repository-scraper.py

@@ -0,0 +1,35 @@
+import requests
+from bs4 import BeautifulSoup
+import csv
+
+page = requests.get('https://github.com/trending')
+
+soup = BeautifulSoup(page.text, 'html.parser')
+
+# get the repo list
+repo = soup.find(class_="position-relative container-lg p-responsive pt-6")
+
+repo_list = repo.find_all(class_='Box-row')
+
+# Open writer with name
+file_name = "github_trending_today.csv"
+# set newline to be '' so that that new rows are appended without skipping any
+f = csv.writer(open(file_name, 'w', newline=''))
+
+f.writerow(['Developer', 'Repo Name', 'Programming Language', 'Total Stars', 'Number of Forks'])
+
+for repo in repo_list:
+ h1 = repo.find(class_='h3 lh-condensed')
+ developer_name = h1.find('span').text.strip().strip(" /")
+ repo_name = h1.find('span').next_sibling
+ stars = repo.find(class_='octicon octicon-star').next_sibling.text 
+ programming_language = repo.find('span', itemprop='programmingLanguage')
+
+ if programming_language:
+ programming_language=programming_language.text.strip()
+ else:
+ programming_language=''
+
+ no_of_forks = repo.find(class_='octicon octicon-repo-forked').next_sibling.text
+
+ f.writerow([developer_name, repo_name, programming_language, stars, no_of_forks])

Python/Git_Trending_Repositories/requirements.txt

@@ -0,0 +1,8 @@
+beautifulsoup4==4.10.0
+bs4==0.0.1
+certifi==2021.10.8
+charset-normalizer==2.0.12
+idna==3.3
+requests==2.27.1
+soupsieve==2.3.1
+urllib3==1.26.8