01_Data-Collection_limited_Ukrainian.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fc691c3b-7355-4fd1-9344-2524e1bb88f1",
   "metadata": {},
   "source": [
    "# Data Collection - Ukrainian inflow\n",
    "**Author**: Andrea Cass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de32fe9a-2b45-48d5-abf3-0e505adbd3a0",
   "metadata": {},
   "source": [
    "## 1. About this notebook\n",
    "\n",
    "The purpose of this Jupyter notebook is to collect immigration-related tweets in English and German published publicly in Germany during the Ukrainian inflow between June 24, 2021 and October 24, 2022\n",
    "\n",
    "Tweets are collected via the Twitter API using my Academic research developer account and Twarc2. To run the code successfuly, you must also have an Academic research developer account with Twitter and provide your details under the \"Authentification\" section. You can apply for an account here: https://developer.twitter.com/en/products/twitter-api/academic-research\n",
    "\n",
    "English and German tweets will be collected separately and saved to separate csv files, titled:\n",
    "> *01a_Data-Collection_limited_Ukrainian_eng.csv*\n",
    "\n",
    "> *01b_Data-Collection_limited_Ukrainian_de.csv*\n",
    "\n",
    "The two files will be merged in the subsequent Jupyter Notebook, titled, \"02_Pre-processing\"\n",
    "\n",
    "**NOTE**: Do NOT merely run all cells without reading instructions from each section. At times, it may be required of you to alter the code. If that is the case, you will be instructed on how to do so."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "785bf389-8f6f-48ee-92b6-3c44c85eea4a",
   "metadata": {},
   "source": [
    "## 2. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c377dcae-1db6-4bbb-8117-22b3d3f02176",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from datetime import date, datetime, timezone\n",
    "import asyncio\n",
    "import json\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from twarc import Twarc2, expansions\n",
    "import tweepy\n",
    "import csv\n",
    "import os\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b190698-c7f2-4c38-91c1-15ddaa836e04",
   "metadata": {},
   "source": [
    "## 3. Working directory & file paths\n",
    "\n",
    "In the Notebook for the data collection of tweets during the Syrian inflow, the working directory was established, and a new folded, \"CASS_thesis\" was created within the working directory. The code below is used to, again, set up your working directory in addition to creating two objects called:\n",
    "* **cwd**: the current working directory\n",
    "* **CASS_thesis**: the folder where all data from my Notebooks will be saved"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48bd95bc-480b-46be-9304-b87315e63485",
   "metadata": {},
   "source": [
    "### 3.1. Current working directory\n",
    "Use the code below to find out what your current working directory is set to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f630782d-ead1-4e81-8c1d-8a41536fc14e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/andycass/Jupyterlab_main-folder/THESIS'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find current working directory\n",
    "\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4616dcc-1800-41af-acea-df4d5b5e2278",
   "metadata": {},
   "source": [
    "If your current working directory is not your desired directory, follow the subsequent steps to change the working diectory by:\n",
    "1. deciding where you would like your working directory to be (e.g., your Desktop)\n",
    "2. entering the file path of your desired working directory into the code below\n",
    "\n",
    "**NOTE**: If you are satisfied with your working directory and do NOT wish to change it, skip the block of code underneath **3.1.1. Changing current working directory** and, instead, proceed from the block of code underneath **3.1.2. Naming current working directory**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f97d7dd3-7237-4ba6-8a54-0907d1ed91e9",
   "metadata": {},
   "source": [
    "#### 3.1.1. Changing current working directory\n",
    "**NOTE**: The code below contains the path to **my** desired working directory to serve as an example. You must alter it to the path of **your** desired working directory. Keep in mind that my example is formatted according to Macbook standards, and Windows formatting differs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2c1a1c2d-1e6e-44d7-aeae-018911c80ac8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# changing current working directory\n",
    "\n",
    "os.chdir('/Users/andycass/Desktop/Thesis_data-and-code')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43c123a5-c92b-4436-b3a0-46fecf9e238c",
   "metadata": {},
   "source": [
    "#### 3.1.2. Naming current working directory\n",
    "Now that your current working directory is established, use the code below to name it \"cwd\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7de6d519-4454-416f-b970-dd3a7d4e583e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# naming the current working directory\n",
    "\n",
    "cwd = Path.cwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "59abc3c7-d4c0-4fd9-9f40-7c7d5f94459a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/Users/andycass/Desktop/Thesis_data-and-code')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# double-checking the current working directory location\n",
    "\n",
    "cwd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c6f13a7-13c5-430a-b8f6-efbe0c231d6a",
   "metadata": {},
   "source": [
    "### 3.2. CASS_thesis\n",
    "Now that the current directory has been named, the code below will allow you to name \"CASS_thesis\" (which was already created in the Notebook titled \"01_Data-Collection_Syrian)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "452dccb3-84f3-4fe8-b110-d04890181c2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# naming the CASS_thesis folder\n",
    "\n",
    "CASS_thesis = cwd / 'CASS_thesis'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a9a8e611-0e6e-45e3-8f2d-e3928486f229",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/Users/andycass/Desktop/Thesis_data-and-code/CASS_thesis')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# double-checking the CASS_thesis location\n",
    "\n",
    "CASS_thesis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "850ae1c4-ede4-49ea-b78e-c9a65970b362",
   "metadata": {},
   "source": [
    "## 4. Authentification\n",
    "**NOTE**: You must provide your unique bearer token from your Academic research account in the quotation marks of the following code. To proceed, insert your bearer token. The code will NOT run otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e0eecefd-e1e6-498c-8fc5-eb6e0d504ea1",
   "metadata": {},
   "outputs": [],
   "source": [
    "bearer_token = \"\" # your bearer token\n",
    "twarc_client = Twarc2(bearer_token=bearer_token)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4b93f7e-44d4-4fc5-b444-48f453e97681",
   "metadata": {},
   "source": [
    "## 5. Search endpoint\n",
    "### 5.1. English tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "dfeeb2c4-b847-4527-b29f-1e81fa178bfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# empty list to store the search results\n",
    "\n",
    "tweets = [] "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0fad500-bb52-43ee-a51d-ce6002b710ae",
   "metadata": {},
   "source": [
    "#### 5.1.1. Defining the query\n",
    "The query will contain:\n",
    "* keywords\n",
    "* country\n",
    "* geo-location\n",
    "* language\n",
    "\n",
    "The keywords include:\n",
    "* refugee\n",
    "* refugees\n",
    "* asylum seeker\n",
    "* ukrainian\n",
    "* ukrainians\n",
    "\n",
    "**NOTE**: There is no case sensitivity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f61ed181-f02c-43d0-a0b0-5a57051e87d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining the query\n",
    "\n",
    "query = '(\"asylum seeker\" OR \"refugee\" OR \"refugees\" OR \"ukrainian\" OR \"ukrainians\") place_country:DE has:geo lang:en'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b4190d57-00d3-4dd4-8679-c2d4472787bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# start and end time of the query\n",
    "\n",
    "#start: June 24, 2021\n",
    "# end: October 24, 2022\n",
    "\n",
    "start = datetime(2021, 6, 24, 0, 0, 0, 0, tzinfo=timezone.utc)\n",
    "end = datetime(2022, 10, 24, 0, 0, 0, 0, tzinfo=timezone.utc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e85bdf6-4efe-457a-8a07-4d9a4f952101",
   "metadata": {},
   "source": [
    "#### 5.1.2. File name\n",
    "All tweets collected will be saved on a file called:\n",
    "> *01a_Data-Collection_limited_Ukrainian_eng.csv*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "01688356-2ea0-4837-aea0-c0291138a554",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = CASS_thesis / \"01a_Data-Collection_limited_Ukrainian-eng.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57b8a241-b657-4bb8-970d-cc257992295f",
   "metadata": {},
   "source": [
    "#### 5.1.3. Collection & saving\n",
    "The following code will collect all tweets that match the given query and start and end times, save them as a dataframe, and save the dataframe as a csv file with the file name previously provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b7245d2b-4b90-4b20-9a0f-479b319ff88d",
   "metadata": {},
   "outputs": [],
   "source": [
    "for tweet in twarc_client.search_all(\n",
    "    query,\n",
    "    start_time=start,\n",
    "    end_time=end):\n",
    "    tweets.append(tweet)\n",
    "    \n",
    "df = pd.json_normalize(tweets, record_path=['data']) \n",
    "\n",
    "df.to_csv(filename, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83641405-7272-49ab-ad0d-c1f15d7d6e10",
   "metadata": {},
   "source": [
    "### 5.2. German tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6a218a0f-89e9-4989-889c-94c1f78c8a29",
   "metadata": {},
   "outputs": [],
   "source": [
    "# empty list to store the search results\n",
    "\n",
    "tweets = [] "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b74fcdc-36e9-42aa-9e79-014408991952",
   "metadata": {},
   "source": [
    "#### 5.2.1. Defining the query\n",
    "The query will contain:\n",
    "* keywords\n",
    "* country\n",
    "* geo-location\n",
    "* language\n",
    "\n",
    "The keywords include:\n",
    "* flüchtling\n",
    "* fluchtling\n",
    "* fluechtling\n",
    "* flüchtlinge\n",
    "* fluchtlinge\n",
    "* fluechtlinge\n",
    "* asylbewerber\n",
    "* asylbewerberin\n",
    "* asylbewerberinnen\n",
    "* asylsuchende\n",
    "* asylsuchenden\n",
    "* asylant\n",
    "* asylantin\n",
    "* asylanten\n",
    "* asylantinnen\n",
    "* syrer\n",
    "* syrerin\n",
    "* syrier\n",
    "* ukrainer\n",
    "* ukrainerin\n",
    "* ukrainerinnen\n",
    "\n",
    "**NOTE**: The list of German keywords is longer than the list of English keywords due to the gendered nature of the language and to take into account potential missing--or alternative spelling of--umlauts (e.g., \"u\" instead of \"ü\" or \"ue\" instead of \"ü\")\n",
    "\n",
    "**NOTE**: There is no case sensitivity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2129c10e-ca5e-4040-9e6c-0866bcd3b12a",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = '(\"flüchtling\" OR \"fluchtling\" OR \"fluechtling\" OR \"flüchtlinge\" OR \"fluchtlinge\" OR \"fluechtlinge\" OR \"asylbewerber\" OR \"asylbewerberin\" OR \"asylbewerberinnen\" OR \"asylsuchende\" OR \"asylsuchenden\" OR \"asylant\" OR \"asylantin\" OR \"asylanten\" OR \"asylantinnen\" OR \"syrer\" OR \"syrerin\" OR \"syrier\" OR \"ukrainer\" OR \"ukrainerin\" OR \"ukrainerinnen\") place_country:DE has:geo lang:de'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f3982bc1-b4cf-4fad-91f3-971e07cb7ffb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# start and end time of the query\n",
    "\n",
    "#start: June 24, 2021\n",
    "# end: October 24, 2022\n",
    "\n",
    "start = datetime(2021, 6, 24, 0, 0, 0, 0, tzinfo=timezone.utc)\n",
    "end = datetime(2022, 10, 24, 0, 0, 0, 0, tzinfo=timezone.utc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20038648-27d7-461d-93ca-b694c2148162",
   "metadata": {},
   "source": [
    "#### 5.2.2. File name\n",
    "All tweets collected will be saved on a file called:\n",
    "> *01b_Data-Collection_limited_Ukrainian_de.csv*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5997c576-4afe-46e4-9f60-0f8debf8ff71",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = CASS_thesis / \"01b_Data-Collection_limited_Ukrainian-de.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9915f139-fd57-4627-95d8-d31a3f35ba63",
   "metadata": {},
   "source": [
    "#### 5.2.3. Collection & saving\n",
    "The following code will collect all tweets that match the given query and start and end times, save them as a dataframe, and save the dataframe as a csv file with the file name previously provided."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "998204e3-65f2-4eb4-8b71-14abae33defa",
   "metadata": {},
   "outputs": [],
   "source": [
    "for tweet in twarc_client.search_all(\n",
    "    query,\n",
    "    start_time=start,\n",
    "    end_time=end):\n",
    "    tweets.append(tweet)\n",
    "    \n",
    "df = pd.json_normalize(tweets, record_path=['data']) \n",
    "\n",
    "df.to_csv(filename, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "694f8a37-cdcf-4e45-95fc-67d85071e6a3",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}