demo/get_dataset_from_zenodo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "abceb1d0",
   "metadata": {},
   "source": [
    "# Retrouver dans  Zenodo les jeux de données de son établissement "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bd649c92",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests, json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "47984414",
   "metadata": {},
   "outputs": [],
   "source": [
    "# a gist example : https://gist.github.com/slint/d30fc0f415300876facbbeb8a0989ab2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bcc7fda",
   "metadata": {},
   "source": [
    "l'API de Zenodo nécessite une clé. cf. https://developers.zenodo.org/#quickstart-upload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9903627c",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"../hide/personnal-keys.json\") as f : \n",
    "    ACCESS_TOKEN = json.load(f)[\"ZENODO_KEY\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02a2f559",
   "metadata": {},
   "source": [
    "## Défnir la recherche sur le champs affiliation\n",
    "\n",
    "cf. https://help.zenodo.org/guides/search/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5392ad48",
   "metadata": {},
   "outputs": [],
   "source": [
    "aff_query = \"(univ* AND (diderot OR descartes))\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2696b58b",
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get(\"https://zenodo.org/api/records\",\n",
    "                 params = {\"q\" : \"creators.affiliation:\" +  aff_query,\n",
    "                           \"type\" : \"dataset\",\n",
    "                           \"size\" : 300,\n",
    "                           \"sort\" : \"mostrecent\",\n",
    "                           \"all_version\" : False,\n",
    "                           \"access_tpoken\" : ACCESS_TOKEN\n",
    "                          })\n",
    "res = r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8593c3a6",
   "metadata": {},
   "source": [
    "Se familiariser avec la structure de la réponse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "068dba1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# export les résultats en JSON dans un fichier local \"zenodo_raw.json\"\n",
    "# ouvrir ce fichier en local et comprends sa structure\n",
    "with open(\"zenodo_raw.json\", 'w') as f : \n",
    "    json.dump(res, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6904154d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imprimer le premier résultat de la requête\n",
    "print(json.dumps(res[\"hits\"][\"hits\"][1],  indent = 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "644f82a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "72\n"
     ]
    }
   ],
   "source": [
    "# nb de jeux de données trouvés\n",
    "print(res[\"hits\"][\"total\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "649a8eb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imprimer le 10e résultat de la requête\n",
    "print(json.dumps(res[\"hits\"][\"hits\"][10], indent = 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "10d06841",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['conceptdoi', 'conceptrecid', 'created', 'doi', 'files', 'id', 'links', 'metadata', 'owners', 'revision', 'stats', 'updated'])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# imprimer les clés d'un hits\n",
    "first_hit = res[\"hits\"][\"hits\"][0]\n",
    "first_hit.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6f17819",
   "metadata": {},
   "source": [
    "## Déduire les informations à récupérer\n",
    "``` \n",
    "id, l'identifiant zenodo, pour déduire le lien https://zenodo.org/record/5005868\n",
    "metadata/publication_date : date du dépot\n",
    "metadata/titre : titre du jeu de de donnée\n",
    "metadata/license : la licence apposée\n",
    "metadata/keywords\n",
    "metadata/doi\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88dce8e2",
   "metadata": {},
   "source": [
    "## Récupérer les données"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3a274ed7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# un dictionnaire pour récupérer les données\n",
    "data = {}\n",
    "\n",
    "# itérer sur les résultats et placer les données souhaitées dans le dictionnaire data\n",
    "for idx, item in enumerate( res[\"hits\"][\"hits\"] ) : \n",
    "    \n",
    "    data[idx] = {}\n",
    "    data[idx][\"id\"] = item[\"id\"]\n",
    "    \n",
    "    data[idx][\"title\"] = item[\"metadata\"].get(\"title\")\n",
    "    data[idx][\"publication_date\"] = item[\"metadata\"].get(\"publication_date\")\n",
    "    data[idx][\"doi\"] = item[\"metadata\"].get(\"doi\")\n",
    "    \n",
    "    #licence\n",
    "    if item[\"metadata\"].get(\"license\") : \n",
    "        data[idx][\"license\"] = item[\"metadata\"][\"license\"][\"id\"]   \n",
    "    \n",
    "    # mots-clés\n",
    "    if item[\"metadata\"].get(\"keywords\") : \n",
    "        data[idx][\"keywords\"] = \",\".join(item[\"metadata\"][\"keywords\"])  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a46071a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      " \"id\": 5005868,\n",
      " \"title\": \"A global inventory of solar photovoltaic generating units - dataset\",\n",
      " \"publication_date\": \"2021-10-27\",\n",
      " \"doi\": \"10.5281/zenodo.5005868\",\n",
      " \"license\": \"CC-BY-4.0\",\n",
      " \"keywords\": \"photovoltaic solar,remote sensing,geospatial data,computer vision\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "## imprimer les données récupérer d'un jeu de donnée\n",
    "print( json.dumps(data[3], indent = 1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3c4d96a",
   "metadata": {},
   "source": [
    "## Placer les données dans un dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "57c730ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "9f7e8432",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame.from_dict(data, orient='index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "cd315212",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>publication_date</th>\n",
       "      <th>doi</th>\n",
       "      <th>license</th>\n",
       "      <th>keywords</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>5003345</td>\n",
       "      <td>Data from: State transitions: a major mortalit...</td>\n",
       "      <td>2018-05-23</td>\n",
       "      <td>10.5061/dryad.km2s0</td>\n",
       "      <td>CC0-1.0</td>\n",
       "      <td>primate,Microcebus murinus,Biological rhythm,P...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>4999955</td>\n",
       "      <td>Data from: Molecular determinants of agonist s...</td>\n",
       "      <td>2015-08-26</td>\n",
       "      <td>10.5061/dryad.k47p7</td>\n",
       "      <td>CC0-1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                              title  \\\n",
       "13  5003345  Data from: State transitions: a major mortalit...   \n",
       "16  4999955  Data from: Molecular determinants of agonist s...   \n",
       "\n",
       "   publication_date                  doi  license  \\\n",
       "13       2018-05-23  10.5061/dryad.km2s0  CC0-1.0   \n",
       "16       2015-08-26  10.5061/dryad.k47p7  CC0-1.0   \n",
       "\n",
       "                                             keywords  \n",
       "13  primate,Microcebus murinus,Biological rhythm,P...  \n",
       "16                                                NaN  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sample(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "151478ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# trier la table par date de dépôt\n",
    "df = df.sort_values( by = \"publication_date\", ascending = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "4dd023a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>publication_date</th>\n",
       "      <th>doi</th>\n",
       "      <th>license</th>\n",
       "      <th>keywords</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7104630</td>\n",
       "      <td>Experimental data of \"Mesoscopic Klein-Schwing...</td>\n",
       "      <td>2022-09-22</td>\n",
       "      <td>10.5281/zenodo.7104630</td>\n",
       "      <td>CC-BY-4.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6808401</td>\n",
       "      <td>Data and code for \"A decade of monitoring micr...</td>\n",
       "      <td>2022-08-09</td>\n",
       "      <td>10.5281/zenodo.6808401</td>\n",
       "      <td>GPL-3.0</td>\n",
       "      <td>micropollutants,wet-weather flows,stormwater,m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6760262</td>\n",
       "      <td>Molecular and electrophysiological features of...</td>\n",
       "      <td>2022-06-27</td>\n",
       "      <td>10.5061/dryad.37pvmcvp1</td>\n",
       "      <td>CC0-1.0</td>\n",
       "      <td>classification,hippocampus,Interneurons,Dentat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5005868</td>\n",
       "      <td>A global inventory of solar photovoltaic gener...</td>\n",
       "      <td>2021-10-27</td>\n",
       "      <td>10.5281/zenodo.5005868</td>\n",
       "      <td>CC-BY-4.0</td>\n",
       "      <td>photovoltaic solar,remote sensing,geospatial d...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id                                              title  \\\n",
       "0  7104630  Experimental data of \"Mesoscopic Klein-Schwing...   \n",
       "1  6808401  Data and code for \"A decade of monitoring micr...   \n",
       "2  6760262  Molecular and electrophysiological features of...   \n",
       "3  5005868  A global inventory of solar photovoltaic gener...   \n",
       "\n",
       "  publication_date                      doi    license  \\\n",
       "0       2022-09-22   10.5281/zenodo.7104630  CC-BY-4.0   \n",
       "1       2022-08-09   10.5281/zenodo.6808401    GPL-3.0   \n",
       "2       2022-06-27  10.5061/dryad.37pvmcvp1    CC0-1.0   \n",
       "3       2021-10-27   10.5281/zenodo.5005868  CC-BY-4.0   \n",
       "\n",
       "                                            keywords  \n",
       "0                                                NaN  \n",
       "1  micropollutants,wet-weather flows,stormwater,m...  \n",
       "2  classification,hippocampus,Interneurons,Dentat...  \n",
       "3  photovoltaic solar,remote sensing,geospatial d...  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5296e17",
   "metadata": {},
   "outputs": [],
   "source": [
    "## exporter la table en CSV\n",
    "df.to_csv(\"data_from_zenodo.csv\", index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cc7e206",
   "metadata": {},
   "source": [
    "----\n",
    "## Vigilance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a67307eb",
   "metadata": {},
   "source": [
    "* la recherche dans l'affiliation se fait pour tous les auteurs (comme le Wos) donc du bruit peut être présent en sortie\n",
    "\n",
    "* l'API donne les résultats avec un systeme de pagination. Vérifier à la fin de la réponse si `links` ne contient pas une url `next`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df37b0eb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}