anime_recap.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df8dec17-bddc-409a-8610-c0ab1a57ed15",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "from openai import OpenAI\n",
    "from elevenlabs.client import AsyncElevenLabs\n",
    "import asyncio\n",
    "import json\n",
    "import os\n",
    "import argparse\n",
    "import concurrent.futures\n",
    "\n",
    "from manga_extraction import extract_all_pages_as_images, save_important_pages, split_volume_into_parts, save_all_pages, extract_panels, scale_base64_image\n",
    "from vision_analysis import analyze_images_with_gpt4_vision, detect_important_pages, get_important_panels, VISION_PRICE_PER_TOKEN \n",
    "from prompts import DRAMATIC_PROMPT, BASIC_PROMPT, BASIC_PROMPT_WITH_CONTEXT,  BASIC_INSTRUCTIONS, KEY_PAGE_IDENTIFICATION_INSTRUCTIONS, KEY_PANEL_IDENTIFICATION_PROMPT, KEY_PANEL_IDENTIFICATION_INSTRUCTIONS\n",
    "from citation_processing import extract_text_and_citations, extract_script\n",
    "from movie_director import make_movie\n",
    "load_dotenv()  # Load environment variables from .env file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cca43991-2e27-4c95-9a6f-f2974e0b6662",
   "metadata": {},
   "outputs": [],
   "source": [
    "volume_number = 10\n",
    "manga = \"naruto\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e1007a3",
   "metadata": {},
   "source": [
    "The goal of this block is to extract a small-scale (fit within a 256px x 256px bounding box) array of pngs corresponding to the pages in the volume. \n",
    "Additionally, an unscaled array of images is extracted as well for full-res images. \n",
    "\n",
    "Later on, the small-scaled images will be sent to GPT-Vision, and the full scaled images will be used for panel extraction to get beautiful high-res panels for the video. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12bbe703-0967-473f-b58c-663e05afabd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize OpenAI client with API key\n",
    "client = OpenAI()\n",
    "# get elevenlabs api key from dotenv\n",
    "narration_client = AsyncElevenLabs(api_key=os.getenv(\"ELEVENLABS_API_KEY\"))\n",
    "\n",
    "print(\"Extracting all pages from the volume...\")\n",
    "volume_scaled_and_unscaled = extract_all_pages_as_images(f\"{manga}/v{volume_number}/v{volume_number}.pdf\")\n",
    "volume = volume_scaled_and_unscaled[\"scaled\"]\n",
    "volume_unscaled = volume_scaled_and_unscaled[\"full\"]\n",
    "print(\"Total pages in volume:\", len(volume))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e3cd6da",
   "metadata": {},
   "source": [
    "The goal of this block is to use the `profile-reference.pdf` and `chapter-reference.pdf` provided by the user as examples in order to identify so-called `important pages`:\n",
    "1) Profile pages: These are pages that contain information about the characters within the volume. Usually most mangas have this towards the beginning of the volume. They show a number of characters, what they look like, their names, and occasionally a small description of the character. This is super useful for GPT-Vision to identify and discern the characters in following steps.\n",
    "2) Chapter pages: Pretty self explanitory, these are the pages that contain the chapter number and title. Mangas typically get creative with how chapter pages look like, some have a full page spread, some have a small box in the corner, some have a full page spread with a small box in the corner. So it's helpful to have a chapter-reference.pdf to use as an example to know what to look for for this specific manga.\n",
    "\n",
    "In a nutshell, this block will process every single page of the volume through GPT-Vision (with parallelized calls to maximize speed) and ask it to tell us if it thinks a page is a profile page or a chapter page. The final results are gathered at the end and extracted as a list of ints in `profile_pages` and `chapter_pages` where each int represents the page index in the volume."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c35d189-d3d8-445e-ad4d-dc94060b4ea9",
   "metadata": {},
   "outputs": [],
   "source": [
    "profile_reference = extract_all_pages_as_images(f\"{manga}/profile-reference.pdf\")[\"scaled\"]\n",
    "chapter_reference = extract_all_pages_as_images(f\"{manga}/chapter-reference.pdf\")[\"scaled\"]\n",
    "\n",
    "profile_pages = []\n",
    "chapter_pages = [] \n",
    "\n",
    "important_page_tokens = 0\n",
    "\n",
    "batch_size = 20\n",
    "\n",
    "print(\"Identifying important pages in the volume...\")\n",
    "# Function to wrap the detect_important_pages call\n",
    "def process_batch(start_idx, pages):\n",
    "    response = detect_important_pages(profile_reference, chapter_reference, pages, client,\n",
    "        KEY_PAGE_IDENTIFICATION_INSTRUCTIONS, KEY_PAGE_IDENTIFICATION_INSTRUCTIONS)\n",
    "    return start_idx, response\n",
    "\n",
    "# Using ThreadPoolExecutor to parallelize API calls\n",
    "with concurrent.futures.ThreadPoolExecutor() as executor:\n",
    "    futures = []\n",
    "    for i in range(0, len(volume), batch_size):\n",
    "        pages = volume[i:i+batch_size]\n",
    "        futures.append(executor.submit(process_batch, i, pages))\n",
    "\n",
    "    for future in concurrent.futures.as_completed(futures):\n",
    "        start_idx, response = future.result()\n",
    "        end_index = start_idx + batch_size - 1\n",
    "        print(f\"Processing pages {start_idx} to {min(end_index, len(volume)-1)}\")\n",
    "        \n",
    "        ip = response[\"parsed_response\"]\n",
    "        print(json.dumps(ip, indent=2))\n",
    "        for page in ip:\n",
    "            if page[\"type\"] == \"profile\":\n",
    "                profile_pages.append(page[\"image_index\"] + start_idx)\n",
    "            elif page[\"type\"] == \"chapter\":\n",
    "                chapter_pages.append(page[\"image_index\"] + start_idx)\n",
    "\n",
    "        important_page_tokens += response[\"total_tokens\"]\n",
    "\n",
    "profile_pages.sort()\n",
    "chapter_pages.sort()\n",
    "\n",
    "print(\"Total tokens to extract profiles and chapters:\", important_page_tokens)\n",
    "print(\"\\n__________\\n\")\n",
    "print(\"Profile pages:\", profile_pages)\n",
    "print(\"Chapter pages:\", chapter_pages)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14fb2ffb",
   "metadata": {},
   "source": [
    "This step is totally optional, but helps developers see what the code did while extracting the `important pages` and saves the GPT-vision identified chapter pages and profile pages into corresponding folders for QA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "780436d6-8bdc-4b04-a11d-348eb9638225",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"{len(volume)}\")\n",
    "print(\"\\n__________\\n\")\n",
    "print(\"Saving important pages to disk for QA...\")\n",
    "save_important_pages(volume, profile_pages, chapter_pages, manga, volume_number)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21eaed41",
   "metadata": {},
   "source": [
    "Now that we have the chapter start pages and the profile pages, the goal of this block is to begin the summarization or \"script-writing\" process. The goal is to have a comprehensive summary of everything that happened in the volume, and importantly, to have **PAGE CITATIONS** sprinkled throughout the summary. This is important because we want to be able to reference the relevant pages and panels later on, and it's important to know where the information in the summary came from.\n",
    "\n",
    "But we can't summarize a whole volume in one GPT-vision prompt (unless the manga volume is tiny), so we split the volume into reasonable-sized chunks for GPT-Vision to process. We also don't want to split the volume in the middle of a chapter, so we use the chapter start pages to split in an intelligent way. \n",
    "\n",
    "The `NUMBER_OF_JOBS` variable determines how many segments the volume will be split into; and the `split_volume_into_parts()` function will do its best to split the volume into `NUMBER_OF_JOBS` parts while respecting chapter boundaries. The result is `jobs` which is a messy object that contains an array of `scaled_images` arrays and an array of `unscaled_images` arrays, where each array is one segment of the volume. \n",
    "\n",
    "The first job (or segment) is sent to GPT-Vision along with the profile page(s) to summarize and get citations. The prompt for the first job is basically the same as the rest of the jobs, except the subsequent jobs will use previous job summaries as context, so the prompt will be slightly different. We basically end up snowballing into a full volume summary this way, in a synchronized way (no parallel calls, as we depend on previous summaries to build the next summary).\n",
    "\n",
    "GPT Vision will do a great job summarizing all the segments with proper character identification, and we gather all the citations as we go and parse them out of the text and map the citations to the actual unscaled page images from the volume. These unscaled page images will later be ran through the `panel_extractor` to get beautiful high-res panels for the video.\n",
    "\n",
    "The final result of this block is a `movie_script` object which contains all the information needed to create the video. It has all of the split up text linked to the unscaled page images that the text refers to. For convenience we also create a `narration_script` string which is a concatenation of all the text segments (so a pure summary), and we log it for the user to see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21d4ac09",
   "metadata": {},
   "outputs": [],
   "source": [
    "character_profiles = [volume[i] for i in profile_pages]    \n",
    "NUMBER_OF_JOBS = 7\n",
    "jobs = split_volume_into_parts(volume, volume_unscaled, chapter_pages, NUMBER_OF_JOBS)\n",
    "parts = jobs[\"parts\"]\n",
    "jobs_unscaled = jobs[\"unscaled_images\"]\n",
    "jobs = jobs[\"scaled_images\"]\n",
    "\n",
    "# Summarize the images in the first job\n",
    "response = analyze_images_with_gpt4_vision(character_profiles, jobs[0], client, BASIC_PROMPT, BASIC_INSTRUCTIONS)\n",
    "recap = response.choices[0].message.content\n",
    "tokens = response.usage.total_tokens\n",
    "movie_script = extract_text_and_citations(response.choices[0].message.content, jobs[0], jobs_unscaled[0])\n",
    "\n",
    "print(\"\\n\\n\\n_____________\\n\\n\\n\")\n",
    "print(response.choices[0].message.content)\n",
    "\n",
    "# iterate thrugh the rest of the jobs while adding context from previous ones\n",
    "for i, job in enumerate(jobs):\n",
    "    if i == 0:\n",
    "        continue\n",
    "    response = analyze_images_with_gpt4_vision(character_profiles, job, client, recap + \"\\n-----\\n\" + BASIC_PROMPT_WITH_CONTEXT, BASIC_INSTRUCTIONS)\n",
    "    recap = recap + \"\\n\\n\" + response.choices[0].message.content\n",
    "    tokens += response.usage.total_tokens\n",
    "    print(\"\\n\\n\\n_____________\\n\\n\\n\")\n",
    "    print(response.choices[0].message.content)\n",
    "    movie_script = movie_script + extract_text_and_citations(response.choices[0].message.content, job, jobs_unscaled[i])\n",
    "\n",
    "print(\"\\n\\n\\n_____________\\n\\n\\n\")\n",
    "print(\"\\n\\n\\n_____________\\n\\n\\n\")\n",
    "print(\"\\n\\n\\n_____________\\n\\n\\n\")\n",
    "\n",
    "narration_script = extract_script(movie_script)\n",
    "print(narration_script)\n",
    "print(\"\\n___________\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52b97d42",
   "metadata": {},
   "source": [
    "This block is responsible for getting all the cited page images from `movie_script` and creating an array of panel images of extracted panels from each cited page image, and incorporate them into the `movie_script` object. This is done by running `panel_extractor` on each cited page image, and then adding the extracted panel images to the `movie_script` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f59dbd29",
   "metadata": {},
   "outputs": [],
   "source": [
    "extract_panels(movie_script)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b1104ab",
   "metadata": {},
   "source": [
    "This block's goal is to identify the `important_panels` for each segment of `movie_script`. In the previous block, we extracted all of the panels from the cited page images, and now we want to ask GPT Vision, given the text that corresponds to the cited page image(s), which panels are the most important to show in the video while that text is being narrated. GPT-Vision does a surprisingly good job at this with limited context, and we can do these GPT Vision requests in parallel to maximize speed. There's all sorts of sketchy error handling, and if GPT Vision screws up (there is no JSON mode for GPT-Vision unfortunately, so we have to run its response through GPT-3.5 with JSON mode enabled as a followup step, and even then there are sometimes problems), we just skip that segment (and use the full page image in the video later on).\n",
    "\n",
    "Once we're done doing this, we do some napkin math to calculate the total costs of GPT tokens and Elevenlabs tokens used in the process, and we log it for the user to see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c08d5793",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"number of segments:\", len(movie_script))\n",
    "for i, segment in enumerate(movie_script):\n",
    "    print(\"segment\", i, \": \", segment[\"text\"])\n",
    "    all_panels_base64 = [panel for sublist in segment[\"panels\"].values() for panel in sublist]\n",
    "    print(len(all_panels_base64))\n",
    "    print(\"number of panels:\", len(all_panels_base64))\n",
    "    print(\"number of images:\", len(segment[\"images\"]))\n",
    "\n",
    "def process_segment(segment_tuple):\n",
    "    i, segment = segment_tuple  # Unpack the tuple\n",
    "    panels = []\n",
    "    for j, page in enumerate(segment[\"images\"]):\n",
    "        if \"panels\" in segment:\n",
    "            if j not in segment[\"panels\"]:\n",
    "                panels.append(page)\n",
    "            else:\n",
    "                for panel in segment[\"panels\"][j]:\n",
    "                    panels.append(panel)\n",
    "        else:\n",
    "            panels.append(page)\n",
    "    \n",
    "    scaled_panels = [scale_base64_image(p) for p in panels]\n",
    "\n",
    "\n",
    "    response = get_important_panels(profile_reference, scaled_panels, client, \n",
    "        segment[\"text\"] + \"\\n________\\n\" + KEY_PANEL_IDENTIFICATION_PROMPT, KEY_PANEL_IDENTIFICATION_INSTRUCTIONS)\n",
    "\n",
    "    important_panels = response[\"parsed_response\"]\n",
    "    # check if important panels is an array\n",
    "    if not isinstance(important_panels, list):\n",
    "        important_panels = []\n",
    "\n",
    "    ip = []\n",
    "    for p in important_panels:\n",
    "        number = p\n",
    "        if isinstance(number, str):\n",
    "            if number.isdigit():\n",
    "                number = int(number)\n",
    "        if not isinstance(number, int):\n",
    "            continue\n",
    "\n",
    "        if number < len(panels):\n",
    "            ip.append(panels[number])\n",
    "        \n",
    "    \n",
    "    return i, ip, response[\"total_tokens\"]\n",
    "\n",
    "# Initialize variables\n",
    "panel_tokens = 0\n",
    "important_panels_info = {}\n",
    "\n",
    "# Use ThreadPoolExecutor to parallelize the processing\n",
    "with concurrent.futures.ThreadPoolExecutor() as executor:\n",
    "    # Create a list of futures\n",
    "    futures = [executor.submit(process_segment, (i, segment)) for i, segment in enumerate(movie_script)]\n",
    "    \n",
    "    # Collect the results as they complete\n",
    "    for future in concurrent.futures.as_completed(futures):\n",
    "        i, ip, tokens = future.result()\n",
    "        if ip:\n",
    "            print(\"Important panels for segment\", i, \"exist.\")\n",
    "        else: \n",
    "            print(\"No important panels for segment\", i)\n",
    "        movie_script[i][\"important_panels\"] = ip  # Assign the important panels back to the segment\n",
    "        panel_tokens += tokens\n",
    "\n",
    "\n",
    "ELEVENLABS_PRICE_PER_CHARACTER = 0.0003\n",
    "print(\"Tokens for extracting profiles and chapters:\", important_page_tokens, \" | \", \"${:,.4f}\".format(VISION_PRICE_PER_TOKEN * important_page_tokens))\n",
    "print(\"Tokens for summarization:\", tokens,  \" | \", \"${:,.4f}\".format(VISION_PRICE_PER_TOKEN * tokens))\n",
    "print(\"Tokens for extracting important panels:\", panel_tokens, \" | \", \"${:,.4f}\".format(VISION_PRICE_PER_TOKEN * panel_tokens))\n",
    "total_gpt_tokens = important_page_tokens + tokens + panel_tokens\n",
    "print(\"Total GPT tokens:\", total_gpt_tokens,  \" | \", \"${:,.4f}\".format(VISION_PRICE_PER_TOKEN * (total_gpt_tokens)))\n",
    "print(\"Total elevenlabs characters:\", len(narration_script), \" | \", \"${:,.4f}\".format(ELEVENLABS_PRICE_PER_CHARACTER * (len(narration_script))))\n",
    "print(\"GRAND TOTAL COST\",\" | \", \"${:,.4f}\".format(VISION_PRICE_PER_TOKEN * (total_gpt_tokens) + ELEVENLABS_PRICE_PER_CHARACTER * (len(narration_script))))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36237786",
   "metadata": {},
   "source": [
    "This block is responsible for making the final movie and saving it to disk. This `make_movie` function does the Elevenlabs narration by narrating all of the movie_script segments (and we parallelize as much as possible to save time), and we end up getting an array of audio clips of the narrated segments. Each segment will now have a corresponding audio clip, the text that was narrated, and the important panels that were identified for that segment, as well as the full page image(s) that were cited. Naturally, we have everything we need to make a movie, and we use moviepy to make it! So we place the audio clips one by one, and get the time length of each audio clip, and split all the images that correspond to the audio clip evenly among that time. So if there is a 10s audio clip, and 5 images, each image will be displayed for 2 seconds in order. Finally, the video is saved to disk and all the temporary files are cleaned up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c972b45e",
   "metadata": {},
   "outputs": [],
   "source": [
    "await make_movie(movie_script, manga, volume_number, narration_client)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8adcb9d2-bf68-4d18-b475-c52145efdd63",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9ebb25c-7473-4f55-b20b-0eff66908581",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d80d4cc7-37a5-44d3-9480-b98dae3c8289",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ac7aac2-9860-4eb8-8d54-8a4c34702fec",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33283437-1e28-43a8-b067-b7b13c412bf1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bb6edb8-6850-4418-87aa-b9496e4d140d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}