diff --git a/quickstarts/Audio.ipynb b/quickstarts/Audio.ipynb
index 41252cda..a619d0fb 100644
--- a/quickstarts/Audio.ipynb
+++ b/quickstarts/Audio.ipynb
@@ -57,18 +57,33 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 30,
"metadata": {
"id": "Y6eH_Aq_NyNi"
},
- "outputs": [],
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/142.2 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r",
+ "\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━\u001b[0m \u001b[32m133.1/142.2 kB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m142.2/142.2 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25h\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/664.5 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r",
+ "\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m\u001b[90m━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m368.6/664.5 kB\u001b[0m \u001b[31m11.0 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r",
+ "\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m \u001b[32m655.4/664.5 kB\u001b[0m \u001b[31m10.8 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m664.5/664.5 kB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25h"
+ ]
+ }
+ ],
"source": [
"!pip install -q -U google-generativeai"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 31,
"metadata": {
"id": "LSe1pMEpR2L2"
},
@@ -90,7 +105,7 @@
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 32,
"metadata": {
"id": "dm-iaNMGPdid"
},
@@ -115,7 +130,7 @@
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": 33,
"metadata": {
"id": "OHvNLws4RRjx"
},
@@ -126,7 +141,7 @@
},
{
"cell_type": "code",
- "execution_count": 3,
+ "execution_count": 34,
"metadata": {
"id": "Cxq31LDwSFH6"
},
@@ -137,7 +152,7 @@
},
{
"cell_type": "code",
- "execution_count": 4,
+ "execution_count": 35,
"metadata": {
"id": "MAObE0BpaAwG"
},
@@ -157,7 +172,7 @@
},
{
"cell_type": "code",
- "execution_count": 5,
+ "execution_count": 36,
"metadata": {
"id": "YmISEsqpafRb"
},
@@ -166,7 +181,33 @@
"name": "stdout",
"output_type": "stream",
"text": [
- "President John F. Kennedy delivers his State of the Union address to the United States Congress on January 30th, 1961. He addresses issues concerning the nation's economy, foreign policy, and the state of the executive branch. The President acknowledges the nation's economic struggles, including high unemployment and a failing agricultural market, but he remains optimistic. He emphasizes the importance of maintaining a strong military, working with allies, and bolstering the United Nations. He suggests expanding a \"Food for Peace\" program to help alleviate hunger and malnutrition, and he urges collaboration with the Soviet Union on scientific endeavors. The President concludes his speech by acknowledging the challenges ahead and urging Americans to work together to overcome them. \n"
+ "## Summary of President John F. Kennedy's 1961 State of the Union Address:\n",
+ "\n",
+ "**Domestic Concerns:**\n",
+ "\n",
+ "* The address primarily focused on the concerning state of the American economy, highlighting issues like recession, unemployment, and falling farm incomes.\n",
+ "* Kennedy pledged to address these issues through measures such as improving unemployment benefits, expanding food assistance programs, and stimulating economic growth.\n",
+ "* He acknowledged other domestic problems like inadequate housing, education, and healthcare, promising to introduce new programs and initiatives to tackle them.\n",
+ "\n",
+ "**International Challenges:**\n",
+ "\n",
+ "* Kennedy emphasized the rising tensions of the Cold War and the threat posed by communist expansion in Asia, Africa, and Latin America.\n",
+ "* He reaffirmed the nation's commitment to containing communism and supporting allies across the globe.\n",
+ "* He proposed a multifaceted approach involving strengthening the military, improving economic aid programs, and utilizing diplomacy to achieve international stability. \n",
+ "\n",
+ "**Specific Actions:**\n",
+ "\n",
+ "* Kennedy outlined plans to bolster the nation's military capabilities by increasing airlift capacity, expanding the Polaris submarine program, and accelerating missile development.\n",
+ "* He advocated for a new and more effective foreign aid program to assist developing nations and promote economic growth in the non-communist world. \n",
+ "* He expressed the desire for increased cooperation with the Soviet Union in areas like scientific exploration and weather prediction, while also remaining firm against communist aggression and subversion.\n",
+ "* He pledged support for the United Nations as a crucial instrument for maintaining peace and international order.\n",
+ "\n",
+ "**Overall Tone and Message:**\n",
+ "\n",
+ "* Despite acknowledging the critical challenges faced by the nation, Kennedy's address maintained a tone of optimism and determination.\n",
+ "* He called upon the American people and government to rise to the occasion, embracing the spirit of sacrifice and service to overcome these obstacles.\n",
+ "* He emphasized the importance of unity, perseverance, and dedication to the national interest in navigating the turbulent years ahead. \n",
+ "\n"
]
}
],
@@ -177,6 +218,142 @@
"print(response.text)"
]
},
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ln36O5eNLltg"
+ },
+ "source": [
+ "## Inline Audio"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "AVaX93lvLqQB"
+ },
+ "source": [
+ "For small requests you can inline the audio data into the request, like you can with images. Use PyDub to trim the first 10s of the audio:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 37,
+ "metadata": {
+ "id": "XLZT7rrzLpzp"
+ },
+ "outputs": [],
+ "source": [
+ "!pip install -Uq pydub"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 56,
+ "metadata": {
+ "id": "0Kbji0xRMIhr"
+ },
+ "outputs": [],
+ "source": [
+ "from pydub import AudioSegment"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 57,
+ "metadata": {
+ "id": "umFIVVlHLlQD"
+ },
+ "outputs": [],
+ "source": [
+ "sound = AudioSegment.from_mp3(\"sample.mp3\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 40,
+ "metadata": {
+ "id": "hKoLR5mdMMdn"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ " \n",
+ " "
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "execution_count": 40,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sound[:10000] # slices are in ms"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "66MNT0mFP4x-"
+ },
+ "source": [
+ "Add it to the list of parts in the prompt:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 52,
+ "metadata": {
+ "id": "420qRCkGNg9j"
+ },
+ "outputs": [],
+ "source": [
+ "response = model.generate_content([\n",
+ " \"Please transcribe this recording:\",\n",
+ " {\n",
+ " \"mime_type\": \"audio/mp3\",\n",
+ " \"data\": sound[:10000].export().read()\n",
+ " }\n",
+ "])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 59,
+ "metadata": {
+ "id": "Rhpeo_3uPLtq"
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/markdown": [
+ "## Transcription of Recording:\n",
+ "\n",
+ "\"The President's State of the Union Address to a joint session of the Congress from the rostrum of the House of Representatives...\" \n"
+ ],
+ "text/plain": [
+ ""
+ ]
+ },
+ "execution_count": 59,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from IPython import display\n",
+ "\n",
+ "display.Markdown(response.text)"
+ ]
+ },
{
"cell_type": "markdown",
"metadata": {