{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4a4862d1",
   "metadata": {},
   "source": [
    "# Python for Beginners -- TD 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05bc27e2",
   "metadata": {},
   "source": [
    "[Fanny Ducel](https://fannyducel.github.io/teaching/) (CC BY-NC-SA) -- 2024"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "248267b5",
   "metadata": {},
   "source": [
    "Reminder (see [Intro](https://members.loria.fr/KFort/files/fichiers_cours/M1_Python4Beg_Intro_2024.pdf)):\n",
    "* work by pairs (or alone)\n",
    "* never the same pair\n",
    "* the work should be finished in class (no homework)\n",
    "\n",
    "* your names should appear in the file name:\n",
    "    * TD0-DucelMonir.ipynb (sometimes, it will be .py)\n",
    "\n",
    "* send your work to your teacher by email at the end of the class:\n",
    "    * fanny.ducel@inria.fr\n",
    "    * nasser-eddine.monir@inria.fr\n",
    "    \n",
    "Do not forget to (it's part of your grade):\n",
    "* comment your code (not everything, but the choices you make)\n",
    "* test the extreme values (add tests to your code, then comment them out, so that we can see them)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d750caa",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "For this week's lab, you will have 6 different exercises to do, all related to a same file. However, exercises 5 and 6 are optional, so it is totally okay if you don't have time to complete them.\n",
    "\n",
    "Further, the exercises are quite independent from one another, so if you're stuck on one, you can skip it and come back to it later (except for ex 5 and 6, that will require you to be done with previous exercises). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72434293",
   "metadata": {},
   "source": [
    "## Ex 1 - Let's have a look at a transcription file\n",
    "\n",
    "For this week's lab, we will work with a \".trn\" file: SBC055.trn (found on: https://www.linguistics.ucsb.edu/sites/secure.lsit.ucsb.edu.ling.d7/files/sitefiles/research/SBC/SBC055.trn). Download it and put it in the same folder as this notebook.\n",
    "\n",
    "It is a transcription from an audio recording, so it has some specificities. Let's have a first look at it and create some functions to get some statistics on what's inside.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "818694eb",
   "metadata": {},
   "source": [
    "### 1) Print the vocabulary = all the words of the file, without duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66b80104",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a196036e",
   "metadata": {},
   "source": [
    "You will notice that there are lots of number (= timestamps) and other unwanted characters (they represent linguistic/phonetic information that we don't need for this lab). Don't worry, we will take care of these in ex 2."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09248628",
   "metadata": {},
   "source": [
    "### 2) Create a function that returns the number of words present in the file. It should return both the number of total words and the number of unique words (= no duplicates!)\n",
    "\n",
    "Optional: You could also create a first function to read the file (that takes the path of the file as input and returns a string with what is inside the file as output), and use it inside the function that returns numbers of words. If you create it, you can re-use for next exercises as well!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6231fcd6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c738838a",
   "metadata": {},
   "source": [
    "### 3) Create a dictionary to see the vocabulary\n",
    "\n",
    "Example:\n",
    "`{'good': 6, 'and': 55, ...}`\n",
    "\n",
    "It indicates that the word 'good' is in the text 6 times, the word 'and' is in the text 55 times, ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf9fde7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01e3052a",
   "metadata": {},
   "source": [
    "### 4) Create a function that looks for a word and returns the number of times it is present in the file (the word we look for should be a parameter of the function). \n",
    "\n",
    "Example:\n",
    "`count_word(\"good\", transcription)` returns 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee998009",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0534d233",
   "metadata": {},
   "source": [
    "## Ex 2 - Let's \"clean\" the file! \n",
    "\n",
    "As you probably noticed in ex 1, the file has unwanted information (= a lot of characters/numbers we don't need for this lab, but that are useful for linguists who work on oral language). \n",
    "\n",
    "For this exercise, you don't have to code, I did the job for you! However, I want you to try and understand my code in depth. Pay special attention to the lines with a #TODO, and write additional comments to explain what is going on (what the specific line of code does, how/why it works, what would happen if we remove it).\n",
    "\n",
    "Even if you are not writing code for this exercise, please focus. Knowing how to read someone else's code is a fundamental skill as you will often have to work with other people, or to use other people's code for your own work.\n",
    "\n",
    "*Tip: add `print` to better see and understand what is happening at a given line!*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e387fb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"SBC055.trn\") as f:\n",
    "    # read by line (so that it's clearer)\n",
    "    file = f.readlines()\n",
    "    \n",
    "# Make a big list of all the cleaned lines\n",
    "cleaned_file = []\n",
    "\n",
    "for line in file:\n",
    "    # timestamps, speaker and utterances are separated by tabs, so we use the tabulation character \\t and split\n",
    "    # to only keep the speaker's name and what they say\n",
    "    cleaned_line = line.split(\"\\t\")[-2:]\n",
    "    \n",
    "    # We remove the non-alphabetical characters that we don't need in the utterances\n",
    "    for char in [\".\", \"\\n\", \"[\", \"]\", \"@\", \"%\", \"<\", \">\", \"(\", \")\", \"=\", \"2\", \"3\"]:\n",
    "        cleaned_line[1] = cleaned_line[1].replace(char, \"\")\n",
    "    \n",
    "    # if the speaker's name is empty, we don't keep the empty string\n",
    "    # Exercise: Try and understand these lines of code (explain it on the following line)\n",
    "    # TODO: your explanation on how it is accomplished (understand the syntax) here\n",
    "    if not cleaned_line[0]:\n",
    "        cleaned_line.pop(0)\n",
    "    \n",
    "    # If there's no speaker's name, we add the utterance to the previous list (with the person's name)\n",
    "    # Exercise: Try and understand these lines of code (explain it on the following line)\n",
    "    # TODO: your explanation on how it is accomplished (understand the syntax) here\n",
    "    if len(cleaned_line) < 2:\n",
    "        cleaned_file[-1][-1] += \" \" +cleaned_line[0]\n",
    "\n",
    "    else:\n",
    "        #TODO: What does the following \"if statement\" does? What happens if you remove it?\n",
    "        if cleaned_line[1]:\n",
    "            cleaned_file.append(cleaned_line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9f1b2ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "cleaned_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12281227",
   "metadata": {},
   "source": [
    "## Ex 3 - Turn it into a function!\n",
    "\n",
    "Paste the code from exercise 2 and turn it into a function. \n",
    "\n",
    "You should choose how to call your function, what parameters are required (if any) and what it should return. Also write the documentation for the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01958d13",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "584079d3",
   "metadata": {},
   "source": [
    "## Ex 4 - Who said what?\n",
    "\n",
    "As we work with a transcription file, there are different speakers, each producing many utterances. We want to use this information to try and see who said what, and who spoke the most.\n",
    "\n",
    "In order to do that, we will create a dictionary. It should contain what each person said. So the keys will be the speakers' names, and the values will be a list of utterances.\n",
    "\n",
    "Work from the output produced by exercise 3 (or exercise 2, as the outputs should be the same for both exercises) and create this dictionary!\n",
    "\n",
    "Example: \n",
    "```\n",
    "{'Student_A': ['I have a question',\n",
    "               'Okay, thank you',\n",
    "               \"I don't understand ex 4\"],\n",
    " 'Student_B': [\"I'm tired\", 'Let me help you'],\n",
    " 'Teacher': ['Hello', \"Let's work\"]}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d7755a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "879d2c71",
   "metadata": {},
   "source": [
    "#### Don't forget to answer the question: who spoke the most? \n",
    "\n",
    "Use the dictionary you just created to find the answer!\n",
    "\n",
    "If you couldn't do it, just create a dictionary manually with the name of the speakers and random utterances (or use the one provided as an example for the previous question)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1127ad77",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6e2b253",
   "metadata": {},
   "source": [
    "## Ex 5 - Compare this transcription to another one!\n",
    "\n",
    "Choose another transcription file from https://www.linguistics.ucsb.edu/research/santa-barbara-corpus and download it (choose the \"TRN\" file!).\n",
    "    \n",
    "Run all the functions from previous exercises to compare the transcriptions. What differences can you observe? (Time to use your linguistics skills!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9fe944f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11ef9797",
   "metadata": {},
   "source": [
    "## Ex 6 - Visualize your data\n",
    "\n",
    "Use the results from previous exercises to create some plots and visualize your results. You can decide what data you want to visualize. \n",
    "\n",
    "For example, you could plot the number of words from SBC055 and the number of words from the file you chose in ex 5 to compare the lexical diversity.\n",
    "\n",
    "If you didn't do ex 5, you could plot the number of total words vs. unique words to see how lexically rich the utterances are. You could also plot the number of utterances per speaker, to give a more visual answer to the question \"Who speaks the most?\" from ex 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aff664d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "\n",
    "# Example of how to use matplotlib (from TD2)\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "xs = [\"Karën\", \"Amandine\", \"Clémentine\", \"Fanny\"]   # a list of names that will be used as X axis\n",
    "ys = [10, 55, 66, 88]           # a list of nbs, corresponding to the xs list, to be used as Y axis\n",
    "\n",
    "plt.bar(xs, ys)\n",
    "plt.show()\n",
    "# Make sure to close the plt object once done\n",
    "plt.close()"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Format de la Cellule Texte Brut",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}