{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false
   },
   "source": [
    "# [Introduction to Data Science](http://datascience-intro.github.io/1MS041-2023/)    \n",
    "## 1MS041, 2023 \n",
    "©2023 Raazesh Sainudiin, Benny Avelin. [Attribution 4.0 International     (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other measurements of performance\n",
    "\n",
    "Recall that in the logistic regression case our function $G(x) \\in [0,1]$ and represents the probability of the label being $1$, we then used the following rule to construct a decision function from this $G$, i.e. \n",
    "$$\n",
    "    g(x) = \n",
    "    \\begin{cases}\n",
    "        1, & \\text{if } G(x) > 1/2 \\\\\n",
    "        0, & \\text{otherwise.}\n",
    "    \\end{cases}\n",
    "$$\n",
    "The parameter $1/2$ can be changed in order to create a trade-off between precision and recall.\n",
    "\n",
    "Lets consider the function\n",
    "$$\n",
    "    g_\\alpha(x) = \n",
    "    \\begin{cases}\n",
    "        1, & \\text{if } G(x) > \\alpha \\\\\n",
    "        0, & \\text{otherwise.}\n",
    "    \\end{cases}\n",
    "$$\n",
    "where $\\alpha \\in [0,1]$, then for each such $\\alpha$ we get a precision and recall, i.e.\n",
    "$$\n",
    "    \\begin{aligned}\n",
    "        \\text{Precision:} \\quad \\text{Pr}(\\alpha) =  P(Y = 1 \\mid g_\\alpha(X) = 1) \\\\\n",
    "        \\text{Recall:} \\quad \\text{Re} (\\alpha) = P(g_\\alpha(X) = 1 \\mid Y = 1).\n",
    "    \\end{aligned}\n",
    "$$\n",
    "\n",
    "These functions can be plotted as functions of $\\alpha$, we can see that below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SVC(kernel=&#x27;linear&#x27;, probability=True)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC(kernel=&#x27;linear&#x27;, probability=True)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "SVC(kernel='linear', probability=True)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_digits\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_digits\n",
    "digits = load_digits()\n",
    "\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "labels = digits['target'] >= 5\n",
    "\n",
    "X = digits['data']\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X,labels)\n",
    "\n",
    "per = SVC(kernel='linear',probability=True)\n",
    "\n",
    "per.fit(X_train,Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SVC(probability=True)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC(probability=True)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "SVC(probability=True)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "per_rbf = SVC(kernel='rbf',probability=True)\n",
    "\n",
    "per_rbf.fit(X_train,Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Utils import classification_report_interval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "            labels           precision             recall\n",
      "\n",
      "             False  0.89 : [0.77,1.00] 0.86 : [0.74,0.99]\n",
      "              True  0.87 : [0.74,0.99] 0.90 : [0.77,1.00]\n",
      "\n",
      "          accuracy                                        0.88 : [0.79,0.97]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report_interval(Y_test,per.predict(X_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "per.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_curve\n",
    "prec,rec,thresh = precision_recall_curve(Y_test,per.predict_proba(X_test)[:,1])\n",
    "prec_rbf,rec_rbf,thresh_rbf = precision_recall_curve(Y_test,per_rbf.predict_proba(X_test)[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tradeoff between precision and recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x168cf3970>]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABI4UlEQVR4nO3deXwV9b3/8ddZkpM9IQuBkLBvAQQlCAKidcOCBa1a6bV1q7bS2lqleq+U/rRa76W11loXUCtqrUhRXKotLtEqi+ACBkRBdkggCdkge05yzpnfH0MCkQA5Icmc5f18PObxnTOZOeeTETlvZr7z/doMwzAQERERsYjd6gJEREQkvCmMiIiIiKUURkRERMRSCiMiIiJiKYURERERsZTCiIiIiFhKYUREREQspTAiIiIilnJaXUB7+Hw+CgsLiY+Px2azWV2OiIiItINhGFRXV5ORkYHdfvzrH0ERRgoLC8nKyrK6DBEREemAgoICMjMzj/vzoAgj8fHxgPnLJCQkWFyNiIiItEdVVRVZWVkt3+PHExRhpPnWTEJCgsKIiIhIkDlZFwt1YBURERFLKYyIiIiIpRRGRERExFIKIyIiImIphRERERGxlMKIiIiIWEphRERERCylMCIiIiKWUhgRERERS/kdRlauXMmMGTPIyMjAZrPx+uuvn/SYFStWkJOTQ1RUFAMHDuSJJ57oSK0iIiISgvwOI7W1tYwZM4bHHnusXfvv3r2b6dOnM2XKFPLy8vj1r3/NrbfeyiuvvOJ3sSIiIhJ6/J6bZtq0aUybNq3d+z/xxBP07duXhx9+GIDs7GzWrVvHgw8+yBVXXOHvx4uIiEiI6fKJ8tauXcvUqVNbbbv44otZtGgRTU1NREREHHOM2+3G7Xa3vK6qquqa4jYsgaKN7ds3oTdM/DnYHV1Ti4iISBd5+8siPtldccJ9rhibyag+id1UUWtdHkaKi4tJT09vtS09PR2Px0NZWRm9e/c+5pj58+dz7733dnVpsOM9+HJZ+/dPHgQDppjrrgQ4ySyEIiIiVmto8vKLJXk0eY0T7ndG3x6hG0bg2KmDDcNoc3uzuXPnMmfOnJbXVVVVZGVldX5hw6dDj34n32/Lm1C2DZb+4Mi2IRfDD17q/JpEREQ6kbvJ1xJEfvqtQdiP8+/oIT3jurGq1ro8jPTq1Yvi4uJW20pKSnA6naSkpLR5jMvlwuVydXVpMOoKczmZHgPgX7eBz3Nk2/Z34K3/AXs7T+Gw6dB/cofKFBER6ahGrw8wL+b/98XDjnshwEpdHkYmTpzIm2++2Wrbu+++y7hx49rsLxKQxl4DY74PhvkflGcuhsI8+MSPR5TzXoBb88DZCSHLZoeI6FN/HxERCXlNh8NIhN0ekEEEOhBGampq2LFjR8vr3bt3s2HDBpKTk+nbty9z585l//79PP/88wDMnj2bxx57jDlz5vDjH/+YtWvXsmjRIpYsWdJ5v0V3cBwVnC5dYPY18Xnbd+zGf0BNMTwwoPPqOedOOP83nfd+IiISkjyHb9E4HYEZRKADYWTdunWcd955La+b+3Zcd911PPfccxQVFZGfn9/y8wEDBrB8+XJuv/12Hn/8cTIyMnjkkUeC+7He9BGQfnf79+/RH/4958iVlc7w2SLwNprr0ckw7gaIsqbjkYiIBK7SmgYA6hrb+Q9oC9iM5t6kAayqqorExEQqKytJSEiwupyO8bjbfyXlhO/TAH8aDl536+1p2fD9xZDQx7/3c0TocWURkRD2wdYSbnj2MwD2/P6Sbv3s9n5/d8vTNELn9BUBiIyBWS/A7hVHtm1aBqVb4NGx/r9fVCJM/AWc9VNwWdeTWkREOkeN20N9o5e0ePN7p7rBfPhiwoBkK8s6IYWRYDR0qrk0O+tn8PJ1sO8z/9+roRI+uB8+fRJGzwJH5In3t9lg+HegTweCj4iIdCnDMPjOI6vYU17HmMxErpvUn8r6JgB6xJzk73cLKYyEgsQ+cGMuuKv9PNCA7bnwn/vh4G5Y2775hljzGFz3JvTJ8bvULuHQH2MRCW15+Qe54+WNpMa5SI13kRgdQUpsJJk9ohk/IIUBqbGA+RjvnvI6ADbuq2TOS0dGGU+KCdwnWPW3eKiw2SCqA/1pTrsSsmfCF0uhZPPJ99//ORR8DM9MPfm+3SXrLDjnDhh8oUbFFZGgVOP2UFHTSFZydJuP3y79rICdpbXsLK1t8/gJA5L57hl9OD+7Z8u22y4cwsPvbW95XVTZ0PmFdxKFEQFnpDmWSnu4a+DFWbB3ddfW5I+Cj2HxldD7dBgy1RwTJmWQ1VWJiLRLk9fHRQ+toKiygcToCCYPTuH84en0jHeRHBvJgNRYvthXCcC3R/ZiwsBkKuubKK12s7O0hk93V/DJ4eVot104lBsmD+DM/32PRo/P0hFWT0ZP04j/DAMaDlldhamhCj59CtY9A03mpUmik+HmlZCYeez+unIiIhYxDIM/527jsz0HuSC7J8N6xTMsPR6bzcaZ//teu95jzV3nk5HUetDLgoo6XlpXwEvrCjhQdeRJy+YnZ6obmli7s5xzh6Xhcnbv05Pt/f5WGJHQUFsG65+Fjx4Fd2Xb+6QOg5mPQN+zurc2EQkIDU1equqbiHU5iXTacdptXTYiqc9nsG7vQTKSouiTZN562VpczcUPrzzuMTYbvPLTSSz9tIC9FbUcqmuirMZNWY05ptTgnnHk3n7OcWv2+Qw+zz/IA29v5axBKcy5aGiX/G7+UBiR8HRwDzx/qdm2xWaHsddC/LGzRbcpJsXcv7MezRYRSzR6fFz2+EdsLqpq2Wa3QaTTTqTDTqTTgctpx+W0E+m0kxAdwZCecUwenEpOvx4kxUQQ6WjfcOqGYTDv9S958RNzANC0eBejMhLwGbBiWykA00b1YuuBavaU1eI7/C3cJymaj+46/5j3O1TXSEFFPX2TY0gM4E6obVEYkfDl9ZiPLB/N02A+NbTxRf/f7/K/wuirOqc2Eek2q7eXsbmokp7xUXxVWMlfV+0+pfdz2G1ERziIinAQHWknOsJB2uEnW2IjncS6nMRHOdl+oIa3vyo+7vv8z7eH89Nvmf3aGpq8bDtQzUc7yjmjbxJnDWx7AtlgpTAi0patb8OOXLPfy8nsWQ1lW811m71r6+oMcekw6HwYfAEMPA9iAneAI5Gulrv5ADf/fV3LVYdm8y8/jctO70Ojx4fb68Xd5KPR66PRYy7uw21pTQOf7j7Ix7vK2VNe266/Mr7pN5dk84MJ/dhcVMlXhVVsLqyirtHLfZeOJCmAx/zoTAojIqdqw4vw+s+AgP9f5Fg2uzkOTP+zwRUPIy7TE0YSNh58ZyuPfWBO6Dq2bxIup4OS6gZO65PIQ1edjt3uXz8Rn8+g2u2hoclLfaOX+iZzqXN7Ka1poLrBQ3WDh1q3h20Hali9o5RrzurHr6dnB+wsud1FYUSkM9QfMucVCngGlH4NO96D7e+Z0wMcLakvzF6tyRQl5P1zw35++Y8NAJw7NI2nrxtHhKN7r2wahhH2IaSZwohIOKvcBzveh8I82P4uVO2HfmfDpY9C8kCrq5MwVlHbyB/f+Zo+SdH0TozG6bAR6bDjdNiJcNiIcNgPL0fWM5KiiI86ccfNFz/J5/EPdrD/UD0Aw9Ljee2WScREajgtKymMiIhp3zp47hKzE68zCr41F86+zeqqJAjVN3r5n1e+IL+ijkiHnQinGSQiHHYiDj+VEuGwEem0ExPpJM51eIlyMqRnHBlJ0fzj0wL+/N42vz87Kzma7F4JpMRFkhQTSXJMJEkxEWQlx/DmxkIWH35yBSCnXw+W/uQsnN18RUSOpTAiIkeU7YB/zzky2/N1/4IBU6ytSSzn9Rl8vKucf27Yzxf7KkmONb/gk2IiGdIzjthIJzEuB7EuJ7GRTl5eV8DL6/d12uefOzSNJq/v8GLQ5PXhOdw2Hl5v8Hg5VNd00vey2WDOhUOZNDiV7N7xuiISIBRGRKQ1w4CXroUtb4DdaS7f5HTBzMdgxMzur0+6zdfFVSz9rIB/fVFEabV/faJsNrh35kjS4lwtT6E0eQ0aPV6zPbytvslLdYOHGreHQ3WNfFVYxaG6RnyGObbH09eO45yhae36zIO15vG7y2o4WNdERW0jh+oaKa9tZE95LXabjXtmjOD84ekdOR3Shdr7/a3oKBIubDYYPQu2vAk+j7l8k6cBlt9pzuLsj6wJGtk2SHy2p4L/eupjPIefeU2MjmD6ab351rA0SqrdbCmqwus1qG30UNfopdZ9pG1o8nLtpP5cO7F/hz7bMAyavAYGhl/DkveIjeTsIamcPSS1Q58rgU9XRkTCTW05NLUx86fPa45ee2iv/+/pcMHPP4Ue/U+5POk6+w/Vc8WCNRRXNXDWwGR+PGUgU4akEelU3wrpGroyIiJti00BjjPK4/eeg3WLzGDSXvs/NweHe/5SGPpt6DcJ+k6CuPZdgpfusbHgED9+fh0l1W4GpcWy6LoziXXpK0ACg66MiMipOfAVPDPt2AkKU4eZwWTAFHNk2Oge1tQXpmrdHj7eVc7KbaWs3F7G7jLzatjQ9Dieu2H8MTO/inQFdWAVke5TWwa7V8LeNeZS8lXrn9sckDUehkw1l/SRZh8W6XQ7S2v47Rtf8fGucpq8R/56d9htXJSdzh+uHE1idHBNtibBS2FERKxTVwH5H5vz++z8z7EjwqYOhRtzITrJkvJCTV2jh12ltWwtruaeN76ixm12Ts5KjuacIWmcMzSNSYNSTjpwmEhnU58REbFOTDIMn24uAAf3mhMUbs+FnR9A2TZ4/z4zlLTF7jD7nyRldV/NFtpZWsOLn+QTHeEgKSaC5NhIesRG0uPw4F49YiOIczk5WNfEjpKaI0tpDTtLalpGHW3WKyGKF24az6C0OA1LLkFBV0ZEpHu99lPY+OLJ9+s/Ba7/V9fXEwDOeeAD8ivqTriPy2nH7fEd9+fJsZFk9YimyWvwu8tGkdNPfXTEeroyIiKBacocc1ZhT33bP2+sg21vQeEG83ZPTHK3ltfdNhdWtQSR6af1wmm3c7CukYraRg7WNnKwron6Jm9LEOmTFM3gnnGtlkFpcSTHhseU9BKaFEZEpHulDoHLHj/+z71N8Jcx5uR+z11ijgibcQbYQ2csjKqGJr7cV8nGfZWs3Fbasv3Ps05vczCwGreHokP1ZPaIITqy/YOFiQQLhRERCSyOCPjhK/D8ZVCyGZ4+H6KTYdB5MOgC8zHhhN5WV+m397cc4N9fFLFx3yF2lh476NyVOZnHHZU0zuVkSHp8V5coYhn1GRGRwHRwD+TebXZ4dVe1/lnmeJj6u6AZgr6+0cvoe99p9ahtn6RoxmQlMiYzidGZSZzZv4dmmZWQoz4jIhLcevSHq543b9vsWwc734cd70NhHuz7FJ65GE77njmwms0BSX0hbRjE9w6YMUx8PoO1u8p58dN8mrwGDruNp68dx+jMRFLiXFaXJxIwdGVERIJLdTH8537IewFo46+vyHizX0rqULNNG2au9xgAzu7r5PnuV8X87t+bKag40lH3ktN68/gPxnZbDSJW06BnIhLaCvPgk6fMWzjeRqjYBRW7wTjOvDo2B8SmHblqYo+AgefC6Kug39mn1EH2ox1lrNxWSnSkg5hIB9UNHh7/YAc+A+JdTmaensH3z+zLqD4JGvdDworCiIiEH08jHNwNpVvNgdXKtpuT+JVth8aa4x8XnwGnXQEjvwu9/Xtyp9HjI+d3uVQfHvX0aFeNy+TemaP0BIyELYUREZFmhgFVhVBXfmRbXRl89Tpsfh0ajprkL763OfrrsOkw4ByIiGrzLX0+gw+2lvDkil18uqeChCgnM8ZkUN/opbbRw5n9k/nR5AHY7boSIuFLYUREpD08btj+LmxaBjvea30FJSLGfKz4KAYGdY1eatwePD7zr08bEJGUQdrk68xOtZpzRwRQGBER8Z/HDXtWwdfLYetbUF3o/3s4o2HkZdBvsjnSbHvY7DDwW0E5forIiSiMiIi0we3x8t7mElZtL8VmsxET6SA6wtHS+bR5PdppJ65mF5v2FPPp7goOVDW0vEdanIuZp2cwdWQ6sZFO8zZQwafw+d/Mgdo6IussuPGdTvotRQKDxhkRkaDl8xnt7mtRUFHHsvX7aPT6iLDbiHDYcTrsRDhsOO02nA47kQ47ToeNjQWH+OfGQg7VNflRTRSQQVSEnbMHpzFjTG+mn9abiG8OUNZnLEy42RwTZcNiczj79jB85u2hgo/hi5dh9Pf8qE0kNCiMiEjAKKtx89TKXSz+eC/piVEMTosjITqCOJeTmEgHsS4nsc2ty0nhoXr+nLuN2sbjPM57HL0Sopgxpjdxrgjqm7zUN3qob/JS1+ilvtHbsu72+BiVkcBFI9KZMiTt5E/F2GyQdaa5+OPJc6BoI7x6E+z+EL79B3DF+fceIkFMt2lExHLlh0PI82v3Ut/kX7AAyOnXgzGZSTR5fXh8Ppq8Bh6vjyafQZPHh8dn0OT1kRbn4tIz+nD24FQcgfSUi7saPvw9rH0cMKDPOPjhMojuYXVlIqdEfUZEJODtO1jH39fu5e8f76Xu8NWN0ZmJ/HBCP5JjIymqrKfa7aHObT69UtfoodZtPjpb5/bS6PUxc0wG10/qHxqP0O5eBS9dA/UHoedIuOJpSB9hdVUiHaYwIiIB69PdFTy5Yif/2VpC899Ap/VJ5LYLh3D+8J7hPUrpga/MGYtrS8xRYs/9Hzj7dnDorroEH3VgFZFTYhgGlfVNlFa7zaXGfWT9G68ToyPISIomKsKOK8J8IiUqwk6U00FU83qEA1eEg/0H63lixc6Wz5k8OIUbJg3gguwwDyHN0kfCzSvhX7fDtrfgg/uhfAdctvCUhqwXCWQKIyJhyOcz2H+onm0Hqvm6uJptB6rZWlxN4aF6eidGU+P2UFrtptHra9f7ldc2squs1q8aTs9K4k9XjWFQmjpqHiOhN/zXEti4BP75c/jiHxDfCy661+rKRLqEwohIiNt2oJroCAcZSdFUNzTxiyV5fL734HGfQKlqqG71OjE6grR4F2lxLnommG1a/JElJdZFaY2bilo3DU0+Gpq8R1qPF3eTj/pGc72hycuhuiYKK+v59fRsBZETsdng9KvNAdFeuxk+Xgi9TjvxQGqOCBh4np7EkaCjMCISwj7YWsINz34GmN9tR/cQi3DYGJQWx7Be8QxNj2dYejy9EqMorXbTIzaStHgXqXGRuJya5M1So2fBR38xB1N75caT7589E2b9vevrEulECiMiIaCyvol9B+uOjB4a4cDldPDkUX0zjg4i5w1L46lrxx07cJcEHpsNvv17WP0QeI+dGfgIA/LXwpY34PEJmDPm+CE+HaY9AGnDTqVakQ7R0zQiQe5gbSPT/rKK4qOGKz+ay2nn37eeTWJ0JAeqGth2oJrzh/ckKSaymyuVLvevObBuUcePT+oHP/4PxKZ2Xk0S1vRor0iIMgyDveV1rNlZzpqdZazdWU55bSNREXYiHHYamrw0ec3/re02+POs07n09D4WVy3dwtsE+9eDt9G/43xe8+mdg7thyFS4+iXziozIKdKjvSIhyOcz+P5fP+bT3RWttifHRvLcDWcyOjMJgCav2YHUZrMR59L/5mHDEQF9z+rYsf+1xByWfvu75q2eEZd2bm0iJ6C/pUQCkM9nUFbjpuBgPfsO1rH/UD37Dtbz+d6DfF1sPu0yfkAykwalMGlQKmOyElt1NI1w2NUfRPzTMxvOvAk+XmAGEoUR6UYKIyIW8fkMiqoa2FVaw+6yWnaV1rKrrJaCCjN8NHqOP8bHGX2TeOnmid1YrYSFxCyz3fImJA8yZyGOjLW2JgkLCiMi3cjt8fLbNzaTl3+QPeW1NDQdP3DYbdA7MZo+PaLJTIoms4e53icphpx+mkBNusDw6bD+WSjbBu/fa07cd9XfoP/ZVlcmIU5hRKQbrdlZzpJP81teRzhs9E2OYWBaHANTYxmQGku/lFgye0TTKzFKt1qke/XoDz/7GDa9DB/Oh4N74NWb4ZaPwRVvdXUSwhRGRLrRFwWVgNnh9LWfTaJPUjROBQ4JJHYHjPm+OXjagglwKB8eHQdn/RTG3QBRiVZXKCFIYUSkG5RUN/BFQSXPr90DwG8uyaZfiu7FSwCLjIHLn4aXr4fqQnjvHlj1J0gZDAkZkHE6ZIyFjDMgJtnqaiXIaZwRkS5wqK6RFz/NZ0P+Ib7YV9lqQLJh6fG8+YuziXTqiogEAU+jedvmo4fNviRt6dHfDCUZZ8CAc8xWBA16JmKpua9+wZJPC1pe22wwOC2OMVlJ/PKCIWQlx1hYnUgH+HxQ8AnUH4SKXVCYB4Wfm+vfNPV+mPSL7q9RAk6XDnq2YMEC/vjHP1JUVMTIkSN5+OGHmTJlynH3X7x4MQ888ADbt28nMTGRb3/72zz44IOkpKR05ONFAo5hGFTUNrKztJa3vixqCSIjeidwz4wRjOqTSKwGH5NgZrdDvzYeJ68/CIUbzHCSv9Yco+Td35idXyf9wrxqInISfl8ZWbp0Kddccw0LFixg8uTJPPnkkzz99NNs3ryZvn37HrP/6tWrOffcc/nzn//MjBkz2L9/P7Nnz2bIkCG89tpr7fpMXRmRQOL2eFm7s5yvi6vZWVLDztIadpbWUlnf1Gq/swen8tjVZ2gOGAkvH/7efBIHwGaHYdNh0q3Qd4K1dYkluuw2zYQJExg7diwLFy5s2Zadnc1ll13G/Pnzj9n/wQcfZOHChezceWT20EcffZQHHniAgoKCY/Zvi8KIBIJV20t59fP9vLf5ANXuY2dPtdmgT1I0w9LjuX5yf6YMSbOgSpEAsP09WPsY7PrAfG1zwI25kJljbV3S7brkNk1jYyPr16/nrrvuarV96tSprFmzps1jJk2axLx581i+fDnTpk2jpKSEZcuWcckllxz3c9xuN263u9UvI2KlVz/fx5yXNra8Tk9wMX5ACoPT4hjUM5ZBaXEMSI0lKsJxgncRCRNDLjSXkq/h37+Cvash7+8KI3JcfoWRsrIyvF4v6enprbanp6dTXFzc5jGTJk1i8eLFzJo1i4aGBjweDzNnzuTRRx897ufMnz+fe++915/SRDqFx+tj38F6dpXVsKu0lje/KMLr8/HlfjMQf2d0b66f1J+xfXtgt2tWU5ET6jkczrkD/r4aNr8O0x4Ap25byrE61KPO9o2ppQ3DOGZbs82bN3Prrbdy9913c/HFF1NUVMSdd97J7NmzWbRoUZvHzJ07lzlz5rS8rqqqIisrqyOlipzQztIa3txYyObCKnaV1bK3vJYm77F3Lu02+M7oDB64crSufoj4Y8A5EJcONQdg9woYcpHVFUkA8iuMpKam4nA4jrkKUlJScszVkmbz589n8uTJ3HnnnQCMHj2a2NhYpkyZwv3330/v3r2POcblcuFyufwpTaTdPF4fr+XtZ/En+WwoOHTMz11OOwNSY/H6DFwRdi45LYPLx/YhPSGq+4sVCXZ2Bwy+CDa8AF+8BP2nQIT+X5LW/AojkZGR5OTkkJuby3e/+92W7bm5uVx6advTTdfV1eF0tv4Yh8P8l2UQDHEiQczj9fHF/ko+2l7GRzvLqKr3kJEUzfaSavaW1wHgsNs4d2gaU4aktswP0ycpWrdgRDrTkAvNMLLpJdizCqb8CnKuB0eE1ZVJgPD7Ns2cOXO45pprGDduHBMnTuSpp54iPz+f2bNnA+Ytlv379/P8888DMGPGDH784x+zcOHClts0t912G+PHjycjI6NzfxsJe0WV9UQ47Cz+OJ9Fq3dR1dD6qZfNRWbfj5TYSG6cMoArczLpGa9/pYl0qRGXwcxH4cM/QNU+WH4H1JbCeb+2ujIJEH6HkVmzZlFeXs59991HUVERo0aNYvny5fTr1w+AoqIi8vOPzEp6/fXXU11dzWOPPcavfvUrkpKSOP/88/nDH/7Qeb+FCLBs/T7+e9lGfEddcEuIcjJpUCpnD0klIymK/YcaiHTYmDEmg5hIDUIm0i1sNhh7LYyeBSv/aC6b/6kwIi00HLwELZ/P4Nk1e6iqb6LB4+XJFUeGpU6Nc3H3jBFcclpvHLrlIhI46irgj4PB8MIvPoeUQVZXJF2oS4eDF7GSYRjsLa9jwYc7eGndvlY/G5mRwAs3TiA+yonToYnoRAJOTDIMOg92vAdfvWY++ithT2FEgkZlXRO/+eeXrN1ZRllNY6ufXTQinZoGDz87bxA9YjWOgUhA6zXaDCNb3gRXAqQNhdRhEN/LvKUjYUdhRAKa12ew7UA1GwoOseDDHRRU1AMQ6bBzWmYiOf168IMJfemXEmtxpSLSbnGHh4Io2mAuzVwJkDoU0oYdaftNhijdng91CiMSsP7w9tf8bc0e6hq9rbb/5JyBzLloqAYfEwlWZ94Isalw4Eso3QZlW6FiN7irYP86c2nWoz/MXg2ueMvKla6nMCJdwuP1UdXgodbtobrBQ43bQ427ieoGD7VuLzXuJmoaPFS7PdQ0eKhtPGq/Bg9FlQ3UHJ6MLs7lZHRmIqdnJXHmgGTOHZKmcUBEgpkjAk670lyaedxQsQtKt0LZNrPd9QEc3AMfL4Rz/9uycqXrKYxIp2r0+Hh+7R4e+2AHh+qaTvn9fnhWX+6dOUpPxIiEOqcLemabS7NNy+CVG2HNo3DmTWbnVwlJCiPSKQzDoLK+idkvrOfjXRUt26MjHMRFOYl3OYmLchLnchLrav26pXUdeR3viiAxOoKs5OjjznskIiFu5OXw0cNQvAk+e1pXR0KYwoj4bW95LR98XULBwXoKKuooOFjPvoo6qt1HRjv9zSXZXDepPxF6vFZEOspuN0dvLd4EB/daXY10IYURabcmr4+7//kVSz7NP+4+afEupo/qxU1TBnZjZSISshIOTxuy/V2oLYfYFGvrkS6hMCLttnJbaasg8l/jsxjeK4Gs5GiyesSQ2SOG6Eg94SIinWjk5fDRX6D0a/jP72DGw1ZXJF1AYUROqtbt4f2vS5izdEPLtseuPoPvjNZEhyLSxSKi4OL/gxcuh6//DVN/p8d8Q5DCiBzDMAzW7iznw22lfLK7gi/3V+I9ava5P145WkFERLpP/7MhugfUlsDCSXDpAhgwxeqqpBMpjMgx3vyiiFuX5LXa1jc5hktPz+DysZkMSNVopyLSjZwu+K9/wKs/hkP58LcZMO0BmPATqyuTTqIwEuZq3R5iIh3YbDa+Kqzksf/s4K0vi1t+/tBVYxg/IJnMHjEWVikiYa/vWfDTNfDWXbDhBXjrTmiqg7Nvs7oy6QQKI2HqUF0jP33hc9buKicqwk7P+CjyK+pafj5+QDL3zBjByIxEC6sUETmKKx4ufQwSM2HF7+H9e2HIRZA+0urK5BQpjIShA1UNXLFwDfsOmpPONTT5WoLIzDEZ3HLeYIb1UgcxEQlANhucNxdKNsOWN+Dfv4Lr/gUOfZ0FM/3XCwM+n8FreftZvqmIL/ZXUlrtbvnZPTNGcMHwdIoq6+mZEKX+ICISHKb+DnZ+APlr4cP/gwvutroiOQUKIyFub3ktP38xj037K1u22W0wpGc8Zw1MZtaZWcREOumboj4hIhJEevSHmX+BZT+CNY/BaVdBVMKRQdIkqCiMhLB1eyr4yd/XU1HbSLzLyY/PGcjkwamM6J2gwclEJPiNvBzengs1B2DBBHPb2XPgwnusrUv8pjASgj7eVc7CD3eycnsphgGn9Unk6evGkZ4QZXVpIiKdx2aDSb+AVX8Cnw/clbD6ISjfDv3P0aO/QcRmGIZx8t2sVVVVRWJiIpWVlSQkJFhdTkBbsa2UHz33WcsgZTPHZPD7K04jJlK5U0RC3Hu/hdV/PvL6hreh12ngirOspHDX3u9vfUOFiEaPjzte3sgbGwsBGNE7gQU/GEt/dUgVkXBx/t3QJweW/tB8/ey3ISYVfvIBJPW1tjY5Ic3vHuS2H6jmgbe/Zvz/vccbGwux2eDyM/rwwk0TFEREJLzY7ZA9A777pDl8vCMS6spg2Y3gabS6OjkB3aYJMk1eH6u3l+Gw29haXM38t7Zw1LQxPPi9MVyZk2ldgSIigeLgHnjiHLMvycjvwhWLwK7O+91Jt2lC0Krtpcx77ctWI6U2G5mRwND0eC49XY+1iYgA5uO/33sWXpwFX70GvUbDlDlWVyVtUBgJEsWVDfxs8edUN3gAiIl0MHFgChMHpXBhdrpuyYiItGXwBXDx/5lz2bx/L9id5hM4NpvVlclRFEaCgGEYzH5hPdUNHgb3jOP1WyYTe3hyOxEROYmRl8E7c8Hngdz/Z066lzXe6qrkKOrAGgQ+2FrChoJDRDrsPHPdmcS5nAoiIiLtFdcTfvjKkdf/vAU+fgLqKqyrSVpRGAlQhmGwdmc5v3ppIz9+fj0AV+T00bDtIiIdMfBb8MNXwRkFZdvg7f+BPw2Dl6+Hr14Hd43FBYY3PU0TgKoamvjlkjw+2Frasu07o3vz+ytGE+fSnTURkQ6rq4BNyyDveSjedGS7wwUDz4Xhl8Cw6ebVFDll7f3+VhgJQA++s5XHPtiBy2nn8rF9uDInk5x+yVaXJSISWoo2whcvwdf/hoO7j2x3RsM1r0G/idbVFiL0aG8Q+7LQnGF33iXZXDuxv7XFiIiEqt5jzGXq/VD6tRlKvnwFSjbD8jvh5hUal6SbqM9IgNldVsvHu8oBGNUn0eJqRETCgM0GPbPhnDvgun9BVCIc2ARb37K6srChMBJADtY2csOzn9LQ5OOMvkmMyUyyuiQRkfASmwI515vrHy+AwO/JEBIURgLAym2l/Gzxes7+w3/YU15Hn6RonrpmHA67Ht8VEel2Z95kPnWz9yPIe8HqasKCwojFvtxfyfXPfsryTcXUNnrJSo7m2RvOJC3eZXVpIiLhKakvfOsuc/2zp62tJUyoA6uFvtxfyczHVuMzICkmgr/dMJ7T+iRi1xURERFr9Ztstg2HLC0jXCiMWKCsxs0dL2/kw8PjiKTFu3jz52fTKzHK4spERASAiMMDTDbWWltHmFAY6UZNXh9LPyvgT+9u5WBdEwCjMxP5f98ZoSAiIhJIopPMtq4CassgNtXSckKd+ox0o7+v3ctvXv+Sg3VNOOw27r9sFG/8/GzO7K8BzUREAkpiJmScAYYXNrxodTUhT1dGuonPZ/DsGnOEvzGZiTx17TjSE3Q1REQkYI24FArz4MCXVlcS8hRGukl1g4eCinoA/vaj8STFRFpckYiInFB8htkeKrC2jjCg2zTdoMbt4ZPd5qiqURF2EqIiLK5IREROqvdosy3Mg8r91tYS4nRlpIvlbj7Azxavp8lrjuJ32el99OiuiEgwSB0GadlQugWenwmX/Ml85Nehf1B2NoWRLlLj9vDcR7v5y/vbW4LI+cN78tuZIy2uTERE2sVuhx+8DM9Og/Id8Pyl5rw1Qy6G4dNh8IXgire6ypBgM4zAH3i/vVMQBwrDMLjqybV8tucgYIaQW84bzOlZSRriXUQk2BzKhxUPmBPn1ZUd2R4ZBz96G3qdZl1tAa6939+6MtIFnv1oD5/tOUhspIPfXTaKmWMycDrUPUdEJCgl9YVLHwOfFwo+ha3/hk2vQHUhbFqmMNIJFEY6ic9n8Mrn+1i0ejdfF1cD8L1xWVw+NtPiykREpFPYHdBvorn0GgOv3gRfvATn3AmuOKurC2r653onMAyDO5d9wZ3LvuDr4mrsNjhrYDLXT+pvdWkiItIVsr8DSf3MqyOv3WyO1CodpisjneBva/bwyuf7cNht3DF1GN8/M4sesRpHREQkZEVEwyUPwYtXwdf/gn3rzFs5Qy6yurKgpCsjp8jnM1jw4U4A7pg6jJ9+a5CCiIhIOBhyIdyYC6lDoaYYFl8JufeA12N1ZUFHYeQU/b9/fklJtRu7DS45rbfV5YiISHfKzIGbV8L4n5ivP3oY3rzV0pKCkcLIKVi7s5zFn+Rjs8H8y0+jb0qM1SWJiEh3i4iG6X+ESx83X29dbm09QUhh5BQsW78PgFnjsph1Zl+LqxEREUsNudhs6w+Bx21pKcFGYaSDDMNgxbZSAKbp9oyIiMSmQnQyYMCBr6yuJqgojHTQ4x/soKzG7CuS3VvDAYuIhD2bDZIHmOs1B6ytJch0KIwsWLCAAQMGEBUVRU5ODqtWrTrh/m63m3nz5tGvXz9cLheDBg3imWee6VDBgeCLfYd4KHcbAHdNG07P+CiLKxIRkYDgcJmtu9raOoKM32Fk6dKl3HbbbcybN4+8vDymTJnCtGnTyM/PP+4xV111Fe+//z6LFi1i69atLFmyhOHDh59S4VZavqkYnwEXj0znJ+cMsrocEREJFM1Dw792M/xfH9iwxNp6goTfE+VNmDCBsWPHsnDhwpZt2dnZXHbZZcyfP/+Y/d9++22+//3vs2vXLpKTkztUZCBNlFfj9jDqnncA+N2lI7lmYn9L6xERkQCydy08dwkYXvN1YhbcugEc4TnGaHu/v/26MtLY2Mj69euZOnVqq+1Tp05lzZo1bR7zxhtvMG7cOB544AH69OnD0KFDueOOO6ivrz/u57jdbqqqqlotgSJ3c3HL+qg+iRZWIiIiAaffRPjvXfDLjRCTApUFkN/296Mc4VcYKSsrw+v1kp6e3mp7eno6xcXFbR6za9cuVq9ezZdffslrr73Gww8/zLJly7jllluO+znz588nMTGxZcnKyvKnzC5hGAbvflXMHS9/AcDwXvGc0beHxVWJiEjAiU6CHv0ha4L5uuRrK6sJCh3qwGqz2Vq9NgzjmG3NfD4fNpuNxYsXM378eKZPn85DDz3Ec889d9yrI3PnzqWysrJlKSgo6EiZneqdr4r5yd/X4/UZ9Ix38dLsiVaXJCIigaznCLMt/NzaOoKAX2EkNTUVh8NxzFWQkpKSY66WNOvduzd9+vQhMfHILY3s7GwMw2Dfvn1tHuNyuUhISGi1WG3jvkoAMntE89otk0mIirC4IhERCWj9J5vtjvfB22RtLQHOrzASGRlJTk4Oubm5rbbn5uYyadKkNo+ZPHkyhYWF1NTUtGzbtm0bdrudzMzMDpTc/X73r80sPDwZ3vWT+tMnKdriikREJOD1Oxti06C2BDb/0+pqAprft2nmzJnD008/zTPPPMOWLVu4/fbbyc/PZ/bs2YB5i+Xaa69t2f/qq68mJSWFG264gc2bN7Ny5UruvPNOfvSjHxEdHfhf6nvLa1m0ejcAafEuLh8bHAFKREQs5oyEcTea6+/+xhwmXtrk97NGs2bNory8nPvuu4+ioiJGjRrF8uXL6devHwBFRUWtxhyJi4sjNzeXX/ziF4wbN46UlBSuuuoq7r///s77LbrQUyt3tax/cMe3iHOF5+NZIiLSAWffBptegopdsPEfcNZsqysKSH6PM2IFq8YZOVTXyLj738PjM3jxxxOYNCi12z5bRERCxDvzYO1jMPHncPH/Wl1Nt+qScUbCSaPHxw+e/gSPzyCzRzRnDUixuiQREQlGaYdHHN/8hjqyHofCyHG89WURXxVW4bTbePKaHOz2th9dFhEROaHTroTYnlCZD7tWWF1NQFIYaYNhGPz+LXOQmitzMhmZoZFWRUSkgyKioefhqyN1ZdbWEqAURtrwdXE1RZUNRDrt/PqSbKvLERGRYJcyxGzf+h/Yr0HQvklhpA0vfLwXgLMGpmhwMxEROXXfugsi46HhEPz9MvB5ra4ooCiMtOGDr0sAuHq89XPiiIhICIjrCZc/aa43VKoj6zcojHzD9gPVFFY2EOGwMWmwHuUVEZFOMuRicB3ug7j3I2trCTAKI9/w1pfmvDvnDEnTLRoREek8DieM+b65/uUr1tYSYBRGjlJe4+bNjYUAjMlKsrYYEREJPc1P1dSUWFtHgFEYOcqP/raO7SXmhH4KIyIi0ulSh5rtjlxNnncUhZHDvD6D7QeqAZg7bTjnDk2zuCIREQk5/c+G8Teb6//8uSbPO0xh5LCdpTXUNXqJjXRw49kDrC5HRERC1cX/B2nZ4K6Cz/5qdTUBQWHksI0FhwAYkh6P06HTIiIiXcThNGfzBfj8eQj8+Wq7nL51D1u13Ryid8oQPc4rIiJdLHumOQjaoXyNyIrCCAA+n8HqHc1hRH1FRESki0XGwMBzzfVdH1hbSwBQGAE2F1VRUdtIbKSDM/omWV2OiIiEg8EXmO3Gf4DPZ20tFlMYAVZuLwVg4qBUItRfREREusNp3zNv1ZRvh13/sboaS+mbF1i1zbxFc85Q9RcREZFu4oqHM35orn/ypLW1WCzsw0hdo4d1eysA9RcREZFulnO92e76ELweKyuxVNiHkU92VdDkNcjsEU3/lBiryxERkXCSOhSc0eBthEN7ra7GMmEfRj7eVQ6Yj/TabDaLqxERkbBit0N8L3M9jOerCfswsrPUnItmRO8EiysREZGw1KO/2e5eYWkZVgr7MLKnvA6A/qmxFlciIiJhqbkT67pnwdtkbS0WCesw4vUZ5DeHkRSFERERsUD2TIjtCTXFsOVNq6uxRFiHkcJD9TR6fUQ67GQkRVtdjoiIhCNnJIy63Fzf95m1tVgkrMPIpv2VAGQlR+Owq/OqiIhYJCbFbEu/DsuJ8xRGgDP7J1tciYiIhLVh08HuhJ3/gU0vW11NtwvrMFJe4wYgK1nji4iIiIV6jYKz55jrG5dYW4sFwjqM1LjN0e6SYyMtrkRERMLekIvMtmSLtXVYwGl1AVZa8IMcGpq8VpchIiICacPNtroI6iogJny6EIT1lRGAqAgHUREOq8sQEZFwF5UAiX3N9TC7OhL2YURERCRgpI8w25LN1tbRzRRGREREAkVPhRERERGxUsogsz2Ub20d3UxhREREJFBEHp6apDAP8j+2tpZupDAiIiISKAZfCMmDoK4cnp0O65+zuqJuoTAiIiISKFzx8JMP4bTvgeGF5XdC2Q6rq+pyCiMiIiKBJCoBLv8rDLoAvI3w3j1WV9TlFEZEREQCjc0GU+8317e9DdUHrK2niymMiIiIBKL0EZAxFnweePl68LitrqjLKIyIiIgEqssWgisB8tfAmkesrqbLKIyIiIgEqp7D4aJ7zfX/3A9FX1hbTxdRGBEREQlkQ799ZH3lH62rowspjIiIiASyhAz4zp/N9d0rwRd6s80rjIiIiAS6M64FVyI0HILNr1tdTadTGBEREQl0DicMm2auL/sRPH+pOWR8iFAYERERCQbTH4AJs8EeAbs+hKcvhIrdVlfVKRRGREREgkFUIkz7A/xiHcT2NMcfqdpvdVWdQmFEREQkmPToDymDzfV1z4JhWFpOZ1AYERERCTYX3gM2B3y5DLa9Y3U1p0xhREREJNj0PQvOvMlcD4GnaxRGREREglH2DLPd/m7Qjz2iMCIiIhKM+p5ldmqtKw/6x3wVRkRERIKRIwJ6jTbXD+6xtJRTpTAiIiISrGLTzLamxNo6TpHCiIiISLCK62m2NcXW1nGKFEZERESCVdows9233to6TlGHwsiCBQsYMGAAUVFR5OTksGrVqnYd99FHH+F0Ojn99NM78rEiIiJytH6TzXb/+qAe/MzvMLJ06VJuu+025s2bR15eHlOmTGHatGnk5+ef8LjKykquvfZaLrjggg4XKyIiIkfpMQBsdvDUQ+U+q6vpML/DyEMPPcSNN97ITTfdRHZ2Ng8//DBZWVksXLjwhMfdfPPNXH311UycOLHDxYqIiMhRnJGQMdZc3/KGtbWcAr/CSGNjI+vXr2fq1Kmttk+dOpU1a9Yc97hnn32WnTt3cs8997Trc9xuN1VVVa0WERERacOY75vt1/+2to5T4FcYKSsrw+v1kp6e3mp7eno6xcVt9+Tdvn07d911F4sXL8bpdLbrc+bPn09iYmLLkpWV5U+ZIiIi4WPAOWa7//OgHYm1Qx1YbTZbq9eGYRyzDcDr9XL11Vdz7733MnTo0Ha//9y5c6msrGxZCgoKOlKmiIhI6EseaLaeemiotLaWDmrfpYrDUlNTcTgcx1wFKSkpOeZqCUB1dTXr1q0jLy+Pn//85wD4fD4Mw8DpdPLuu+9y/vnnH3Ocy+XC5XL5U5qIiEh4ckRARCw01UJdBcQkW12R3/y6MhIZGUlOTg65ubmttufm5jJp0qRj9k9ISGDTpk1s2LChZZk9ezbDhg1jw4YNTJgw4dSqFxEREUjqa7aH9lhaRkf5dWUEYM6cOVxzzTWMGzeOiRMn8tRTT5Gfn8/s2bMB8xbL/v37ef7557Hb7YwaNarV8T179iQqKuqY7SIiItJBPfpD6Rao2G11JR3idxiZNWsW5eXl3HfffRQVFTFq1CiWL19Ov379ACgqKjrpmCMiIiLSiVIGmW3hBkvL6CibYQT+kG1VVVUkJiZSWVlJQkKC1eWIiIgEll0r4PmZEN0Dbv8KImOtrgho//e35qYREREJdv0mQ1I/qD8IaxdYXY3fFEZERESCncMJF9xtrn/yBHg91tbjJ4URERGRUDDiUvM2TV0ZFHxsdTV+URgREREJBY4IyDzTXA+yp2oURkREREJF9OEBz+orrK3DTwojIiIioSK6h9nWlVtbh58URkREREJFj/5mW/K1pWX4S2FEREQkVGScbrbFmywtw18KIyIiIqHCFW+21YXgbbK2Fj8ojIiIiISKlMFH1quLravDTwojIiIiocLpgrhe5nrZNmtr8YPCiIiISCjJ/o7Z5v3d2jr8oDAiIiISSkZebrb71ltbhx8URkREREJJ2nCzrcyHxjpra2knhREREZFQEptyZCTWIOk3ojAiIiISanqPMdv84JgwT2FEREQk1Aw812zz11hbRzspjIiIiISa5n4jFbusraOdFEZERERCTc9ssy3ZAu4aa2tpB4URERGRUNOjPyRmgc8DRRusruakFEZERERCUfIAs63cb20d7aAwIiIiEoriM8y2utDaOtpBYURERCQUxR+eo6aqyNo62kFhREREJBQlHL4yUqXbNCIiImKF1CFmW/SFtXW0g8KIiIhIKMocD3anOUdN8ZdWV3NCCiMiIiKhyBUHwy8x19c/a20tJ6EwIiIiEqqGHQ4jAT5hnsKIiIhIqIqIMluP29o6TkJhREREJFTFpJptbZm1dZyEwoiIiEioiks325oSa+s4CYURERGRUBV/OIw0VkNjrbW1nIDCiIiISKiKjIOIGHO95oC1tZyAwoiIiEiostkgrqe5HsC3ahRGREREQllLvxFdGRERERErNF8ZqVYYEREREStEJZptYZ61dZyAwoiIiEgo83nNdssb1tZxAgojIiIioWzYdLNtfqomACmMiIiIhLKMM8y2odLaOk5AYURERCSURSeZrdcdsAOfKYyIiIiEssg4cwGoKrK2luNQGBEREQllNhsk9DHXq/ZZW8txKIyIiIiEusRMsy352to6jkNhREREJNQNvtBsv/iHtXUch8KIiIhIqBt1hdkW5oG7xtpa2qAwIiIiEuri04/MUVO21dpa2qAwIiIiEg6aO7EG4Oy9CiMiIiLhIDbNbBVGRERExBLNYaS21No62qAwIiIiEg5iU822tszaOtqgMCIiIhIOmsNIXbm1dbRBYURERCQcRCWabQBOmKcwIiIiEg4URkRERMRS0clmW6unaURERMQKzfPTVO4Hn8/aWr6hQ2FkwYIFDBgwgKioKHJycli1atVx93311Ve56KKLSEtLIyEhgYkTJ/LOO+90uGARERHpgIQ+gA28bqgLrCdq/A4jS5cu5bbbbmPevHnk5eUxZcoUpk2bRn5+fpv7r1y5kosuuojly5ezfv16zjvvPGbMmEFeXt4pFy8iIiLt5IyE+N7m+qECa2v5BpthGIY/B0yYMIGxY8eycOHClm3Z2dlcdtllzJ8/v13vMXLkSGbNmsXdd9/drv2rqqpITEyksrKShIQEf8oVERGRZoumQsEncOkCOOMHXf5x7f3+9uvKSGNjI+vXr2fq1Kmttk+dOpU1a9a06z18Ph/V1dUkJycfdx+3201VVVWrRURERE7RwG+Z7aaXLS3jm/wKI2VlZXi9XtLT01ttT09Pp7i4uF3v8ac//Yna2lquuuqq4+4zf/58EhMTW5asrCx/yhQREZG2DP+O2RZtsLSMb+pQB1abzdbqtWEYx2xry5IlS/jtb3/L0qVL6dmz53H3mzt3LpWVlS1LQUFg3dsSEREJSs1jjTTWBtQTNU5/dk5NTcXhcBxzFaSkpOSYqyXftHTpUm688UZefvllLrzwwhPu63K5cLlc/pQmIiIiJ5OQAY5I8DZCZT706G91RYCfV0YiIyPJyckhNze31fbc3FwmTZp03OOWLFnC9ddfz4svvsgll1zSsUpFRETk1DgiIHmQuV6xy9pajuLXlRGAOXPmcM011zBu3DgmTpzIU089RX5+PrNnzwbMWyz79+/n+eefB8wgcu211/KXv/yFs846q+WqSnR0NImJiZ34q4iIiMhJxaZCKVAbOBPm+R1GZs2aRXl5Offddx9FRUWMGjWK5cuX069fPwCKiopajTny5JNP4vF4uOWWW7jllltatl933XU899xzp/4biIiISPs1z95bW2ptHUfxe5wRK2icERERkU7y7zvgs7/ClF/BBe0b76ujumScEREREQlysWlmWxs4Q8IrjIiIiIST2BSzrQucPiMKIyIiIuEkprnPiK6MiIiIiBUCsAOrwoiIiEg4ab4yUqcrIyIiImKF5g6sDZXgbbK2lsMURkRERMJJdA+wHf76D5BOrAojIiIi4cRuh+hkcz1AOrEqjIiIiISbAOvEqjAiIiISbpr7jeg2jYiIiFgi5vDAZ7pNIyIiIpaIDazHexVGREREwk2AjcKqMCIiIhJuImPM1tNgbR2HKYyIiIiEG0ek2Xobra3jMIURERGRcGN3mq1GYBURERFLtFwZURgRERERK+g2jYiIiFjKEWG2CiMiIiJiCVeC2bqrrK3jMIURERGRcBNzeKK8ugpr6zhMYURERCTcRPcwW4URERERsURE86Bn9dbWcZjCiIiISLhp7sBq+MDns7YWFEZERETCT3MYgYB4okZhREREJNxExh8Za6S2xNpaUBgREREJP3Y7JPU118u2W1sLCiMiIiLhqddpZlu8ydo6UBgREREJT3HpZhsAA58pjIiIiISj5k6sHre1daAwIiIiEp5ie5ptdZG1daAwIiIiEp7Shptt4QZLywCFERERkfDUJ8dsK3ZCU4OlpSiMiIiIhKOohCPrTXXW1YHCiIiISHhyRIDdaa43WTtHjcKIiIhIuGp+vPfTpywtQ2FEREQkXE39ndl+9LClHVkVRkRERMLVqCsgc7y5XrHLsjIURkRERMKZ02V1BQojIiIiYi2FEREREbGUwoiIiIhYSmFERERELKUwIiIiIpZSGBERERFLKYyIiIiIpRRGRERExFIKIyIiImIphREREREBDMs+WWFERERELKUwIiIiIpZSGBERERFLKYyIiIiIpRRGRERExFIKIyIiImIphRERERGxlMKIiIiIWKpDYWTBggUMGDCAqKgocnJyWLVq1Qn3X7FiBTk5OURFRTFw4ECeeOKJDhUrIiIiocfvMLJ06VJuu+025s2bR15eHlOmTGHatGnk5+e3uf/u3buZPn06U6ZMIS8vj1//+tfceuutvPLKK6dcvIiIiAQ/v8PIQw89xI033shNN91EdnY2Dz/8MFlZWSxcuLDN/Z944gn69u3Lww8/THZ2NjfddBM/+tGPePDBB0+5eBEREQl+foWRxsZG1q9fz9SpU1ttnzp1KmvWrGnzmLVr1x6z/8UXX8y6detoampq8xi3201VVVWrRURERLrA6Ktgyq8gdZhlJTj92bmsrAyv10t6enqr7enp6RQXF7d5THFxcZv7ezweysrK6N279zHHzJ8/n3vvvdef0kRERKQjxl5rdQUd68Bqs9lavTYM45htJ9u/re3N5s6dS2VlZctSUFDQkTJFREQkCPh1ZSQ1NRWHw3HMVZCSkpJjrn4069WrV5v7O51OUlJS2jzG5XLhcrn8KU1ERESClF9XRiIjI8nJySE3N7fV9tzcXCZNmtTmMRMnTjxm/3fffZdx48YRERHhZ7kiIiISavy+TTNnzhyefvppnnnmGbZs2cLtt99Ofn4+s2fPBsxbLNdee+T+0+zZs9m7dy9z5sxhy5YtPPPMMyxatIg77rij834LERERCVp+3aYBmDVrFuXl5dx3330UFRUxatQoli9fTr9+/QAoKipqNebIgAEDWL58ObfffjuPP/44GRkZPPLII1xxxRWd91uIiIhI0LIZzb1JA1hVVRWJiYlUVlaSkJBgdTkiIiLSDu39/tbcNCIiImIphRERERGxlMKIiIiIWEphRERERCylMCIiIiKWUhgRERERSymMiIiIiKX8HvTMCs1DoVRVVVlciYiIiLRX8/f2yYY0C4owUl1dDUBWVpbFlYiIiIi/qqurSUxMPO7Pg2IEVp/PR2FhIfHx8dhstk5736qqKrKysigoKNDIrl1M57p76Dx3D53n7qHz3D268jwbhkF1dTUZGRnY7cfvGRIUV0bsdjuZmZld9v4JCQn6g95NdK67h85z99B57h46z92jq87zia6INFMHVhEREbGUwoiIiIhYKqzDiMvl4p577sHlclldSsjTue4eOs/dQ+e5e+g8d49AOM9B0YFVREREQldYXxkRERER6ymMiIiIiKUURkRERMRSCiMiIiJiqZAPIwsWLGDAgAFERUWRk5PDqlWrTrj/ihUryMnJISoqioEDB/LEE090U6XBzZ/z/Oqrr3LRRReRlpZGQkICEydO5J133unGaoObv3+mm3300Uc4nU5OP/30ri0wRPh7nt1uN/PmzaNfv364XC4GDRrEM888003VBi9/z/PixYsZM2YMMTEx9O7dmxtuuIHy8vJuqjY4rVy5khkzZpCRkYHNZuP1118/6THd/l1ohLB//OMfRkREhPHXv/7V2Lx5s/HLX/7SiI2NNfbu3dvm/rt27TJiYmKMX/7yl8bmzZuNv/71r0ZERISxbNmybq48uPh7nn/5y18af/jDH4xPP/3U2LZtmzF37lwjIiLC+Pzzz7u58uDj77ludujQIWPgwIHG1KlTjTFjxnRPsUGsI+d55syZxoQJE4zc3Fxj9+7dxieffGJ89NFH3Vh18PH3PK9atcqw2+3GX/7yF2PXrl3GqlWrjJEjRxqXXXZZN1ceXJYvX27MmzfPeOWVVwzAeO211064vxXfhSEdRsaPH2/Mnj271bbhw4cbd911V5v7//d//7cxfPjwVttuvvlm46yzzuqyGkOBv+e5LSNGjDDuvffezi4t5HT0XM+aNcv4zW9+Y9xzzz0KI+3g73l+6623jMTERKO8vLw7ygsZ/p7nP/7xj8bAgQNbbXvkkUeMzMzMLqsx1LQnjFjxXRiyt2kaGxtZv349U6dObbV96tSprFmzps1j1q5de8z+F198MevWraOpqanLag1mHTnP3+Tz+aiuriY5ObkrSgwZHT3Xzz77LDt37uSee+7p6hJDQkfO8xtvvMG4ceN44IEH6NOnD0OHDuWOO+6gvr6+O0oOSh05z5MmTWLfvn0sX74cwzA4cOAAy5Yt45JLLumOksOGFd+FQTFRXkeUlZXh9XpJT09vtT09PZ3i4uI2jykuLm5zf4/HQ1lZGb179+6yeoNVR87zN/3pT3+itraWq666qitKDBkdOdfbt2/nrrvuYtWqVTidIfu/e6fqyHnetWsXq1evJioqitdee42ysjJ+9rOfUVFRoX4jx9GR8zxp0iQWL17MrFmzaGhowOPxMHPmTB599NHuKDlsWPFdGLJXRprZbLZWrw3DOGbbyfZva7u05u95brZkyRJ++9vfsnTpUnr27NlV5YWU9p5rr9fL1Vdfzb333svQoUO7q7yQ4c+faZ/Ph81mY/HixYwfP57p06fz0EMP8dxzz+nqyEn4c543b97Mrbfeyt1338369et5++232b17N7Nnz+6OUsNKd38Xhuw/lVJTU3E4HMck7JKSkmMSX7NevXq1ub/T6SQlJaXLag1mHTnPzZYuXcqNN97Iyy+/zIUXXtiVZYYEf891dXU169atIy8vj5///OeA+aVpGAZOp5N3332X888/v1tqDyYd+TPdu3dv+vTp02qq9OzsbAzDYN++fQwZMqRLaw5GHTnP8+fPZ/Lkydx5550AjB49mtjYWKZMmcL999+vq9edxIrvwpC9MhIZGUlOTg65ubmttufm5jJp0qQ2j5k4ceIx+7/77ruMGzeOiIiILqs1mHXkPIN5ReT666/nxRdf1P3edvL3XCckJLBp0yY2bNjQssyePZthw4axYcMGJkyY0F2lB5WO/JmePHkyhYWF1NTUtGzbtm0bdrudzMzMLq03WHXkPNfV1WG3t/7acjgcwJF/ucups+S7sMu6xgaA5sfGFi1aZGzevNm47bbbjNjYWGPPnj2GYRjGXXfdZVxzzTUt+zc/znT77bcbmzdvNhYtWqRHe9vB3/P84osvGk6n03j88ceNoqKiluXQoUNW/QpBw99z/U16mqZ9/D3P1dXVRmZmpnHllVcaX331lbFixQpjyJAhxk033WTVrxAU/D3Pzz77rOF0Oo0FCxYYO3fuNFavXm2MGzfOGD9+vFW/QlCorq428vLyjLy8PAMwHnroISMvL6/lEepA+C4M6TBiGIbx+OOPG/369TMiIyONsWPHGitWrGj52XXXXWece+65rfb/8MMPjTPOOMOIjIw0+vfvbyxcuLCbKw5O/pznc8891wCOWa677rruLzwI+ftn+mgKI+3n73nesmWLceGFFxrR0dFGZmamMWfOHKOurq6bqw4+/p7nRx55xBgxYoQRHR1t9O7d2/jBD35g7Nu3r5urDi4ffPDBCf/ODYTvQpth6NqWiIiIWCdk+4yIiIhIcFAYEREREUspjIiIiIilFEZERETEUgojIiIiYimFEREREbGUwoiIiIhYSmFERERELKUwIiIiIpZSGBERERFLKYyIiIiIpRRGRERExFL/Hxq6Z02FD3wVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.plot(thresh,prec[:-1])\n",
    "plt.plot(thresh,rec[:-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Precision and recall curve\n",
    "\n",
    "It is customary to plot the precision as a function of recall into the so called precision and recall curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0, 1.0)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi4AAAGiCAYAAADA0E3hAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAt+klEQVR4nO3de3gUdZ7v8U+nO+lOAgkSINxDUFCEI2oyIMHoeouLLg57Zg7McVZEcdaMOghZHUXmiHrck9HZYRwv4Kgg+iwqq6Iy80Qls47IbXRgAiLECzfDJSEmkKRz66S76/yRpE1IAunQSeeXvF/PU0+6fv2rqm9ThPrwq+oqm2VZlgAAAAwQEe4CAAAAOorgAgAAjEFwAQAAxiC4AAAAYxBcAACAMQguAADAGAQXAABgDIILAAAwBsEFAAAYg+ACAACMEXRw+eSTTzRz5kwNHz5cNptN77777hmX2bhxo1JSUuRyuTR27Fg9//zznakVAAD0cUEHl6qqKk2ePFnPPvtsh/ofPHhQN9xwg9LT05WXl6eHHnpICxYs0Ntvvx10sQAAoG+znc1DFm02m9555x3NmjWr3T4PPPCA1q9fr/z8/EBbZmamdu3apW3btnV20wAAoA9ydPUGtm3bpoyMjBZt119/vVauXKn6+npFRka2Wsbj8cjj8QTm/X6/Tpw4oYSEBNlstq4uGQAAhIBlWXK73Ro+fLgiIkJzWW2XB5eioiIlJia2aEtMTJTX61VJSYmGDRvWapns7Gw9+uijXV0aAADoBocPH9bIkSNDsq4uDy6SWo2SNJ2dam/0ZPHixcrKygrMl5eXa/To0Tp8+LDi4uK6rlD0aE9+8KVe3fatbr88WVnXjQ93OX1T2WFpxTTJ7pJ+uS+4ZS1LsvyS3ydZvlN+Wq3brXba/f5Tfjb1bae9xby/jf6ntLfZ399smeaT1fKnrDb6NL1/yns6w7raXc8pfdTW+6ebb2v9VsN61OkrB9DTnTNGytx8duto/vfP3rH4UFFRoVGjRql///5nt+1mujy4DB06VEVFRS3aiouL5XA4lJCQ0OYyTqdTTqezVXtcXBzBpQ9zxfRThDNGrph+/D0IF39/yWmT5JFevbbxAO9tOLj7vc2mU4NJ40Ef4WVrnE7bIZj1RTRMsjW+trXx2tbydVvvd2aZwHbPdpnG0xcdXqbp/c4sY2v8Iw52mabtBbmMzSa5i6S/vSTVHJZenHqagH2G4HxqqL08S7p2acf/qoTwMo8uDy7Tpk3TH//4xxZtGzZsUGpqapvXtwDowaIHSpGxUn2VVPJViFdukyLsks3e7GfEKfOna49oo1977Wez3ohmU/MDRnvTmd7vrj620G8LPV9VqZT3n5K3VnIfC916P18rDUyWRqRIiRNDt94OCDq4VFZWat++74eIDx48qJ07d2rgwIEaPXq0Fi9erKNHj+rVV1+V1PANomeffVZZWVn62c9+pm3btmnlypV6/fXXQ/cp0KccOVmtjV9/px+MOUcxUd1ythNNnP2kez6TSvc1HsgdjdMpr5sO8h0OEnYOhEBXiE2QFn4hVRzR6UO2rf3A2ny5I59Jr/9Eqjgqrf+F5IqX7j/Q4VNHoRD0lrZv366rrroqMN90Lcqtt96q1atXq7CwUAUFBYH3k5OTlZOTo0WLFum5557T8OHD9fTTT+tHP/pRCMpHX9J0XPvT54X60+eF+p+XjNCyOReHtaY+KX5kwwTADP0GN0yhcO41Uurt0slD0v6PpNryhtPD3Rhczuo+Lt2loqJC8fHxKi8v59qGPmzHtye15J3dOlFVp2K3R5eNHag3/nVauMsCgL7H45ayG/8D87OPGkZO/T7JFScNGhfo1hXHb8bZYYyUpHP0wcIr9KfPj+me1/LCXU6b/H5LVXVeuWu9qvR4VVPn0wXD+svpsIe7NADoGi9e3XJ+zn9KE2Z22eYILkAjv9+S2+NVeXW9ymrqVFHjVaWnXhW1XlXWNoWRerlrvXJ7Gudr61UZeO1VZZ1Xp45h/uPEoXr+lpTwfCgA6ApR/aQLZ0kHN0oRkQ3Xt9WWSfXVUun+Lt00wQW9js9vqay6Tieq6lRWU98YROpVVl2n8pp6ldfUq6yxrbymXuXVDf0qaurlD9GJ00i7TVH2CFXV+XSotCo0Kz1LlmWpus6nqjqvqjw+VXkaRoWaflbXndrmU3Xd9+/X+yzdecVYZUwcGu6PAiDcbDZp9ist2969S9q5pss3TXCBsb4trdbidbt1sqohpJRWeQJh5Wyu3IqJsis+OlJxrkj1dznU3+VQv6bXTof6OVu39XdFql9TX6dDTkeEtuwr1b+s/DQkn7XO65e7tmG0p6K2XhU1Xrlr6wOvK5req2kcIfLUN4STxuDR9Ppsr2h7ZdshgguAsCK4wDhN14sUltfq9c8K2u0XHx2pc2IiFR8dqfiYKA2IjtSApvnoSA04ta3xZ1dcj2JZlmrr/SqrqdPJqoZTUd+PBDUFkIbQ4W72uqKmIZDU1PtCVovNJsVGORTrtCu2MYg1zDvUr7EtNtBmVz+nQ18WubV66yH5Ojgk5fNbgZGbphGb5iM5lbX1qqrzNbTXft+nqq7xlFtj2LJH2PTUTy7WD8YMDNnnB2A2gguMkz5ukDKvPFfVdV4NjI1qMSXEOjUwNkoDYiIVaQ/NA73O1tfH3Tr//3ygOu/Z3zk2NsquuGajQQ2vG0Z84qIdje2RgcARCCbO70NIdKQ96LtY/unzY1q99ZCOnKzREx98KXdtfSBguGu9gdNP7sYQEsqg9d/5xSELLh6vL3CarKnups8xLrGfLhjKtxaBno7gAuO4Iu16cMYF4S7jjEYNjJbNJvktBUJLpN2m+OiGYDWgcdSnaQSovSAS1/i6n9MhR5jCWFTjdo+crNGKjzt+4V2k3dYiQLX9unHkx/X96E8/l0OvfVqg9buOyevzq7TS831IahyZaXjtU6WnXpVN1+w0XiAdeN1spKfK41Odr/3w6IqM0I5fXadYJ/8sAj0Zv6FAF0lKiNXmB67Wyao6DYiJ1DkxUYqJCn60oydIHzdYt09PVkVtfeN1Pw0Bq19j4Gh+yqm/6/sRnrM57fbnvcclSS9tPqiXNh8M1UeRJEVHNoSlhlrt+uJohWrr/ar0eFsEF8uy5PH6A4GpKQg1D0VNwchd2/Ji57RzB+nn/3BuSOsOlmVZqvP5Vd14jVPTBdgtftZ5W7xvj7DptuljNKS/K6y1A+0huABdaMSAaI0YEB3uMs5adJRdD8+8sFu3+T9GxstmU+CC4pgoe6tRm+bhI9bZcKF08xGd5qEqcMosyt5q5Oq8h3Lk9Vu645XtqvP6WwQSbye/arZ5X4nmX56sKEfHR8marg1qHo4aAkbDiFF1nVdVdT5Vexp/Nm9v4/3O1h8dadeCa8aduSMQBgQXAD3SDy8eoX8YP0SySf2cDtkjum6kalA/p4oqarX7aHm7ffo1u04oEIoaT2sFApPLociICP17Tr4sS3p351F5vP7GENJwXZC7+QXLjfNNIaW6LnTXBp3K6YhQrNOhmCi7YqMcinE2/oyyB9p3Hi7TnmMV2vj1d4EA1TwkJQ+K1f/94SQjRw3RexBcAPRY8THd8wT5NT+bqp0FZc1GcFqO2sRE2hXRweBUXefVv+fkS5J++dbnnaon0m4LXGTd9I2vptGimMZve8VENc47T/nZ/P1m/TpyfdRvPvxSe45VaMe3J7Xj25Ot3t+yr1Tz0sbovCH9O/W5gFAguADo884d3E/nDu4XknXFRDl05xVj9deDJxTXbFQmcP+fZhciNw9H/Z2Rjae2zu7aoLPxv6eMlrvWK5/favGV+Jgoh/5fTr4qPV7d+8ZOJca59MSPLtLg/s6w1Im+jeACACG2+IYJ4S6hU0aeE6PHfjipzfde2XpIXx13a8+xCu05VqG/fFWs2amjurlCgOACAOiAF+am6NODJ/TK1kPac6xC//HhV1q56aCW3nSh0s4dFJJt+P2WKuu+f/aXu7Y+cA1Qi2eF1XolSbdNH6OkhNiQbBvmILgAAM4oKSFWSQmx2tV4AW+x26Nit0d/3FWoaWMTVFPvC4SKhnvu1AcCh/uU+UpP222VHm/QdT1y00RJ3z+Lq7LZV9Mrm4WdU7/GPnF4vG6eOjrUf0zoBgQXAECHPXTDBF0zYYj+9Hmh1v39qN7cflj/tf1whx8H0RFR9ojG54F9f11Qf1dk43PBHMovdOuzQyf09o4j2rCnKPBNrWBLmDFpqAbERKqm3tfqG17fh536QNDqF+XQHeljFR0VnmuQ0IDgAgDosFinQ1dfkKiaOr/W/f1oi/vERNjU8MDRxoDRdLPCwAXIzR9KGnhYacMdopvPn+ni5Ld3HNFnh040jNqcMkrTvIYWF0I3e0jqyi0HZVlS+pN/UXVdcIEnaVCsbpo8PKg/M4QWwQUAELQbLxqmC4f/g7w+f+ARFZ15DlZnzLpkhEYnxKje6z8lmETKFRlxxhr+8lWx9n9X1eLUVETj/YKaB63mgWfzvhIdOVmj6k6czkJoEVwAAJ2SPCg8F8baI2xn9eDN9+65XPuLK78fhXGdOXTd8crfdORkTae3idAhuAAA+pR+TocmjxoQ7jLQSeF51CwAAEAnEFwAAIAxCC4AAMAYBBcAADpo99Fyvb+7UNV1fLsoXLg4FwCAM2r4xtGaTwu05tMCLbj6PGVlnB+4Y29ZTb1OVtWpvKZeZdX1KqupU1l1vTxev3586UiNTogJc/29B8EFAIAzmJ06UkfLanSiyqPjFR6t2nJIr312WOU1dar3nf4OdodKqvTgjAs0NM6liIiuv89Nb0dwAQDgDDImDlXGxKF6a8cR3ffmrlbPVoqyR2hATGTDFB2lATGRKnZ7tPNwmdbvOqb1u47p2glD9NKtPwjjp+gdCC4AAHTQP18yQqPOiZbPsgIB5ZyYqDbv2JtXcFJzXvir6rx+SdIXRyvCUXKvQ3ABAKCD7BE2TR2b0KG+l4w+R7seztAXx8r1v57f1sWV9R18qwgAgC4SHWVXdCRPkw4lggsAADAGwQUAABiD4AIAAIxBcAEAoBucrK7T/NV/07INX4W7FKMRXAAA6EL9nA1f4PV4/frvL4v19Ef7VFLpCXNV5iK4AADQhcYMitXv5kzW/defr6Zbvfj8p7/bLtpHcAEAoIv98yUjdfdV58lu45b/Z4vgAgAAjEFwAQCgm52srtNXRW5t/qZEH39VLI/XF+6SQmfjE9KvR0s7X+uS1XPLfwAAutk/PrWpxfz915+vf75khIbFu1o988gYA0Y3/KyvluolfZUjjf2nkG+G4AIAQDeZOCJeuw6XSZIGxkbpRFWdJOk3H36l33z4lW6fnqyHZ16o2nqfiis8+q6yVt+5PSp2e/Sd26Paep/mThujUQNjwvgp2pH+b9LYq6Qv/yRtfbrLNkNwAQCgm6z7eZq+c3uU0C9KkfYIvZt3VAvX7gy8v2rLQb25/bDcHm+76/D6LS2dObEbqg2SPVIaPVU6vrtLN0NwAQCgm9gjbBoa7wrMz7pkhK4cP1hfHCvXLSs/k6RAaHE6IjQkzqkh/V0a3M+po2U12n20XLX1veh6mE4guAAAEEbnxEbp8vMG6b27p6u6zqchcU4N7u9Uf6ejxfUuz370jXYfLQ9jpT0DwQUAgDCz2WyaPGrAWa+nyuOVz7IU54o8+6J6KIILAAAGOfBdlV7adEDFbo+KK2p1vMKj4+5aFVd4VOnxyh5h0yu3TdHUsQMVaQ/jXU+ObJfe+GnIV0twAQDAAE2njT49eEKfHjzRbj+f39K/rPxU9gibls68UHOnjemmChs54xp+ugulkmMhXz3BBQAAA1w/cag+/qpYNptNiXEuDenvVGKcs/G1S0PinPqv7Yf1h40HJDUEmK37SlsEl3qfX9+5PXI6IpTQz9k1hU6YKV2fLZ08JG16PuSrt1mW1eOf9FRRUaH4+HiVl5crLi4u3OUAANAj+fyW8gsr9OGeIj3z0T4Nj3dpwrA4FTWeUiqt8siypAibtGHRFTpvSP+uK+bYTlU8c4Xif+0O6fGbERcAAHoJe4RNk0bE66sityTpWHmtjpXXturnt6R9xVVdG1y6CMEFAIBe5h8nDdWxshr5LEtD41wNp5MaTyv966vb9feCMj383hf69fv5+u3syUpJGhjukjuM4AIAQC8T63ToF9eMa/O9xLiGG+AVuz2SpNy9xUYFF54ODQBAH/L4rEl69uZLlHFhYrhL6RSCCwAAfUhCP6f+6aLhGt0TH9TYAQQXAABgDIILAAAwBsEFAAAYg+ACAACMQXABAADGILgAAABjEFwAAIAxCC4AAMAYBBcAAGCMTgWX5cuXKzk5WS6XSykpKdq0adNp+69Zs0aTJ09WTEyMhg0bpttuu02lpaWdKhgAAPRdQQeXtWvXauHChVqyZIny8vKUnp6uGTNmqKCgoM3+mzdv1ty5czV//nzt2bNHb775pv72t7/pjjvuOOviAQBA3xJ0cFm2bJnmz5+vO+64QxMmTNBTTz2lUaNGacWKFW32/+tf/6oxY8ZowYIFSk5O1uWXX64777xT27dvP+viAQBA3xJUcKmrq9OOHTuUkZHRoj0jI0Nbt25tc5m0tDQdOXJEOTk5sixLx48f11tvvaUbb7yx3e14PB5VVFS0mAAAAIIKLiUlJfL5fEpMbPko7MTERBUVFbW5TFpamtasWaM5c+YoKipKQ4cO1YABA/TMM8+0u53s7GzFx8cHplGjRgVTJgAA6KAPvijUzS/+VX/eezzcpXRIpy7OtdlsLeYty2rV1mTv3r1asGCBHn74Ye3YsUMffPCBDh48qMzMzHbXv3jxYpWXlwemw4cPd6ZMAADQjhinQ5J0qLRaW/eX6uWtB8NcUcc4guk8aNAg2e32VqMrxcXFrUZhmmRnZ2v69Om6//77JUkXXXSRYmNjlZ6erscff1zDhg1rtYzT6ZTT6QymNAAAEIRbLktShE3a/12V/rjrmHx+K9wldUhQIy5RUVFKSUlRbm5ui/bc3FylpaW1uUx1dbUiIlpuxm63S2oYqQEAAN1vcH+nFl47XtdPbBh4yC9064fPbtbT//1NmCs7vaBPFWVlZemll17SqlWrlJ+fr0WLFqmgoCBw6mfx4sWaO3duoP/MmTO1bt06rVixQgcOHNCWLVu0YMECTZkyRcOHDw/dJwEAAEHr13jKqLymXruOlOv5jftlWVaPHVwI6lSRJM2ZM0elpaV67LHHVFhYqEmTJiknJ0dJSUmSpMLCwhb3dJk3b57cbreeffZZ/du//ZsGDBigq6++Wk888UToPgUAAOiUy88bpMd+OFHfuT165qN9qq7z6X88skEjBkTrvXumyxVpD3eJLdisnhqpmqmoqFB8fLzKy8sVFxcX7nIAAOh13LX1Sn38z/J4/YG2VfNSNW5If40aGBP8Co/tVMUzVyj+1+6QHr95VhEAAFB/V6TW33O5Vs1LVZyr4YTM7au3K/3Jv+jTAz3nMT0EFwAAIEk6f2h/XX1Boi4ZfU6L9oMlVWGqqDWCCwAAaOHleT/QZ0uu0dUXDAl3Ka0QXAAAQAsRETYN6e9SRNv3lg0rggsAADAGwQUAABiD4AIAAIxBcAEAAMYguAAAAGMQXAAAwGn9Ob9Yv37/Sx05WR3uUoJ/VhEAAOgbImwN34f+c/5x/Tn/uOq8fj0888Kw1kRwAQAAbbplWpIqPV4Vuz3aV1ypmnpvuEviVBEAAGhb+rjBeu1nl2nWxcPDXUoAwQUAABiD4AIAAIxBcAEAAB1yvMKj93cXqthdG7YaCC4AAKBDPvqyWD9f83fd9+bnYauB4AIAAE5r6tgE9Xc51M/Z8GXk4gpGXAAAQA/1gzED9fnSDD3/LynhLoXgAgAAzszWeDO6cCO4AAAAYxBcAACAMQguAADAGAQXAABgDIILAAAIms9vhWW7BBcAABCUb4ordf6v3tfs57fJsro3wBBcAABAhwyJc0pqGG3x+i19duiEFq7dqaXvfdFtIzAEFwAA0CHjE/vr7Z9P08pbU9V0W5f3dh7TK9u+1c7DJ7ulBke3bAUAAPQKKUkDJUlL/+lCfX6kXB99Vayy6nrV+7pnxIXgAgAAgjZverIk6dplG1VWXd9t2+VUEQAAMAbBBQAAGIPgAgAAjEFwAQAAxiC4AAAAYxBcAACAMQguAADAGAQXAABgDIILAAAwBsEFAAAYg+ACAACMQXABAADGILgAAABjEFwAAIAxCC4AAMAYBBcAAGAMggsAADAGwQUAAJy1V7cd0q/e3a2SSk+XbsfRpWsHAAC9mt1mkyTl7C6SJI0d1E+3X57cZdtjxAUAAHTaXVedqyvGD9bogTGSpDqfv0u3R3ABAACd9sOLR+jV26doSvLAbtkewQUAABiD4AIAAIxBcAEAAMYguAAAAGMQXAAAgDEILgAAwBgEFwAAYAyCCwAAMAbBBQAAGIPgAgAAjEFwAQAAxiC4AAAAY3QquCxfvlzJyclyuVxKSUnRpk2bTtvf4/FoyZIlSkpKktPp1LnnnqtVq1Z1qmAAANBzWZbkrq3vsvU7gl1g7dq1WrhwoZYvX67p06frD3/4g2bMmKG9e/dq9OjRbS4ze/ZsHT9+XCtXrtR5552n4uJieb3esy4eAAD0LL/58Es98cGX+o/LLWV0wfqDDi7Lli3T/Pnzdccdd0iSnnrqKX344YdasWKFsrOzW/X/4IMPtHHjRh04cEADBzY88nrMmDFnVzUAAOhRhsa5JEl+q2H+qyJ3lwSXoE4V1dXVaceOHcrIaFlKRkaGtm7d2uYy69evV2pqqp588kmNGDFC48eP13333aeampp2t+PxeFRRUdFiAgAAPdddV52rV2+fojuvHNul2wlqxKWkpEQ+n0+JiYkt2hMTE1VUVNTmMgcOHNDmzZvlcrn0zjvvqKSkRHfddZdOnDjR7nUu2dnZevTRR4MpDQAAhFFMlENXjB+sb09Ud+l2OnVxrs1mazFvWVartiZ+v182m01r1qzRlClTdMMNN2jZsmVavXp1u6MuixcvVnl5eWA6fPhwZ8oEAAC9TFAjLoMGDZLdbm81ulJcXNxqFKbJsGHDNGLECMXHxwfaJkyYIMuydOTIEY0bN67VMk6nU06nM5jSAABAD3K0rP1LQs5GUCMuUVFRSklJUW5ubov23NxcpaWltbnM9OnTdezYMVVWVgbavv76a0VERGjkyJGdKBkAAPRUEY0nYL4t7ZpTRkGfKsrKytJLL72kVatWKT8/X4sWLVJBQYEyMzMlNZzmmTt3bqD/zTffrISEBN12223au3evPvnkE91///26/fbbFR0dHbpPAgAAwu6q84do2tgEnZ/Yv0vWH/TXoefMmaPS0lI99thjKiws1KRJk5STk6OkpCRJUmFhoQoKCgL9+/Xrp9zcXP3iF79QamqqEhISNHv2bD3++OOh+xQAAKBHGD4gWq//62Xat8srvRH69dssy7JCv9rQqqioUHx8vMrLyxUXFxfucgAAwBns27VZQ964QfG/dof0+M2zigAAgDEILgAAwBgEFwAAYAyCCwAAMAbBBQAAGIPgAgAAjEFwAQAAxiC4AAAAYxBcAACAMQguAADAGAQXAABgDIILAAAwBsEFAAAYg+ACAACMQXABAADGILgAAABjEFwAAIAxCC4AAMAYBBcAAGAMggsAADAGwQUAABiD4AIAAIxBcAEAAMYguAAAAGMQXAAAgDEILgAAwBgEFwAAYAyCCwAAMAbBBQAAGIPgAgAAjEFwAQAAxiC4AACAkBs86nzlXfrvIV8vwQUAAIRc/MDBuuT6W0O+XoILAAAwBsEFAAAYg+ACAACMQXABAADGILgAAABjEFwAAIAxCC4AAMAYBBcAAGAMggsAADAGwQUAABiD4AIAAIxBcAEAAMYguAAAAGMQXAAAgDEILgAAwBgEFwAAYAyCCwAAMAbBBQAAGIPgAgAAjEFwAQAAxiC4AAAAYxBcAACAMQguAADAGAQXAABgDIILAAAwBsEFAAAYg+ACAACMQXABAADGILgAAABjEFwAAIAxOhVcli9fruTkZLlcLqWkpGjTpk0dWm7Lli1yOBy6+OKLO7NZAADQxwUdXNauXauFCxdqyZIlysvLU3p6umbMmKGCgoLTLldeXq65c+fqmmuu6XSxAACgb7NZlmUFs8DUqVN16aWXasWKFYG2CRMmaNasWcrOzm53uZ/85CcaN26c7Ha73n33Xe3cubPdvh6PRx6PJzBfUVGhUaNGqby8XHFxccGUCwAAwqSiokLx8fEhPX4HNeJSV1enHTt2KCMjo0V7RkaGtm7d2u5yL7/8svbv36+lS5d2aDvZ2dmKj48PTKNGjQqmTAAA0EsFFVxKSkrk8/mUmJjYoj0xMVFFRUVtLvPNN9/owQcf1Jo1a+RwODq0ncWLF6u8vDwwHT58OJgyAQBAL9WxJHEKm83WYt6yrFZtkuTz+XTzzTfr0Ucf1fjx4zu8fqfTKafT2ZnSAABALxZUcBk0aJDsdnur0ZXi4uJWozCS5Ha7tX37duXl5emee+6RJPn9flmWJYfDoQ0bNujqq68+i/IBAEBfEtSpoqioKKWkpCg3N7dFe25urtLS0lr1j4uL0+7du7Vz587AlJmZqfPPP187d+7U1KlTz656AADQpwR9qigrK0u33HKLUlNTNW3aNL3wwgsqKChQZmampIbrU44ePapXX31VERERmjRpUovlhwwZIpfL1aodAADgTIIOLnPmzFFpaakee+wxFRYWatKkScrJyVFSUpIkqbCw8Iz3dAEAAOiMoO/jEg5d8T1wAADQtcJ+HxcAAIBwIrgAAABjEFwAAIAxCC4AAMAYBBcAAGAMggsAADAGwQUAABiD4AIAAIxBcAEAAMYguAAAAGMQXAAAgDEILgAAwBgEFwAAYAyCCwAAMAbBBQAAGIPgAgAAjEFwAQAAxiC4AAAAYxBcAACAMQguAADAGAQXAABgDIILAAAwBsEFAAAYg+ACAACMQXABAADGILgAAABjEFwAAIAxCC4AAMAYBBcAAGAMggsAADAGwQUAABiD4AIAAIxBcAEAAMYguAAAAGMQXAAAgDEILgAAwBgEFwAAYAyCCwAAMAbBBQAAGIPgAgAAjEFwAQAAxiC4AAAAYxBcAACAMQguAADAGAQXAABgDIILAAAwBsEFAAAYg+ACAACMQXABAADGILgAAABjEFwAAIAxCC4AAMAYBBcAAGAMggsAADAGwQUAABiD4AIAAIxBcAEAAMYguAAAAGMQXAAAgDEILgAAwBgEFwAAYAyCCwAAMAbBBQAAGKNTwWX58uVKTk6Wy+VSSkqKNm3a1G7fdevW6brrrtPgwYMVFxenadOm6cMPP+x0wQAAoO8KOrisXbtWCxcu1JIlS5SXl6f09HTNmDFDBQUFbfb/5JNPdN111yknJ0c7duzQVVddpZkzZyovL++siwcAAH2LzbIsK5gFpk6dqksvvVQrVqwItE2YMEGzZs1SdnZ2h9YxceJEzZkzRw8//HCb73s8Hnk8nsB8RUWFRo0apfLycsXFxQVTLgAACJOKigrFx8eH9Pgd1IhLXV2dduzYoYyMjBbtGRkZ2rp1a4fW4ff75Xa7NXDgwHb7ZGdnKz4+PjCNGjUqmDIBAEAvFVRwKSkpkc/nU2JiYov2xMREFRUVdWgdv/3tb1VVVaXZs2e322fx4sUqLy8PTIcPHw6mTAAA0Es5OrOQzWZrMW9ZVqu2trz++ut65JFH9N5772nIkCHt9nM6nXI6nZ0pDQAA9GJBBZdBgwbJbre3Gl0pLi5uNQpzqrVr12r+/Pl68803de211wZfKQAA6POCOlUUFRWllJQU5ebmtmjPzc1VWlpau8u9/vrrmjdvnl577TXdeOONnasUAAD0eUGfKsrKytItt9yi1NRUTZs2TS+88IIKCgqUmZkpqeH6lKNHj+rVV1+V1BBa5s6dq9///ve67LLLAqM10dHRio+PD+FHAQAAvV3QwWXOnDkqLS3VY489psLCQk2aNEk5OTlKSkqSJBUWFra4p8sf/vAHeb1e3X333br77rsD7bfeeqtWr1599p8AAAD0GUHfxyUcuuJ74AAAoGuF/T4uAAAA4URwAQAAxiC4AAAAYxBcAACAMQguAADAGAQXAABgDIILAAAwBsEFAAAYg+ACAACMQXABAADGILgAAABjEFwAAIAxCC4AAMAYBBcAAGAMggsAADAGwQUAABiD4AIAAIxBcAEAAMYguAAAAGMQXAAAgDEILgAAwBgEFwAAYAyCCwAAMAbBBQAAGIPgAgAAjEFwAQAAxiC4AAAAYxBcAACAMQguAADAGAQXAABgDIILAAAwBsEFAAAYg+ACAACMQXABAADGILgAAABjEFwAAIAxCC4AAMAYBBcAAGAMggsAADAGwQUAABiD4AIAAIxBcAEAAMYguAAAAGMQXAAAgDEILgAAwBgEFwAAYAyCCwAAMAbBBQAAGIPgAgAAjEFwAQAAxiC4AAAAYxBcAACAMQguAADAGAQXAABgDIILAAAwBsEFAAAYg+ACAACMQXABAADGILgAAABjEFwAAIAxCC4AAMAYBBcAAGAMggsAADBGp4LL8uXLlZycLJfLpZSUFG3atOm0/Tdu3KiUlBS5XC6NHTtWzz//fKeKBQAAfVvQwWXt2rVauHChlixZory8PKWnp2vGjBkqKChos//Bgwd1ww03KD09XXl5eXrooYe0YMECvf3222ddPAAA6FtslmVZwSwwdepUXXrppVqxYkWgbcKECZo1a5ays7Nb9X/ggQe0fv165efnB9oyMzO1a9cubdu2rc1teDweeTyewHx5eblGjx6tw4cPKy4uLphyAQBAmFRUVGjUqFEqKytTfHx8aFZqBcHj8Vh2u91at25di/YFCxZYV1xxRZvLpKenWwsWLGjRtm7dOsvhcFh1dXVtLrN06VJLEhMTExMTE1MvmPbv3x9M3Dgth4JQUlIin8+nxMTEFu2JiYkqKipqc5mioqI2+3u9XpWUlGjYsGGtllm8eLGysrIC82VlZUpKSlJBQUHoEhs6pSk9M/oVfuyLnoN90bOwP3qOpjMmAwcODNk6gwouTWw2W4t5y7JatZ2pf1vtTZxOp5xOZ6v2+Ph4/hL2EHFxceyLHoJ90XOwL3oW9kfPERERui8xB7WmQYMGyW63txpdKS4ubjWq0mTo0KFt9nc4HEpISAiyXAAA0JcFFVyioqKUkpKi3NzcFu25ublKS0trc5lp06a16r9hwwalpqYqMjIyyHIBAEBfFvTYTVZWll566SWtWrVK+fn5WrRokQoKCpSZmSmp4fqUuXPnBvpnZmbq22+/VVZWlvLz87Vq1SqtXLlS9913X4e36XQ6tXTp0jZPH6F7sS96DvZFz8G+6FnYHz1HV+yLoL8OLTXcgO7JJ59UYWGhJk2apN/97ne64oorJEnz5s3ToUOH9PHHHwf6b9y4UYsWLdKePXs0fPhwPfDAA4GgAwAA0FGdCi4AAADhwLOKAACAMQguAADAGAQXAABgDIILAAAwRo8JLsuXL1dycrJcLpdSUlK0adOm0/bfuHGjUlJS5HK5NHbsWD3//PPdVGnvF8y+WLduna677joNHjxYcXFxmjZtmj788MNurLZ3C/b3osmWLVvkcDh08cUXd22BfUiw+8Lj8WjJkiVKSkqS0+nUueeeq1WrVnVTtb1bsPtizZo1mjx5smJiYjRs2DDddtttKi0t7aZqe69PPvlEM2fO1PDhw2Wz2fTuu++ecZmQHLtD9tSjs/DGG29YkZGR1osvvmjt3bvXuvfee63Y2Fjr22+/bbP/gQMHrJiYGOvee++19u7da7344otWZGSk9dZbb3Vz5b1PsPvi3nvvtZ544gnrs88+s77++mtr8eLFVmRkpPX3v/+9myvvfYLdF03KysqssWPHWhkZGdbkyZO7p9herjP74qabbrKmTp1q5ebmWgcPHrQ+/fRTa8uWLd1Yde8U7L7YtGmTFRERYf3+97+3Dhw4YG3atMmaOHGiNWvWrG6uvPfJycmxlixZYr399tuWJOudd945bf9QHbt7RHCZMmWKlZmZ2aLtggsusB588ME2+//yl7+0LrjgghZtd955p3XZZZd1WY19RbD7oi0XXnih9eijj4a6tD6ns/tizpw51q9+9Str6dKlBJcQCXZfvP/++1Z8fLxVWlraHeX1KcHui9/85jfW2LFjW7Q9/fTT1siRI7usxr6oI8ElVMfusJ8qqqur044dO5SRkdGiPSMjQ1u3bm1zmW3btrXqf/3112v79u2qr6/vslp7u87si1P5/X653e6QPgm0L+rsvnj55Ze1f/9+LV26tKtL7DM6sy/Wr1+v1NRUPfnkkxoxYoTGjx+v++67TzU1Nd1Rcq/VmX2RlpamI0eOKCcnR5Zl6fjx43rrrbd04403dkfJaCZUx+5OPR06lEpKSuTz+Vo9pDExMbHVwxmbFBUVtdnf6/WqpKREw4YN67J6e7PO7ItT/fa3v1VVVZVmz57dFSX2GZ3ZF998840efPBBbdq0SQ5H2H+1e43O7IsDBw5o8+bNcrlceuedd1RSUqK77rpLJ06c4DqXs9CZfZGWlqY1a9Zozpw5qq2tldfr1U033aRnnnmmO0pGM6E6dod9xKWJzWZrMW9ZVqu2M/Vvqx3BC3ZfNHn99df1yCOPaO3atRoyZEhXldendHRf+Hw+3XzzzXr00Uc1fvz47iqvTwnm98Lv98tms2nNmjWaMmWKbrjhBi1btkyrV69m1CUEgtkXe/fu1YIFC/Twww9rx44d+uCDD3Tw4EEeOxMmoTh2h/2/ZYMGDZLdbm+VlouLi1slsyZDhw5ts7/D4VBCQkKX1drbdWZfNFm7dq3mz5+vN998U9dee21XltknBLsv3G63tm/frry8PN1zzz2SGg6elmXJ4XBow4YNuvrqq7ul9t6mM78Xw4YN04gRIxQfHx9omzBhgizL0pEjRzRu3Lgurbm36sy+yM7O1vTp03X//fdLki666CLFxsYqPT1djz/+OCP03ShUx+6wj7hERUUpJSVFubm5Ldpzc3OVlpbW5jLTpk1r1X/Dhg1KTU1VZGRkl9Xa23VmX0gNIy3z5s3Ta6+9xnnjEAl2X8TFxWn37t3auXNnYMrMzNT555+vnTt3aurUqd1Veq/Tmd+L6dOn69ixY6qsrAy0ff3114qIiNDIkSO7tN7erDP7orq6WhERLQ91drtd0vf/20f3CNmxO6hLebtI09fbVq5cae3du9dauHChFRsbax06dMiyLMt68MEHrVtuuSXQv+krVYsWLbL27t1rrVy5kq9Dh0iw++K1116zHA6H9dxzz1mFhYWBqaysLFwfodcIdl+cim8VhU6w+8LtdlsjR460fvzjH1t79uyxNm7caI0bN8664447wvUReo1g98XLL79sORwOa/ny5db+/futzZs3W6mpqdaUKVPC9RF6DbfbbeXl5Vl5eXmWJGvZsmVWXl5e4KvpXXXs7hHBxbIs67nnnrOSkpKsqKgo69JLL7U2btwYeO/WW2+1rrzyyhb9P/74Y+uSSy6xoqKirDFjxlgrVqzo5op7r2D2xZVXXmlJajXdeuut3V94LxTs70VzBJfQCnZf5OfnW9dee60VHR1tjRw50srKyrKqq6u7uereKdh98fTTT1sXXnihFR0dbQ0bNsz66U9/ah05cqSbq+59/vKXv5z23/+uOnbbLIuxMgAAYIawX+MCAADQUQQXAABgDIILAAAwBsEFAAAYg+ACAACMQXABAADGILgAAABjEFwAAIAxCC4AAMAYBBcAAGAMggsAADDG/wekDBd04Gh4/gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(rec,prec)\n",
    "plt.plot(rec_rbf,prec_rbf)\n",
    "plt.xlim(0,1)\n",
    "plt.ylim(0,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Average precision\n",
    "$$\n",
    "    p(r) = \\text{Pr}(\\text{Re}^{-1}(r))\n",
    "$$\n",
    "then $p(r)$ is the curve above where the $x$-axis is $r$ (recall) and the $y$ axis is precision.\n",
    "Average precision is thus just\n",
    "$$\n",
    "    \\text{AP} = \\int_0^1 p(r) dr\n",
    "$$\n",
    "or also called Area Under the Precision Recall Curve."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we call $\\mathbb{P}(g_\\alpha(X) = 1) = t$, the detection level, where $t(1) = 0$ and $t(0) = 1$. Then\n",
    "$$\n",
    "    p(t) = \\text{Pr}(\\alpha) = \\mathbb{P}(Y=1 \\mid g_\\alpha(X)=1) = Re(\\alpha) \\frac{\\mathbb{P}(Y = 1)}{\\mathbb{P}(g_\\alpha(X) = 1)}\n",
    "$$\n",
    "and\n",
    "$$\n",
    "    r(t) = \\text{Re}(\\alpha)\n",
    "$$\n",
    "then we get\n",
    "$$\n",
    "    p(t) = \\frac{\\mathbb{P}(Y = 1) r(t)}{t}\n",
    "$$\n",
    "\n",
    "This equation captures the trade-off between precision and recall as functions of the detection level.\n",
    "\n",
    "It may look like average precision is a strange quantity, and indeed it is. See for instance the review paper on the course website."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9481477402057775"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import average_precision_score\n",
    "average_precision_score(Y_test,per.predict_proba(X_test)[:,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9950172870414039"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "average_precision_score(Y_test,per_rbf.predict_proba(X_test)[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reciever Operating Characteristic (ROC)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets consider Recall for the $0$ class vs Recall for the $1$ class\n",
    "$$\n",
    "    \\text{FPR}(\\alpha) = \\mathbb{P}(g(X)=1 \\mid Y = 0) \\quad \\text{also goes by the name false positive rate}\n",
    "$$\n",
    "$$\n",
    "    \\text{Re}(\\alpha) = \\mathbb{P}(g(X)=1 \\mid Y = 1) \\quad \\text{also goes by the name true positive rate}\n",
    "$$\n",
    "\n",
    "We can plot these using sklearn as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "fpr,tpr,thresholds = roc_curve(Y_test,per.predict_proba(X_test)[:,1])\n",
    "fpr_rbf,tpr_rbf,thresholds_rbf = roc_curve(Y_test,per_rbf.predict_proba(X_test)[:,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1690fa490>]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABGIElEQVR4nO3deXxU1f3/8dedSWZCAgmEQNjCqgKKIATZqVUwFre6glpBFKy41ALVVuT7reK3LdqfWrQCagGpG0ULuFSqplbZFYgBUfY1CIkxLEkIkG3u74+bmSQkgUy2O5N5Px+PeczNnXtnPpfhkjfnnnOuYZqmiYiIiIhNHHYXICIiIqFNYURERERspTAiIiIitlIYEREREVspjIiIiIitFEZERETEVgojIiIiYiuFEREREbFVmN0FVIfH4+Hw4cM0a9YMwzDsLkdERESqwTRNcnNzadeuHQ5H1e0fQRFGDh8+TEJCgt1liIiISA0cPHiQDh06VPl6UISRZs2aAdbBREdH21yNiIiIVEdOTg4JCQm+3+NVCYow4r00Ex0drTAiIiISZM7VxUIdWEVERMRWCiMiIiJiK4URERERsZXCiIiIiNhKYURERERspTAiIiIitlIYEREREVspjIiIiIitFEZERETEVn6HkZUrV3LdddfRrl07DMPgvffeO+c+K1asIDExkYiICLp27crLL79ck1pFRESkEfI7jOTl5dGnTx9eeumlam2/b98+rr76aoYPH05qaiqPP/44Dz/8MEuWLPG7WBEREWl8/L43zahRoxg1alS1t3/55Zfp2LEjs2bNAqBnz55s3LiRZ599lptvvtnfjxcREZFGpt5vlLdu3TqSkpLKrbvqqquYP38+hYWFhIeHV9gnPz+f/Px83885OTn1UtvG92dT+H0qneOiaBvdpPSFpq1g0IMQHlEvnysiIiKl6j2MZGRkEB8fX25dfHw8RUVFZGVl0bZt2wr7zJw5kxkzZtR3abD7MwbnfgY/VvJas7Zw8WhwBsWNjUVERIJWg4ymOfPWwaZpVrrea9q0aWRnZ/seBw8erJe6drYYzktFP2djwt0w/DfWI6679eJ798MLveF0/bTKiIiIiKXe/9vfpk0bMjIyyq3LzMwkLCyMli1bVrqP2+3G7XbXd2nsaX0V83dewP3tu9F/RA9rZWxX+HAyeAoh5xB8+GtoWr5lh9Y9IHF8vdcnIiISCuo9jAwePJgPP/yw3LpPP/2U/v37V9pfpCFFuZwAnMwvKl3Z907ofRssug12J8N3SyvfOb4XtDzv7B/gbgYOZx1VKyIi0jj5HUZOnDjB7t27fT/v27ePTZs2ERsbS8eOHZk2bRqHDh3i9ddfB2DSpEm89NJLTJ06lXvvvZd169Yxf/58Fi1aVHdHUUORbuvwT+QXl3/BGQajnoFv+oGnqPxr3y2Do3th3ohzf0CrHjBpNTjtDV0iIiKBzO8wsnHjRi6//HLfz1OnTgXgrrvuYuHChaSnp5OWluZ7vUuXLixfvpwpU6Ywe/Zs2rVrx4svvhgQw3p9LSMFRRVfbNkNLn+84voWXeBfU6zLOOfy43brkk+vG+G8kbUrVkREpJEyTG9v0gCWk5NDTEwM2dnZREdH19n7Lkn5nt+8u5mfXNCK1+8ZUP0di4uAc/yxzRsJ6Zus5bAI+OUX1gidM4U3gbD67x8jIiLS0Kr7+zukx61GlVymKddnpDqqM9z357Nh63uw61NI3wxzBlW+nTsGbnwZelztXw0iIiKNRIiHEesyTV5B8Tm2rIE2vazHxbfCa6Pg5JHKt8vPhnfHQ79x4CjzdTjDIPFu63KRiIhIIxbSYSTSVdIyUlmfkbrSqjs8shvMSgKPpxiWTIDt/4INf6v4+uFNMObN2n2+MxxcUbV7DxERkXoU0mHE1zJy5miauuZwUOn8cs5wuGUBpL4BOYdL13uKYM2LsH8VPNOpdp9tOGHowzDiCahikjkRERE7hXYYaYiWkXMJc8OlEyuuP3UMvn699u9vFsPqv0DGFrjmeWhRy3AjIiJSx0I6jET6hvYW4/GYOBwB1HJw/V+t8FBbKQth+SOw+z/w79/Cja+A4YCIuhuVJCIiUhshHUa8o2kAThYW09QdYH8cdTFZ2oB7rcs+Hz8GOz8uvexzyS+sET+6dCMiIjYLsN++Dcsd5sBhgMe0hvcGXBipK4Puh4Pry09tv+ktKC6EyDPuDxTZ0tre3bRhaxQRkZDVSH/7Vo9hGES5w8g9XVQ/w3sDya2vwU2vWstrXoD//h9seafybZs0t4Yk1xVdFhIRkbMI6TACVifW3NNF5Pk78Vkw8l72GTYFImIgN7386zv+DZlbrT4myx+p289OGAhJf4SES+v2fUVEJOiFfBiJdJd2Yg0ZDqfVl+RMrXrA+w9CcUHdf+bBr2D+SDg/yWp16T267j9DRESCUsiHEe/w3jw7h/cGit6jodfNYHrq9n3zfoTP/wSpb1rT4+/6FFpfaM1QKyIiIa+SmbhCi3d4b0hcpqkOh9O6nFOXj+h28POXYNIqcLqsz1n/ir3HKSIiASPkw0jpzfJC6DKNXdpcDHd9aC1/8w7kVXG/HhERCSkhH0Z8LSO6TNMwEgZC2z5QdBo2zINTx61H4Sm7KxMREZuEfBgpnRJeLSMNwjBg4CRr+Ys/WZOwPdMJZnaAHR/bW5uIiNgi5DuwRrrVZ6TBXXQTfDkXMr4pXecpgs9mwN4vKm4f2xX63wPOkP/rKiLSKIX8v+7eWVfVMtKAwiPgvpVWAAErlPztCmuOk8ytle+zfxXcPB/CXA1Xp4iINIiQDyOR3qG9ahlpWIZROglb+0TrBn5ZOytuV5QP61+FbR/A4jth9OtWmBERkUYj5MNIVChOehaI+txW9WvdroB//AJ2fQJvj4bbF4ErquFqExGRehXyHVi9LSMn1DISuM4bAXf+E1xNYd8KePNmOJ1jd1UiIlJHQj6MRLm8LSMKIwGt8zAY+551T520dfD69XDyqN1ViYhIHQj5yzSRbm+fEV2mCXgJl1qTpr1+AxxOhddGQdfLS193RULi3dA8wbYSRUTEfyEfRtQyEmTa9oG7l8PrP4cft1uPsr5dCvf+FyJj7alPRET8FvJhxDeaRh1Yg0frnjDxM9i8yJrJ1eubd+HYPnj3LrhzaeloHRERCWghH0Z884yoA2twaZ4Al/22/LpeN8P8JNi3Ej6eBtc8a09tIiLil5DvwOqdgfVkYTEej2lzNVIr8RfBTa8CBmz4G2xcYHdFIiJSDSEfRrz3pjFNOF2kSzVBr8c1cMX/WMvLH4V9q+ytR0REzinkw0hEuAPDsJY110gjMfw30OsWa7r5N26wQsmuZGs2VxERCTgh32fEMAyiXGGcyC/iZH4xNLO7Iqk1w4CfvwRH91hDgNe/aj1cTeH8K6HHtdDlJ+B0gTsaHCGfyUVEbBXyYQQg0uXkRH4ReRre23iEN7FG1Hz9uhVKdiVDbjp8t8x6eMV2hWtnQdfLbCtVRCTUKYwAUe4wyM3X/Wkam8hYGDbZWvZ4rFaS7f+yHt6b8h3da83mmjgernzKmuFVREQalMIIVssI6M69jZrDAR0SrcfIJ6C4EAry4L//BxvmQcpCq/Xk2r9AwsDy+xoOiIi2pWwRkVCgMEJJywi6c29IcYZDk+ZwzXNw0Y3w/kPWhGlvj658+66Xw01/g6atGrRMEZFQoDBC6ZTwahkJUZ2Hwf1r4fM/wvq/QXElo272fg6v/AQuvB4wyr8W5oZLJ0Dzjg1SrohIY6MwQtmb5SmMhCxXJFz1R6vfiOkp/9qR3fDOOKufyVcvV77/qaNw/V/rv04RkUZIYYQyLSO6TCMOJ+Asv651T+vmeyl/t0JHWVk7YduH1qida1/QMGERkRpQGKH0Znm6c69Uyd0MhjxUcX3OYSuMAGxdZt0fR0RE/KL/xgFRbm+fEbWMiJ+i28FPp1nL//2DNUpHRET8ojCCWkaklgY/CJFx1pwlqW/YXY2ISNBRGEF9RqSW3M3gst9ay188AwUn7a1HRCTIKIxQOprmpEbTSE0ljreG9p7IqHrEjYiIVEphBGjqHdqrlhGpqTA3XP4/1vKaWXDqmK3liIgEE42mQdPBSx25+BZY8wJkfgfvjodWPUtfC28Cg+6Hpq1tK09EJFApjKDp4KWOOJww4vewaAzs/cJ6lHXqGIx8svJ9I2LAMCp/TUSkkVMYQS0jUocuuApumGvN2uqVfQi++QekvGY9KtP1pzD2PQUSEQlJCiNAlEstI1JHDAMuuaP8uoI8OJwKWTuq3m/vF7D9I+h5bb2WJyISiBRGgEjvpGcFRZimiaH/nUpdckXBg1+Bp4qWt8//BKuft27U131UyZT0IiKhQ6NpKG0ZMU04Xeg5x9YiNWAY4Ayv/DH0YXDHQOZW+Hap3ZWKiDQ4hRGgSXjp/0TzNAurNLQmLWDor6zlL/6kKeVFJOQojAAOh+GbhfWk7k8jdhh4f+mU8pvetrsaEZEGpTBSwjsL6wmNqBE7uJvC8KnW8oo/Q1G+vfWIiDQghZESvpYRXaYRu/SfAM3aQc73sLGKIcAiIo2QwkgJ7517NSW82CY8oswN9/4Ep3PsrUdEpIEojJSIcnv7jKhlRGzU905o0QVOZ8P/Ow+O7LG7IhGReqcwUkItIxIQnOFw1Z+s5eJ8WHwn5J+wtyYRkXqmMFLC1zKiPiNitx5Xw5St0DTemnvkg19Zk+CIiDRSNQojc+bMoUuXLkRERJCYmMiqVavOuv1bb71Fnz59iIyMpG3bttx9990cOXKkRgXXF1/LiIb2SiCIaQ+jXwdHGHy3FNbNtrsiEZF643cYWbx4MZMnT2b69OmkpqYyfPhwRo0aRVpaWqXbr169mnHjxjFhwgS+++473n33XTZs2MDEiRNrXXxdaur2hhG1jEiA6DgIfva0tZz8e9i30t56RETqid9h5Pnnn2fChAlMnDiRnj17MmvWLBISEpg7d26l23/55Zd07tyZhx9+mC5dujBs2DDuu+8+Nm7cWOvi65Lvzr26TCOB5NKJ0Od2MIvh3bsh+3u7KxIRqXN+hZGCggJSUlJISkoqtz4pKYm1a9dWus+QIUP4/vvvWb58OaZp8sMPP/DPf/6Ta665psrPyc/PJycnp9yjvkWVtIxoBlYJKIYB1/4F2lwMJ7Ng8VgoPG13VSIidcqvMJKVlUVxcTHx8fHl1sfHx5ORkVHpPkOGDOGtt95izJgxuFwu2rRpQ/PmzfnrX/9a5efMnDmTmJgY3yMhIcGfMmtELSMSsMKbwJg3rXvYHP4a/v1buysSEalTNerAahhGuZ9N06ywzmvr1q08/PDD/P73vyclJYWPP/6Yffv2MWnSpCrff9q0aWRnZ/seBw8erEmZfvHeufekhvZKIGrRGW6eDxjw9d8hZaHNBYmI1J0wfzaOi4vD6XRWaAXJzMys0FriNXPmTIYOHcqjjz4KQO/evYmKimL48OH84Q9/oG3bthX2cbvduN1uf0qrtciSob3qwCoB67wRMOJ/4bOnYPmjEH8xdEi0uyoRkVrzq2XE5XKRmJhIcnJyufXJyckMGTKk0n1OnjyJw1H+Y5xO6xe/GUBzJ6hlRILCsKnQ41ooLoA3b4TsQ3ZXJCJSa35fppk6dSrz5s1jwYIFbNu2jSlTppCWlua77DJt2jTGjRvn2/66665j6dKlzJ07l71797JmzRoefvhhBgwYQLt27eruSGpJfUYkKBgG3DAXWp5nTRn/nyfsrkhEpNb8ukwDMGbMGI4cOcJTTz1Feno6vXr1Yvny5XTq1AmA9PT0cnOOjB8/ntzcXF566SV+85vf0Lx5c6644gqeeeaZujuKOhCleUYkWEREw7Wz4O/XwrdL4Yr/sfqUiIgEKcMMpGslVcjJySEmJobs7Gyio6Pr5TP2ZeVx+bNf0MwdxpYZV9XLZ4jUqTdugj2fwaX3wjXP2l2NiEgF1f39rXvTlIgqc5kmCPKZCAybYj2nvgEnfrS3FhGRWlAYKRFZcpnGY0J+kcfmakSqofMwaN8fik7DVy/bXY2ISI0pjJRoEu70LavfiAQFwyhtHdnwNzhd/zMVi4jUB4WREk6H4QskGt4rQaP71RB3gTWyRhOhiUiQUhgpI8qt4b0SZBwOGDrZWl43G4rybS1HRKQmFEbKiHR5h/eqZUSCyMW3QnR7OJEBm/9hdzUiIn5TGCnDN/GZ+oxIMAlzweCHrOU1L4BHYVpEgovCSBlN3d4p4RVGJMj0G2fd1ffoHtj2od3ViIj4RWGkjEi3LtNIkHI3hQH3Wcur/wKaK0dEgojCSBneic/UMiJBacAvITwS0jfB3i/srkZEpNoURsrwdWDV0F4JRlEtod9d1vLqv9hbi4iIHxRGyvAO7T2pDqwSrAY/CI4w2LcCDqXYXY2ISLUojJShlhEJes0T4OLR1vLqWbaWIiJSXQojZajPiDQKQ39tPW/7ELJ22VuLiEg1KIyU4R1Nc0KjaSSYte4B3a8BTGveERGRAKcwUkZT9RmRxsJ7A73N/4Ccw/bWIiJyDgojZZT2GVEYkSCXcCl0GgaeQuueNSIiAUxhpAzfaBp1YJXGwNs6su4lWP4o5B2xtx4RkSoojJRReqM8tYxII3DeCGhzsbW8/lX4dLq99YiIVEFhpIwol/feNGoZkUbAMOCWhdD1p9bPmxfBZ0/ZWZGISKUURsqIdOuuvdLIxJ0Hdy4DVzPr51XPQfo39tYkInIGhZEyospMembqRmPSWDgcMGlV6c/v3W9fLSIilVAYKcPbMlLsMckv8thcjUgdiu0CVz9rLf/wLRzZY289IiJlKIyU4W0ZAfUbkUZowL3Q9XJreev79tYiIlKGwkgZTodBRLj1R6J+I9IoXXi99fzNYnvrEBEpQ2HkDBpRI41aj2vBcMCP22HpfXZXIyICKIxU4BtRo1lYpTFq2houvMFa3vKupooXkYCgMHIGX8uIbpYnjdWtr0HCIDCL4es37K5GRERh5EyRLqtl5Ehevs2ViNSjSydaz1//HYrVCigi9lIYOUO40/oj2flDrs2ViNSjC6+HyJaQcwh2J9tdjYiEOIWRM0Q3CQfgVIHmGZFGLMwNl/zCWt64wN5aRCTkKYycYWi3lgAcPn7K5kpE6lnieOt5VzIcO2BrKSIS2hRGztCueRMADmcrjEgj17JbyU30TKvviIiITRRGzuALI8dP21yJSAPof4/1/PUbUFxoby0iErIURs7QviSMZJ3I53ShhvdKI9f9amgaD3mZsP0ju6sRkRClMHKG5pHhNAm3hvdmZKt1RBo5Zzj0HWstqyOriNhEYeQMhmHQrnkEoE6sEiIS7wIM2LdCd/MVEVsojFTC22/kkMKIhILmHeH8JGs55TV7axGRkKQwUon26sQqoab/3dZz6ltQqL/3ItKwFEYqUTqiRi0jEiLOT4LoDnDqKGz70O5qRCTEKIxUQnONSMhxOEv6jqCOrCLS4BRGKuHtwKo+IxJS+o4FwwlpayFzm93ViEgIURipRPsyl2lM07S5GpEGEt0Wuo+yllMW2lqKiIQWhZFKtImxWkZOF3pI11wjEkq8M7JuWgQFJ+2tRURChsJIJdxhTiJd1sRnK3b+aHM1Ig2o6+XQojPkZ8N3S+2uRkRChMJIFfp0aA7AwaP636GEEIej9G6+GzXniIg0DIWRKoy8MB6AvT/m2VyJSAO75E5whMOhjZC+2e5qRCQEKIxUoWurKAD2ZSmMSIhp2gp6Xmctq3VERBqAwkgVusaVhJEjeRR7NKJGQoy3I+uWdyE/195aRKTRUxipQocWkYQ7DQqKPJqJVUJP52HQ8nwoOGEFEhGReqQwUgWnw6BTS6t1ZK8u1UioMYzS+9VsXACab0dE6pHCyFn4LtX8eMLmSkRs0Od2cLohYwsc+truakSkEVMYOYsu6sQqoSwyFnrdZC3rfjUiUo8URs6iW1xTQJdpJIR5O7J+uwROHbO3FhFptBRGzsLbMqK5RiRkdbgUWl8ERadg82K7qxGRRkph5Cy8fUYOZ5/idGGxzdWI2KBsR9aU19SRVUTqRY3CyJw5c+jSpQsREREkJiayatWqs26fn5/P9OnT6dSpE263m27durFgQeBfg46NchEdEYZpwv4jah2RENV7DIRHwY/b4a1bIC/L7opEpJHxO4wsXryYyZMnM336dFJTUxk+fDijRo0iLS2tyn1Gjx7NZ599xvz589mxYweLFi2iR48etSq8IRiGQddWJf1GdKlGQlVENFx8i7W8+z+wbra99YhIoxPm7w7PP/88EyZMYOLEiQDMmjWLTz75hLlz5zJz5swK23/88cesWLGCvXv3EhsbC0Dnzp1rV3UD6hoXxaaDxzWiRkLbyCdh7+dwPA0yvrG7GhFpZPxqGSkoKCAlJYWkpKRy65OSkli7dm2l+3zwwQf079+fP//5z7Rv354LLriARx55hFOnqp7VND8/n5ycnHIPu3jvUbNHc41IKIuMhZvmWcs/bLW3FhFpdPxqGcnKyqK4uJj4+Phy6+Pj48nIyKh0n71797J69WoiIiJYtmwZWVlZPPDAAxw9erTKfiMzZ85kxowZ/pRWb7qUDO9Vy4iEvNY9refcw9Yw3yYt7K1HRBqNGnVgNQyj3M+maVZY5+XxeDAMg7feeosBAwZw9dVX8/zzz7Nw4cIqW0emTZtGdna273Hw4MGalFkndPdekRIR0RDT0VpW64iI1CG/wkhcXBxOp7NCK0hmZmaF1hKvtm3b0r59e2JiYnzrevbsiWmafP/995Xu43a7iY6OLvewS+eS+9McP1nI0bwC2+oQCQjxF1rPmQojIlJ3/AojLpeLxMREkpOTy61PTk5myJAhle4zdOhQDh8+zIkTpX0udu7cicPhoEOHDjUouWE1cTlpFxMBwL4s9RuRENe6JIz88J29dYhIo+L3ZZqpU6cyb948FixYwLZt25gyZQppaWlMmjQJsC6xjBs3zrf9HXfcQcuWLbn77rvZunUrK1eu5NFHH+Wee+6hSZMmdXck9cg7vHePhvdKqIu/yHpWy4iI1CG/h/aOGTOGI0eO8NRTT5Genk6vXr1Yvnw5nTp1AiA9Pb3cnCNNmzYlOTmZX/3qV/Tv35+WLVsyevRo/vCHP9TdUdSzLnFRrN6dpX4jIt6Wkcxt1mysVfQVExHxh2GagT+/c05ODjExMWRnZ9vSf+S1NfuY8eFWrroonlfG9m/wzxcJGMWF8Me24CmEyVugeUe7KxKRAFbd39+6N001dInTiBoRAJzhEHeBtax+IyJSRxRGqqFbSZ+R/UdOUuwJ+IYkkfoVr06sIlK3FEaqoV3zJrjCHBQUeTh8vOqZY0VCgrffiKaFF5E6ojBSDU6HQeeWkQDs1aUaCXXeETVb39cdfEWkTiiMVJO338he3aNGQl3HQaXLByq/J5WIiD8URqrJO9eIOrFKyIuIgd63WctZO+ytRUQaBYWRaiptGVEYEaF1D+v5R4UREak9hZFq6qYb5omUauUNI9vtrUNEGgWFkWrqEmddpjl0/BSnCoptrkbEZq26W89Zu8Cj80FEakdhpJpio1w0jwwHYP8RtY5IiGveCcIioOg0HD9gdzUiEuQURvygfiMiJRxOiDvfWla/ERGpJYURP5ROC6/hvSLqNyIidUVhxA/eaeE18ZkIpf1G1DIiIrWkMOIHXaYRKUMtIyJSRxRG/NC1VeksrKapG+ZJiPOFkZ3g8dhbi4gENYURP3RuGYVhQM7pIlbu0j05JMS16AKOcCjMg5zv7a5GRIKYwogfIsKdRIY7AfjP1h9srkbEZs4wjagRkTqhMOKnP9/SB4CNB47ZXIlIAPB1YlW/ERGpOYURP13apQUA2zNyyDldaHM1IjZTJ1YRqQMKI35q3SyCjrGRmCakph23uxwRe2l4r4jUAYWRGujf2WodSdl/1OZKRGwWVxJGMreBRpiJSA0pjNRA/06xgPqNiNCyGxhOKDgBR/bYXY2IBCmFkRrwtoykph2nsFjzK0gIC3Nb96kBSN9kaykiErwURmrgvFZNiY4I41RhMdvSc+wuR8ReieOt58OptpYhIsFLYaQGHA6D/p1LLtXs16UaCXHt+lrPCiMiUkMKIzWU2KmkE6v6jUio84aR9M3gKba3FhEJSgojNdS/JIxs2H9U96mR0BZ3AYRHlnRi3W13NSIShBRGaqhPQnPCnQaZufl8f+yU3eWI2MfhhLbWzMS6VCMiNaEwUkMR4U56tY8BYOMBzTciIU79RkSkFhRGasF7qUadWCXkKYyISC0ojNRCYieNqBEBynRi/QaKi+ytRUSCjsJILXhH1OzMzCX7lG6aJyEsthu4mkHRKcjSfWpExD8KI7XQqpmbzi2tm+Z9nabWEQlhDge0u8Ra1qUaEfGTwkgteSc/S9GlGgl1CiMiUkMKI7Xk68SqETUS6tSJVURqSGGklrw3zdt0UDfNkxDnDSMZ30JRgb21iEhQURippa5xTWkeGc7pQg/fHdZN8ySEtegCETFQnA8/brO7GhEJIgojteRwGGXmG9GlGglhhqFLNSJSIwojdcA734humichT2FERGpAYaQOePuNbNh/TDfNk9CmMCIiNaAwUgcubh+Dy+kg60Q+aUdP2l2OiH18M7Fuhpx0e2sRkaChMFIHIsKdXNyh5KZ5mm9EQllMAoRFWMsb/mZvLSISNBRG6kjpfCMKIxLCDAO6X20tH9ljby0iEjQURupIokbUiFh6j7GeFUZEpJoURuqIN4zsyjzB8ZOa8ElCWMvzrOcju8GjiQBF5NwURupIy6ZuuraKAnTTPAlxLTqBI8y6g2/uYburEZEgoDBSh0onP1MYkRDmDIcWna3lI7ttLUVEgoPCSB3qXzL5mcKIhDzvpZqsXfbWISJBQWGkDiWWTH62+fvjFBTpWrmEMF+/EXViFZFzUxipQ13jooiNcpFf5OHbw9l2lyNin7KdWEVEzkFhpA4ZhuEbVZOiSzUSyuLOt56P6DKNiJybwkgd83Zi3aD5RiSUeVtGjqdBUb69tYhIwFMYqWPem+alHNBN8ySENY0HV1MwPXB0n93ViEiAUxipY73ax+AKc3Akr4D9R3TTPAlRhqF+IyJSbQojdcwd5qSP76Z5ulQjIUxhRESqSWGkHiSWzDeSopvmSSjzhRF1YhWRs6tRGJkzZw5dunQhIiKCxMREVq1aVa391qxZQ1hYGJdccklNPjZoeDuxvrPxoM2ViNjIN6JGc42IyNn5HUYWL17M5MmTmT59OqmpqQwfPpxRo0aRlpZ21v2ys7MZN24cI0aMqHGxwcI7vNdjwn+2/mBzNSI2adnNetZlGhE5B7/DyPPPP8+ECROYOHEiPXv2ZNasWSQkJDB37tyz7nffffdxxx13MHjw4BoXGyxaRLkY2TMegJn/3kZRsWZjlRDkvUyT9yOcOm5rKSIS2PwKIwUFBaSkpJCUlFRufVJSEmvXrq1yv9dee409e/bwxBNPVOtz8vPzycnJKfcINs+P6UNslIs9P+Zx3xsp/JBz2u6SRBqWuxk0bWMt61KNiJyFX2EkKyuL4uJi4uPjy62Pj48nIyOj0n127drFY489xltvvUVYWFi1PmfmzJnExMT4HgkJCf6UGRCiI8KZPNK6Zv7Z9kwm/n2j5h2R0KNOrCJSDTXqwGoYRrmfTdOssA6guLiYO+64gxkzZnDBBRdU+/2nTZtGdna273HwYHB2BL19QEfG9LeC1JZD2Wz+XverkRATp+G9InJu1WuqKBEXF4fT6azQCpKZmVmhtQQgNzeXjRs3kpqaykMPPQSAx+PBNE3CwsL49NNPueKKKyrs53a7cbvd/pQWkMKdDp65pTeFxR6Wph7i7a8OcElCc7vLEmk4mmtERKrBr5YRl8tFYmIiycnJ5dYnJyczZMiQCttHR0ezZcsWNm3a5HtMmjSJ7t27s2nTJgYOHFi76oPE7QM7AvDh5nRyThfaXI1IA2pZMrw3S2FERKrmV8sIwNSpUxk7diz9+/dn8ODBvPrqq6SlpTFp0iTAusRy6NAhXn/9dRwOB7169Sq3f+vWrYmIiKiwvjHr36kF57duyq7ME7y/6TBjB3WyuySRhuFtGTm6BzwecGieRRGpyO9/GcaMGcOsWbN46qmnuOSSS1i5ciXLly+nUyfrF2x6evo55xwJNYZhcPsAq3Xk7a/S1JFVQkeLTuAIg8KTkJtudzUiEqAMMwh+M+bk5BATE0N2djbR0dF2l1Mjx08WMPBPn5Ff5OG9B4eq74iEjr8mWn1Gxr0PXX9qdzUi0oCq+/tbbaYNpHmki2subgvA218dsLkakQakTqwicg4KIw1IHVklJHnDiDqxikgVFEYakLcj66nCYt5PPWR3OSINQy0jInIOCiMNqGxH1rfUkVVChcKIiJyDwkgDu6lfe9xhDrZn5GpGVgkNcSVzjRw/AEX59tYiIgFJYaSBqSOrhJym8eBqCqYHju23uxoRCUAKIzZQR1YJKYYBLbtZy7pUIyKVUBixgTqySsjxTQuvu/eKSEUKIzY4syNrsUcdWaWR83Zi3fKuvXWISEBSGLFJ2Y6s3R5fztP/3m53SSL1JzLWenb4fTssEQkBCiM2aR7pYtzg0hvmzVu1l8PHT9lYkUg9atfPej511N46RCQgKYzYaPo1F7LlySQGdY2lyGMyb9U+u0sSqR/elpGTCiMiUpHCiM2aRYTzwE+t6+mL1qdxNK/A5opE6oE3jBSc0FwjIlKBwkgAGH5+HL3aR3OqsJi/r91vdzkidc8dA4bTWlbriIicQWEkABiGwf2XWa0jC9fuJy+/yOaKROqYw1HmUs0Re2sRkYCjMBIgftarDV3iosg+Vcii9Wl2lyNS95oojIhI5RRGAoTTYXDfT7oC8LdVe8kvKra5IpE6FtnSelYYEZEzKIwEkBv7tSc+2s0POfm8p5lZpbHxXqbR8F4ROYPCSABxhzmZOMxqHXllxV7NzCqNi69lRGFERMpTGAkwtw/sSEyTcPZm5THx7xs4VaDLNdJIqAOriFRBYSTANHWHMX5IZwA+3/Ejt/3tSzJzT9tblEhdUMuIiFRBYSQAPXB5Nx5JuoDoiDA2HzzOjbPXsiMj1+6yRGpHHVhFpAoKIwHIHebkoSvO5/2HhtG5ZSSHjp/ilrlrWbnzR7tLE6k5De0VkSoojASwLnFRLHtgKAM6x5KbX8TdCzfw1lcH7C5LpGZ0mUZEqqAwEuBaRLl4Y+IAbuzbnmKPyfRl3/LHj7bi0UgbCTYa2isiVVAYCQLuMCfPj+7DlJEXAPC3Vfu4/60UjbSR4OJtGSk4AYXqlC0ipRRGgoRhGPx65Pm8cNsluJwOPvnuB8a8uo7MHP2jLkEioszN8tQ6IiJlKIwEmZ9f0p637h1Ii8hwvvk+mxtmr2Fbeo7dZYmcm2GUmWtEYURESimMBKFLO8ey7IGhdI2L4nD2aW59eR1f7Mi0uyyRc9PwXhGphMJIkOocF8XSB4YwqGssJ/KLuGfhBt74UiNtJMBpeK+IVEJhJIg1j3Tx+j0DublfBzwm/O973/J//9qqe9pI4NKU8CJSCYWRIOcKc/Dsrb15JMkaaTN/9T7ueyOFvPwimysTqYT3Ms2pY/bWISIBRWGkETAMg4euOJ8Xb++LK8zBf7b9wOhX1vGDRtpIoFGfERGphMJII3J9n3YsuncgsVEuvjucww2z17D1sEbaSADRZRoRqYTCSCOT2CmW9x4YSrdWUaRnn+aWl9dy+6tfMus/O+0uTURTwotIpRRGGqGOLSNZev9QhnRrycmCYtbtPcKs/+wiI1uXbcRmukwjIpVQGGmkYiLD+fs9A3jt7ktJiG0CwNb0bJurkpDnHdqbvsnWMkQksCiMNGLhTgeXd29NYscWAPzydY2yEZtFtytdzs2wrw4RCSgKIyFgYFerabzIY/KrRakUFXtsrkhCVkz70uWcw/bVISIBRWEkBIzun8Afb+yFK8zBf7dnMuPDrZimJkYTmxWesrsCEQkQCiMhwOkw+MXATrx42yUYBrzx5QGGPfM5L/13l2ZrlYbXprf1XHjS3jpEJGAojISQn/Vqy/SrewJw6Pgpnv10J/cs3ED2yUKbK5OQ4oqyngvy7K1DRAKGwkiImTi8Kx9PHs7vftaDiHAHK3b+yPWzV7MjI9fu0iRUhEdaz7pMIyIlFEZCUI820dz/024suX8I7Zs34cCRk9w4Zw3Lt6TbXZqEgnBrqDmFahkREYvCSAi7qF0MH/5qGEPPsyZHe+Ctr/nzx9vVj0Tql+8yjfqMiIhFYSTExUa5+PvdA/jlT7oCMOeLPfz6H6kKJFJ/fJdpFEZExKIwIoQ5HTx+dU+eH90Hw4B/fZPO75Z8g0eBROqDt2VEYURESiiMiM9N/Tow9xf9cDoM/pnyPU9++J3mI5G6520Z0WUaESmhMCLl/KxXW569tTeGAa+vO8DTH29XIJG65evAqjAiIhaFEangxr4d+OMNFwPwyoq9vPTf3TZXJI2K5hkRkTMojEil7hjYkf+5xpog7bnkncxbtdfmiqTRUAdWETmDwohUaeLwrvzmygsA+MNH23j7qzSbK5JGwaVJz0SkPIUROauHrjiPSZd1A2D6e1tYlvq9zRVJ0PN1YNVlGhGxKIzIWRmGwe9+1p27BnfCNOE372zm35qpVWpDl2lE5AwKI3JOhmHwxHUXcWtiBzwmPPyPVD7fkWl3WRKsNAOriJxBYUSqxeEwePrm3lzbuy2FxSaT3khh3Z4jdpclwUgtIyJyhhqFkTlz5tClSxciIiJITExk1apVVW67dOlSrrzySlq1akV0dDSDBw/mk08+qXHBYh+nw+AvYy5hZM/W5Bd5mPD3DaQcOGZ3WRJsXAojIlKe32Fk8eLFTJ48menTp5Oamsrw4cMZNWoUaWmVj7RYuXIlV155JcuXLyclJYXLL7+c6667jtTU1FoXLw0v3OngpTv6Mfz8OE4WFDP+tfV8eyjb7rIkmJRtGfF47K1FRAKCYfo5vebAgQPp168fc+fO9a3r2bMnN9xwAzNnzqzWe1x00UWMGTOG3//+99XaPicnh5iYGLKzs4mOjvanXKknJwuKuGvBejbsP0ZslIvFvxzE+fHN7C5LgkH+CZjZ3lp+/HBpHxIRaXSq+/vbr5aRgoICUlJSSEpKKrc+KSmJtWvXVus9PB4Pubm5xMbGVrlNfn4+OTk55R4SWCJdYcwffym9O8RwNK+AX8z7igNHNFRTqsHbMgLqxCoigJ9hJCsri+LiYuLj48utj4+PJyMjo1rv8dxzz5GXl8fo0aOr3GbmzJnExMT4HgkJCf6UKQ0kOiKcv989gO7xzcjMzeeOv33FoeOayErOweGAMN2fRkRK1agDq2EY5X42TbPCusosWrSIJ598ksWLF9O6desqt5s2bRrZ2dm+x8GDB2tSpjSAFlEu3pw4kK5xURw6foo7531FZu5pu8uSQKeb5YlIGX6Fkbi4OJxOZ4VWkMzMzAqtJWdavHgxEyZM4J133mHkyJFn3dbtdhMdHV3uIYGrVTM3b04cSPvmTdiXlcfYees5lldgd1kSyDTXiIiU4VcYcblcJCYmkpycXG59cnIyQ4YMqXK/RYsWMX78eN5++22uueaamlUqAa1d8ya8fe9A4qPd7Pghl3EL1pNzutDusiRQ+UbUqJ+RiNTgMs3UqVOZN28eCxYsYNu2bUyZMoW0tDQmTZoEWJdYxo0b59t+0aJFjBs3jueee45BgwaRkZFBRkYG2dkaDtrYdGoZxVsTB9IyysWWQ9nc89oGThYU2V2WBCLdLE9EyvA7jIwZM4ZZs2bx1FNPcckll7By5UqWL19Op06dAEhPTy8358grr7xCUVERDz74IG3btvU9fv3rX9fdUUjAOK91M16fMIDoiDA2HjjGL19P4XRhsd1lSaAJ916mUcuIiNRgnhE7aJ6R4PN12jHGzvuKvIJiRvRozYu398XpMAhzGIQ5dReCkPfmLbA7GX4+G/reaXc1IlJP6mWeEZHq6texBfPHX4o7zMFn2zO56IlP6PG/H9N7xqeaQl5KL9OoA6uIoDAi9WhQ15a8MjaRFpHhvnUnC4pZviXdxqokIHgv06gDq4gAYXYXII3bT7u3Zv30keQXefjom8P8bskWUg4cI+d0Ic3cYdWan0YaIXVgFZEy1DIi9S7c6aCpO4yBXVoCsOngcXo/+SlJf1nJnh9P2Fyd2MI76Zk6sIoICiPSgDq1jGRQ19J7Eu3KPMFNc9aybs8RG6sSW/gu06jPiIgojEgDMgyDRfcOYucfRrH+8RH069ic7FOFjFvwFW98eYCiYt1OPmSoA6uIlKEwIg3KMAxcYQ5aR0fw9r2DuLZ3WwqLTf73vW+5+MlP+cW8L5n1n52s2Z2lCdMaM98MrAojIqIOrGKjiHAnL97Wl66tmvLqyj2cKixmze4jrNltXbYJcxhc1D6GHvHNcDgqdnSNdDl58PLziI1yNXTpUlsuXaYRkVIKI2Irh8Ng6pUXMHnE+ezKPMH6/UfZuP8oG/Yd5XD2aTYfPM7mg8er3D/rRD4v3Na34QqWuuHrwKowIiIKIxIgHA6D7m2a0b1NM8YOsm4tcOj4KTbsO8rBoxV/YeUXeZj9xW7e33SYu4Z0pl/HFg1dstSG5hkRkTIURiRgtW/ehPZ921f5embuad7Z+D1PfbiVJfcPwVnJpRwJUOrAKiJlqAOrBK1HkroT5XKy6eBxnv10h93liD/CNemZiJRSGJGg1To6gj/ddDEAc7/Yw3uph2yuSKrNpcs0IlJKl2kkqP38kvbsyMhlzhd7+O2Sb/jPth/KTTHvcjp46Irz6BIXZWOVUoE6sIpIGQojEvQeSerOzh9O8J9tP/CvbyrehG/HDzk8P/oSyvYoadnUrSHBdvJ2YC3OB08xOJz21iMitlIYkaDncBjM/kVf/r0lg2MnC3zrTRNm/Wcn3x7KIekvK8vtE+YweOL6i3wjd6SBeTuwgjXXiLuZfbWIiO0URqRRcIc5uaGSkTeto93837+2Ulhs+tYVe0yyTxXyv+99yyffZhDlLv1feZQ7jEmXdeOCeP1yrFdhEYABmNalGoURkZCmMCKN2rW923Ft73bl1pmmyUv/3c1zyTtZvTurwj4rd2ax7IEhJMRGVnhN6ohhWCNqCvPUiVVEFEYk9BiGwa9GnM+Q81qyLT233GtvfnmA7Rm53PXaepbeP4TmkepXUm9cJWFEnVhFQp7CiISsxE6xJHaKLbduZM94bpyzhr0/5nHv6xt5Y8JAIsLVubJe6GZ5IlJC84yIlNEmJoKFdw+gWUQYG/Yf4zfvbsbjMc+9o/hPN8sTkRIKIyJn6N6mGa+MTSTcafDRN+k8/fF2u0tqnMI1JbyIWBRGRCoxpFsc/++WPgC8unIvo19ex92vrWf1roodXqWGvBOfqWVEJOQpjIhU4Ya+7Xn0qu4ArN9/lM93/MjE1zfw3eFsmytrJLyXaQo0mkYk1CmMiJzFAz/txpL7B/PCbZcw7Lw4Thd6uO+NFI7lFZx7Zzk73SxPREoojIichWEYJHaK5eeXtGf2Hf3oGBvJ98dO8fA/UilWx9ba8c7CqnlGREKewohINcVEhvPK2ESahDtZtSuL//fJDrtLCm7qwCoiJRRGRPzQs200f76lNwAvr9jDR5XcmE+qSfOMiEgJhRERP13Xpx2//ElXAB7952Z2ZOSeYw+plDqwikgJhRGRGvjtVd0Zel5LThYUM/619WRkn7a7pOCjDqwiUkJhRKQGwpwO/np7P9pER5CefZrpy7bYXVLwcekyjYhYdG8akRqKjXIxeeT5PLZ0C6t2ZzHqhVW+17rGRXFVrzZc0aM1Td06zSrl68CqyzQioU7/SorUwpUXxvP7D76joMjDtvQc3/pt6Tl8tCUdV5iD4efFMaJnPCMvbE3rZhE2Vhtg1IFVREoojIjUQsumbj6behn7skr/d1/sMVm//ygff5vBvqw8PtueyWfbM5m5PIxFvxxEr/YxNlYcQHSZRkRKKIyI1FJCbCQJsZHl1l3eozW/vao7uzJP8PG3GSxcu5+jeQWMW7Ced+4bzHmtm9pUbQAJ946mURgRCXXqwCpSTwzD4IL4Zjw84nxWPPpTLm4fw9G8AsbO/4rvj+kXsG6UJyJeCiMiDaBZRDh/v2cA3VpFkZ59mrHz1/Njbr7dZdnLpZYREbEojIg0kNgoF29OHEj75k3Yl5XHuAXryT5VaHdZ9gnXvWlExKIwItKA2sY04a2JA4lr6mZbeg73LNzAyYIiu8uyh7cDq6cIikM4lImIwohIQ+scF8UbEwYQHRFGyoFj3PdGCvlFxXaX1fC8HVhBc42IhDiNphGxQc+20Sy8ZwB3zvuKVbuyGPSnz3CHOX2vu8IczLj+Ii7v0drGKuuZMxwMJ5jFVifWJs3trkhEbKIwImKTfh1b8OrY/kx8fQPHThYC5S9V/HH5Ng4dr/q+LTFNwrnqoja4woK0gdMwrE6s+TnqxCoS4hRGRGw07Pw4vpw2gu+PlYaO/KJibn/1K3ZnnuB/3vv2rPv/+ZbejO6fUN9l1p/wSCuMaHivSEhTGBGxWfNIF80jXeXWPXPLxXz8bUaV+2xLzyXt6EmeeP87nv739rO+f5voCJ69tQ8Xtouuk3rrlGZhFREURkQC0o19O3Bj3w5Vvr54Qxq/W7KFU4XFnCo8e+fXo3kF3PbqOl68vS8/7R5gfVB8s7CqA6tIKFMYEQlCYy7tyNDz4jhVcPYgUlDs4b43Uvj+2CnuWbiB5KmX0a1VAE1Fr1lYRQSFEZGg1aFF5Lk3Al6/ZwBXPLcCjwn3vZFC8pSfYBhGPVdXTd7LNOrAKhLSgrQbvohUV9dWTVly/xAAdmee4A8fbWP5lnSbqyrhvUyjlhGRkKYwIhICEju1YPyQzgDMX72PB976mrV7suwtCtSBVUQAXaYRCRm//Vl3XGEO1u7J4ttDOdw57yvCHNX//4grzMGfbrqY6/u0q7uivH1GdJlGJKQpjIiEiEhXGI9f3ZOdP+Ry3V9Xk1/koaDYU+39C4o9TF+2hf1ZeRjA5T1a06t9TO2K8l2m0WgakVCmMCISYi6Ib0bK/15Jjh93DDaB+99M4Zvvs3k+eScAb69PY+1jV9SuM6zvMk3VM82KSOOnMCISgpq6w2jq9u/0n31HP+av3kd+kYd/phwkPfs0XaYtB+CC+KZ88NAwIsKd53iXM4R7R9OoZUQklCmMiEi1JMRG8uT1FwGQc7qQj74pHZGz84cTzPjwO+KaugHo1qopN/Rtf+43dWk0jYgojIhIDbx0e1+euv4iTOC2V79kd+YJFq0/WG6bbq2a0r1NM9/PDgPCnGd0mFUHVhFBYUREasAwDFqWtII8fdPF/OubdEzTBGDFzh/Zf+Qk1720utw+TofBjOsv4s5BnUpXqgOriFDDeUbmzJlDly5diIiIIDExkVWrVp11+xUrVpCYmEhERARdu3bl5ZdfrlGxIhJ4+neO5cnrL2LGz3sx4+e9eOiK83FU0qe12GMyb9VeZi7fVjrHiTqwigg1aBlZvHgxkydPZs6cOQwdOpRXXnmFUaNGsXXrVjp27Fhh+3379nH11Vdz77338uabb7JmzRoeeOABWrVqxc0331wnByEigeOWxA5cc3FbCj2lw4ZX7cziwbe/Zv+Rk7yyci+L1qex+YkkDG8H1vxcOJ0NhqP0gVHmZ6P0WUQaHcP0tq1W08CBA+nXrx9z5871revZsyc33HADM2fOrLD97373Oz744AO2bdvmWzdp0iQ2b97MunXrqvWZOTk5xMTEkJ2dTXR0AN4GXUTOqrDYw8I1+/kh5zTzVu8DYP30EbQ+thkWJPn3ZlUGlbLrz3yubJ+zvVZ2/bk+y4/3M4xq1F5H71fhz6Oq9zsz7NXn+53tmKnis87ch2q+X3X/DBVy61N1f3/71TJSUFBASkoKjz32WLn1SUlJrF27ttJ91q1bR1JS+X9srrrqKubPn09hYSHh4eEV9snPzyc/P7/cwYhI8Ap3Orj3J10BfGHkTx9tIz7C4D5XO2ILDlf/zcySFhfz7HcsFvGHiYGJAxMwDQemlY58y77nkmVrH0fJeu+y9R7Wft73M8qs97532fUlr5VZNs9Ypsznmmcsc9b1RrkaqfK9reWWQ8dzXp9hDf5nD36GkaysLIqLi4mPjy+3Pj4+noyMjEr3ycjIqHT7oqIisrKyaNu2bYV9Zs6cyYwZM/wpTUSCRI82zdiekct7m6wA8gr/j3CKKf0n0uP9p5oy/8RS+k+4BwPKbWcYZrl9HHhK9jtju3KvU2a7kvVG5dsZZ9RU5tdP1dsZ5Y/Bux1n1FT67K2pkmM0Kqu9ij+LKmsq2e6cx1j653K22s98n9Ln8jWd+WdV6fdcoabKt6tYU9n39uA0/GroL8d6/5KAWzbo1vwtg87G7wdCMIQRrzNnXDRN86yzMFa2fWXrvaZNm8bUqVN9P+fk5JCQkFCTUkUkwDx7ax8+/jYDM5T+la8hT8lD/GBaAcXAg2FS8myFKzDPWPaGJ2sZ37be/Srbx7t9aZClzHKln3XG5xh4Kl8HJfuU+eyS5Qqf5dvfAyZn2cdTemwVPr98vfGdetvwhVn8CiNxcXE4nc4KrSCZmZkVWj+82rRpU+n2YWFhtGzZstJ93G43brfbn9JEJEj0ah9T+3vaiEij4tfQXpfLRWJiIsnJyeXWJycnM2TIkEr3GTx4cIXtP/30U/r3719pfxEREREJLX7PMzJ16lTmzZvHggUL2LZtG1OmTCEtLY1JkyYB1iWWcePG+bafNGkSBw4cYOrUqWzbto0FCxYwf/58Hnnkkbo7ChEREQlafvcZGTNmDEeOHOGpp54iPT2dXr16sXz5cjp1smZVTE9PJy0tzbd9ly5dWL58OVOmTGH27Nm0a9eOF198UXOMiIiICID/84zYQfOMiIiIBJ/q/v6u0XTwIiIiInVFYURERERspTAiIiIitlIYEREREVspjIiIiIitFEZERETEVgojIiIiYiuFEREREbGVwoiIiIjYyu/p4O3gnSQ2JyfH5kpERESkury/t8812XtQhJHc3FwAEhISbK5ERERE/JWbm0tMTEyVrwfFvWk8Hg+HDx+mWbNmGIZRZ++bk5NDQkICBw8ebLT3vGnsx6jjC36N/Rgb+/FB4z9GHV/NmaZJbm4u7dq1w+GoumdIULSMOBwOOnToUG/vHx0d3Sj/gpXV2I9Rxxf8GvsxNvbjg8Z/jDq+mjlbi4iXOrCKiIiIrRRGRERExFYhHUbcbjdPPPEEbrfb7lLqTWM/Rh1f8Gvsx9jYjw8a/zHq+OpfUHRgFRERkcYrpFtGRERExH4KIyIiImIrhRERERGxlcKIiIiI2KrRhZE5c+bQpUsXIiIiSExMZNWqVWfdfsWKFSQmJhIREUHXrl15+eWXK2yzZMkSLrzwQtxuNxdeeCHLli2rr/LPyZ/jW7p0KVdeeSWtWrUiOjqawYMH88knn5TbZuHChRiGUeFx+vTp+j6USvlzfF988UWltW/fvr3cdoH0/YF/xzh+/PhKj/Giiy7ybRNI3+HKlSu57rrraNeuHYZh8N57751zn2A6B/09vmA8B/09xmA7D/09vmA7B2fOnMmll15Ks2bNaN26NTfccAM7duw45352n4eNKowsXryYyZMnM336dFJTUxk+fDijRo0iLS2t0u337dvH1VdfzfDhw0lNTeXxxx/n4YcfZsmSJb5t1q1bx5gxYxg7diybN29m7NixjB49mq+++qqhDsvH3+NbuXIlV155JcuXLyclJYXLL7+c6667jtTU1HLbRUdHk56eXu4RERHREIdUjr/H57Vjx45ytZ9//vm+1wLp+wP/j/GFF14od2wHDx4kNjaWW2+9tdx2gfId5uXl0adPH1566aVqbR9s56C/xxds5yD4f4xewXIe+nt8wXYOrlixggcffJAvv/yS5ORkioqKSEpKIi8vr8p9AuI8NBuRAQMGmJMmTSq3rkePHuZjjz1W6fa//e1vzR49epRbd99995mDBg3y/Tx69GjzZz/7WbltrrrqKvO2226ro6qrz9/jq8yFF15ozpgxw/fza6+9ZsbExNRVibXi7/F9/vnnJmAeO3asyvcMpO/PNGv/HS5btsw0DMPcv3+/b10gfYdlAeayZcvOuk2wnYNlVef4KhPI5+CZqnOMwXgeetXkOwymc9A0TTMzM9MEzBUrVlS5TSCch42mZaSgoICUlBSSkpLKrU9KSmLt2rWV7rNu3boK21911VVs3LiRwsLCs25T1XvWl5oc35k8Hg+5ubnExsaWW3/ixAk6depEhw4duPbaayv8r60h1Ob4+vbtS9u2bRkxYgSff/55udcC5fuDuvkO58+fz8iRI+nUqVO59YHwHdZEMJ2DdSGQz8HaCpbzsLaC7RzMzs4GqPB3rqxAOA8bTRjJysqiuLiY+Pj4cuvj4+PJyMiodJ+MjIxKty8qKiIrK+us21T1nvWlJsd3pueee468vDxGjx7tW9ejRw8WLlzIBx98wKJFi4iIiGDo0KHs2rWrTus/l5ocX9u2bXn11VdZsmQJS5cupXv37owYMYKVK1f6tgmU7w9q/x2mp6fz73//m4kTJ5ZbHyjfYU0E0zlYFwL5HKypYDsPayPYzkHTNJk6dSrDhg2jV69eVW4XCOdhUNy11x+GYZT72TTNCuvOtf2Z6/19z/pU01oWLVrEk08+yfvvv0/r1q196wcNGsSgQYN8Pw8dOpR+/frx17/+lRdffLHuCq8mf46ve/fudO/e3ffz4MGDOXjwIM8++yw/+clPavSeDaGm9SxcuJDmzZtzww03lFsfaN+hv4LtHKypYDkH/RWs52FNBNs5+NBDD/HNN9+wevXqc25r93nYaFpG4uLicDqdFVJaZmZmhTTn1aZNm0q3DwsLo2XLlmfdpqr3rC81OT6vxYsXM2HCBN555x1Gjhx51m0dDgeXXnppgyf62hxfWYMGDSpXe6B8f1C7YzRNkwULFjB27FhcLtdZt7XrO6yJYDoHayMYzsG6FMjnYU0F2zn4q1/9ig8++IDPP/+cDh06nHXbQDgPG00YcblcJCYmkpycXG59cnIyQ4YMqXSfwYMHV9j+008/pX///oSHh591m6res77U5PjA+t/Y+PHjefvtt7nmmmvO+TmmabJp0ybatm1b65r9UdPjO1Nqamq52gPl+4PaHeOKFSvYvXs3EyZMOOfn2PUd1kQwnYM1FSznYF0K5POwpoLlHDRNk4ceeoilS5fy3//+ly5dupxzn4A4D+ukG2yA+Mc//mGGh4eb8+fPN7du3WpOnjzZjIqK8vV6fuyxx8yxY8f6tt+7d68ZGRlpTpkyxdy6das5f/58Mzw83PznP//p22bNmjWm0+k0n376aXPbtm3m008/bYaFhZlffvllwB/f22+/bYaFhZmzZ88209PTfY/jx4/7tnnyySfNjz/+2NyzZ4+Zmppq3n333WZYWJj51VdfBfzx/eUvfzGXLVtm7ty50/z222/Nxx57zATMJUuW+LYJpO/PNP0/Rq8777zTHDhwYKXvGUjfYW5urpmammqmpqaagPn888+bqamp5oEDB0zTDP5z0N/jC7Zz0DT9P8ZgOw/9PT6vYDkH77//fjMmJsb84osvyv2dO3nypG+bQDwPG1UYMU3TnD17ttmpUyfT5XKZ/fr1Kzec6a677jIvu+yyctt/8cUXZt++fU2Xy2V27tzZnDt3boX3fPfdd83u3bub4eHhZo8ePcqdZA3Nn+O77LLLTKDC46677vJtM3nyZLNjx46my+UyW7VqZSYlJZlr165twCMqz5/je+aZZ8xu3bqZERERZosWLcxhw4aZH330UYX3DKTvzzT9/zt6/Phxs0mTJuarr75a6fsF0nfoHeZZ1d+5YD8H/T2+YDwH/T3GYDsPa/J3NJjOwcqODTBfe+013zaBeB4aJcWLiIiI2KLR9BkRERGR4KQwIiIiIrZSGBERERFbKYyIiIiIrRRGRERExFYKIyIiImIrhRERERGxlcKIiIiI2EphRERERGylMCIiIiK2UhgRERERWymMiIiIiK3+PyRxW/0RZzOLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(thresholds,fpr)\n",
    "plt.plot(thresholds,tpr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However more common is to consider plotting them against eachother"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1692daa60>]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAi9ElEQVR4nO3de2zUVf7/8df0MtOLdISipUCtxRUF+YlrG7BlG6OL5QsGY6JLE/Yn6MLGRl2ELu5SMSCEbKNflyBKwQtITJBt8BY36QrdZBcKdC/tlo2x/OIFlhZpbVq1UynbQjm/P9h2GTqFfobOnM70+UgmaU/PZ+bdk+rnxTmfz/m4jDFGAAAAlsTYLgAAAIxshBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVsXZLmAwzp8/r1OnTmnUqFFyuVy2ywEAAINgjFFHR4fGjx+vmJiB5z8iIoycOnVKGRkZtssAAABBaGxs1MSJEwf8eUSEkVGjRkm68MukpKRYrgYAAAyGz+dTRkZG33l8IBERRnqXZlJSUggjAABEmCtdYsEFrAAAwCrCCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqx2HkwIEDmj9/vsaPHy+Xy6UPP/zwisfs379f2dnZSkhI0KRJk7Rt27ZgagUAAFHIcRg5ffq0pk+frldffXVQ/Y8fP6558+YpPz9fdXV1evbZZ7Vs2TK99957josFAADRx/GzaebOnau5c+cOuv+2bdt0ww03aNOmTZKkKVOmqKamRi+99JIeeughpx8Pm4yRznbargLAIBhjdOZsj+0yEEESk0bJFWPn6o2QPyivurpaBQUFfm1z5szR9u3bdfbsWcXHx/c7pqurS11dXX3f+3y+UJeJKzFG2jFHavyr7UoADIJLUpLtIhBROlc2KOkar5XPDnkEam5uVlpaml9bWlqazp07p9bW1oDHlJaWyuv19r0yMjJCXebIZIzUfXpwr9OtBBEAQEiEfGZE6v/oYGNMwPZeJSUlKi4u7vve5/PZDSTRuDxhjPTW/0jNnzg/duUXkpt/c8GuaFqGMEb6v9v/pv/XPPSzwFW/ukdJ7tghf19En8SkUdY+O+RhZNy4cWpubvZra2lpUVxcnFJTUwMe4/F45PF4Ql3a4LA84S/jLil5rDRAkATCwRijh7dVq/bEt7ZLGWIJQ/puOZmjlTp69ID/8AOGi5CHkdzcXP3+97/3a9u3b59ycnICXi8y7JztjO4gMu7/SI99PPhwEZ9EEIkQ0TRzcKnO7p4oDCLS1PQU7SnKHbL/xBLjYwkiiAiOw8j333+vL774ou/748eP68iRIxozZoxuuOEGlZSU6KuvvtLbb78tSSoqKtKrr76q4uJi/fznP1d1dbW2b9+u3bt3D91vMdQuXpbpvmh5JhqXJwgXUePi8GGM9JNt1apviv6Lv2uemx01yxCEB4xUjsNITU2N7rnnnr7ve6/tWLx4sXbu3KmmpiY1NDT0/TwrK0sVFRVasWKFtmzZovHjx2vz5s3D97beyy3LuJMkd3L4awKuIHqXLS4vJ3O0UpPdnMCBCOcyvVeTDmM+n09er1ft7e1KSUkJ3QcZc+GukZd+0P9nGXdJP3OwnAEMIBTLJ53dPcrZ8Md+7UM97T/cMJMADG+DPX+H5W6aiBBoRuTiZRmWM3CJYEJFOJZPLl624GQNIBIQRnpdeqEqd43gMobrsgjLFgAiEWEkkJVfEESi0FAuj1zt3RyhWj5hJgRAJCKMBOJmSSYahOvukmDu5iA0AMB/EUYQlcK1jMKyCABcPcIIosbFMyEDLaOwqRQADD+EEUScQNd+XG4ZhrtLAGB4I4wgojhdfmEZBQCGP8IIgmbj2SdXuovl0mUYZkIAYPgjjGDQhtuzTwLdxUL4AIDIQxjBoAy3Tb5YfgGA6EEYwaCcORueu1MGixkQAIgehBHpwppDd6ftKiIGd6cAAIYSYSTQA/JwWUnuWCW5+dMBAAwNziiBHpAXn2SvnmHk0k3EAAAIBcLIxXhAnqQLIaSzu8f63TIAgJGBMHIxHpB3xbtmcjJHKzHe2UPhAAC4HMII/Fx61wybiAEAQo0wggHVPDebvTwAACEXY7sADF9JbmZBAAChRxgBAABWsUwzAl3uAXfcwgsACDfCyAgz3J4xAwAAyzQjzEDPmLkUt/ACAMKFmZER7OJnzFyKW3gBAOFCGBnBeMYMAGA4YJkGAABYRRgZQXqfOQMAwHDCHP0IwV00AIDhipmREcAYo7bT3X5BhLtlAADDBTMjUS7QjAjPnAEADCfMjES5S/cVyckcTRABAAwrzIxEgcFu786MCABgOCKMDGOXCxn/7SP9ZFu16pt8V3w/nsILABiOCCPD1FDf/cIFqwCA4YowMgwFuvvlSqamp2hPUa4Gmvhge3cAwHBFGBlmBrr7ZaBnyPQibAAAIhVhZJjh7hcAwEhDGBnGuPsFADASsM/IMMbdLwCAkYAwAgAArCKMAAAAqwgjw4gxxm/HVAAARgIuYB0mhnqTMwAAIgVhJAwGs617Z3f/W3rZMRUAMBIQRkIsmBkPbukFAIwkXDMSYpduYnYlbHIGABhpmBkJI7Z1BwCgP8JIGCW5Y5XkZsgBALgYyzQAAMAqwggAALCKMAIAAKwijAAAAKsIIyHE9u4AAFwZt3aECNu7AwAwOMyMhMilm52xvTsAAIExMxIGbO8OAMDAgpoZKSsrU1ZWlhISEpSdna2qqqrL9t+1a5emT5+upKQkpaen67HHHlNbW1tQBUeiJDe7qgIAMBDHYaS8vFzLly/X6tWrVVdXp/z8fM2dO1cNDQ0B+x88eFCLFi3SkiVL9Omnn2rPnj36+9//rqVLl1518QAAIPI5DiMbN27UkiVLtHTpUk2ZMkWbNm1SRkaGtm7dGrD/X/7yF914441atmyZsrKy9KMf/UiPP/64ampqrrp4AAAQ+RyFke7ubtXW1qqgoMCvvaCgQIcPHw54TF5enk6ePKmKigoZY/T111/r3Xff1f333z/g53R1dcnn8/m9AABAdHIURlpbW9XT06O0tDS/9rS0NDU3Nwc8Ji8vT7t27VJhYaHcbrfGjRuna6+9Vq+88sqAn1NaWiqv19v3ysjIcFImAACIIEFdwHrpxZjGmAEv0Kyvr9eyZcu0Zs0a1dbW6uOPP9bx48dVVFQ04PuXlJSovb2979XY2BhMmQAAIAI4urV37Nixio2N7TcL0tLS0m+2pFdpaalmzZqlZ555RpJ0++23Kzk5Wfn5+dqwYYPS09P7HePxeOTxeJyUNqyw8yoAAIPnaGbE7XYrOztblZWVfu2VlZXKy8sLeExnZ6diYvw/Jjb2wuZfxhgnHx8RendezdnwR9ulAAAQERwv0xQXF+vNN9/Ujh07dPToUa1YsUINDQ19yy4lJSVatGhRX//58+fr/fff19atW3Xs2DEdOnRIy5Yt04wZMzR+/Pih+02GCXZeBQDAGcc7sBYWFqqtrU3r169XU1OTpk2bpoqKCmVmZkqSmpqa/PYcefTRR9XR0aFXX31Vv/zlL3Xttdfq3nvv1QsvvDB0v8Uwxc6rAABcmctEwFqJz+eT1+tVe3u7UlJShvbNu09Lv/nPDM2zpyR38lW9XWf3OU1ds1eSVL9+jpLc7LgPABiZBnv+5kF5AADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwijACAACsIowAAACrCCNDyBijzu4e22UAABBR4mwXEC2MMXp4W7VqT3xruxQAACIKMyNDwBijttPdfkEkJ3O0EuNjLVYFAEBkYGbkKvQuy/xkW7Xqm3x97TXPzVZqslsul8tidQAARAbCSJAGWpbJyRxNEAEAwAHCSJDOnO3xCyJT01O0pyhXSe5YgggAAA4QRoYAyzIAAASPC1iHALMhAAAEjzASBPYTAQBg6LBM4xD7iQAAMLSYGXHo0gtX2U8EAICrw8zIVeDCVQAArh4zI1eBC1cBALh6hBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVhFGAACAVYQRAABgFWEEAABYRRgBAABWEUYAAIBVhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVhFGAACAVYQRAABgFWEEAABYRRgBAABWEUYAAIBVhBEAAGAVYQQAAFhFGHHAGKPO7h7bZQAAEFWCCiNlZWXKyspSQkKCsrOzVVVVddn+XV1dWr16tTIzM+XxeHTTTTdpx44dQRVsizFGD2+rVs6GP9ouBQCAqBLn9IDy8nItX75cZWVlmjVrll577TXNnTtX9fX1uuGGGwIes2DBAn399dfavn27fvCDH6ilpUXnzp276uLD6czZHtWe+Lbv+5zM0UqMj7VYEQAA0cFxGNm4caOWLFmipUuXSpI2bdqkvXv3auvWrSotLe3X/+OPP9b+/ft17NgxjRkzRpJ04403Xl3VltU8N1upyW65XC7bpQAAEPEcLdN0d3ertrZWBQUFfu0FBQU6fPhwwGM++ugj5eTk6MUXX9SECRM0efJkrVy5UmfOnBnwc7q6uuTz+fxew0mSO5YgAgDAEHE0M9La2qqenh6lpaX5taelpam5uTngMceOHdPBgweVkJCgDz74QK2trXriiSf0zTffDHjdSGlpqdatW+ekNAAAEKGCuoD10lkBY8yAMwXnz5+Xy+XSrl27NGPGDM2bN08bN27Uzp07B5wdKSkpUXt7e9+rsbExmDIBAEAEcDQzMnbsWMXGxvabBWlpaek3W9IrPT1dEyZMkNfr7WubMmWKjDE6efKkbr755n7HeDweeTweJ6UBAIAI5WhmxO12Kzs7W5WVlX7tlZWVysvLC3jMrFmzdOrUKX3//fd9bZ999pliYmI0ceLEIEoGAADRxPEyTXFxsd58803t2LFDR48e1YoVK9TQ0KCioiJJF5ZYFi1a1Nd/4cKFSk1N1WOPPab6+nodOHBAzzzzjH72s58pMTFx6H4TAAAQkRzf2ltYWKi2tjatX79eTU1NmjZtmioqKpSZmSlJampqUkNDQ1//a665RpWVlfrFL36hnJwcpaamasGCBdqwYcPQ/RYAACBiuYwxxnYRV+Lz+eT1etXe3q6UlJShffPu09Jvxl/4+tlTkjs5YLfO7nOaumavJKl+/RwluR3nOAAARpTBnr95Ng0AALCKMAIAAKwijAwCT+sFACB0uPDhCnqf1nvxQ/IAAMDQYWbkCnhaLwAAocXMiAM8rRcAgKHHzIgDPK0XAIChRxgBAABWEUYAAIBVhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVhFGAACAVYQRAABgFWEEAABYRRgBAABWEUYuwxijzu4e22UAABDV4mwXMFwZY/TwtmrVnvjWdikAAEQ1ZkYGcOZsj18QyckcrcT4WIsVAQAQnZgZGYSa52YrNdktl8tluxQAAKIOMyODkOSOJYgAABAihBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVhFGAACAVYQRAABgFWEEAABYRRgBAABWEUYAAIBVhBEAAGDVyA4jxkjdnbarAABgRIuzXYA1xkg75kiNfw3wI6PO7h4LRQEAMPKM3DByttM/iGTcJcUnyRijh7dVq/bEt/ZqAwBgBBm5YeRiK7+QksdKLpfOdJ/zCyI5maOVGB9rsTgAAKIbYUSS3EmSy9Wvuea52UpNdssV4GcAAGBojOwLWK8gyR1LEAEAIMQIIwAAwCrCCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKuCCiNlZWXKyspSQkKCsrOzVVVVNajjDh06pLi4ON1xxx3BfCwAAIhCjsNIeXm5li9frtWrV6uurk75+fmaO3euGhoaLntce3u7Fi1apB//+MdBFwsAAKKP4zCyceNGLVmyREuXLtWUKVO0adMmZWRkaOvWrZc97vHHH9fChQuVm5sbdLEAACD6OAoj3d3dqq2tVUFBgV97QUGBDh8+POBxb731lr788kutXbt2UJ/T1dUln8/n9wIAANHJURhpbW1VT0+P0tLS/NrT0tLU3Nwc8JjPP/9cq1at0q5duxQXFzeozyktLZXX6+17ZWRkOCkTAABEkKAuYHW5XH7fG2P6tUlST0+PFi5cqHXr1mny5MmDfv+SkhK1t7f3vRobG4MpEwAARIDBTVX8x9ixYxUbG9tvFqSlpaXfbIkkdXR0qKamRnV1dXrqqackSefPn5cxRnFxcdq3b5/uvffefsd5PB55PB4npQEAgAjlaGbE7XYrOztblZWVfu2VlZXKy8vr1z8lJUWffPKJjhw50vcqKirSLbfcoiNHjmjmzJlXVz0AAIh4jmZGJKm4uFiPPPKIcnJylJubq9dff10NDQ0qKiqSdGGJ5auvvtLbb7+tmJgYTZs2ze/466+/XgkJCf3aAQDAyOQ4jBQWFqqtrU3r169XU1OTpk2bpoqKCmVmZkqSmpqarrjnCAAAQC+XMcbYLuJKfD6fvF6v2tvblZKSMjRv2n1a+s34C18/e0pyJ0uSOrvPaeqavZKk+vVzlOR2nNcAAIAGf/7m2TQAAMAqwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsIowAgAArCKMXMQYo87uHttlAAAworCj138YY/TwtmrVnvjWdikAAIwozIz8x5mzPX5BJCdztBLjYy1WBADAyMDMSAA1z81WarJbLpfLdikAAEQ9ZkYCSHLHEkQAAAgTwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsIowAgAArCKMAAAAqwgjAADAKsIIAACwijACAACsIowAAACr4mwXMBwYY9R5tsd2GQAAjEiEEUk/3f43HW44Y7sMAABGJJZpJNU1fNv3dU7maCXGx1qsBgCAkWXEzowYY+S6pK3mudlKTXbL5br0JwAAIFRG7MzImUuuEcnJHE0QAQDAghE7M3Kxql/do9TRowkiAABYMGJnRi6W5I4liAAAYAlhBAAAWEUYAQAAVhFGAACAVYQRAABgFWEEAABYRRgBAABWEUYAAIBVhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVgUVRsrKypSVlaWEhARlZ2erqqpqwL7vv/++7rvvPl133XVKSUlRbm6u9u7dG3TBAAAgujgOI+Xl5Vq+fLlWr16turo65efna+7cuWpoaAjY/8CBA7rvvvtUUVGh2tpa3XPPPZo/f77q6uquungAABD5XMYY4+SAmTNn6s4779TWrVv72qZMmaIHH3xQpaWlg3qP2267TYWFhVqzZs2g+vt8Pnm9XrW3tyslJcVJuQPq/L5dSS/dcOHrlQ1KusY7JO8LAAAuGOz529HMSHd3t2pra1VQUODXXlBQoMOHDw/qPc6fP6+Ojg6NGTNmwD5dXV3y+Xx+LwAAEJ0chZHW1lb19PQoLS3Nrz0tLU3Nzc2Deo/f/va3On36tBYsWDBgn9LSUnm93r5XRkaGkzIBAEAECeoCVpfL5fe9MaZfWyC7d+/W888/r/Lycl1//fUD9ispKVF7e3vfq7GxMZgyAQBABIhz0nns2LGKjY3tNwvS0tLSb7bkUuXl5VqyZIn27Nmj2bNnX7avx+ORx+NxUhoAAIhQjmZG3G63srOzVVlZ6ddeWVmpvLy8AY/bvXu3Hn30Ub3zzju6//77g6sUAABEJUczI5JUXFysRx55RDk5OcrNzdXrr7+uhoYGFRUVSbqwxPLVV1/p7bfflnQhiCxatEgvv/yy7rrrrr5ZlcTERHm93MECAMBI5ziMFBYWqq2tTevXr1dTU5OmTZumiooKZWZmSpKampr89hx57bXXdO7cOT355JN68skn+9oXL16snTt3Xv1vAAAAIprjfUZsYJ8RAAAiT0j2GQEAABhqhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVhFGAACAVYQRAABgFWEEAABYRRgBAABWEUYAAIBVhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVhFGAACAVYQRAABgFWEEAABYRRgBAABWEUYAAIBVhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVhFGAACAVYQRAABgFWEEAABYRRgBAABWEUYAAIBVhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVhFGAACAVYQRAABgFWEEAABYRRgBAABWEUYAAIBVhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWEUYAQAAVgUVRsrKypSVlaWEhARlZ2erqqrqsv3379+v7OxsJSQkaNKkSdq2bVtQxQIAgOjjOIyUl5dr+fLlWr16terq6pSfn6+5c+eqoaEhYP/jx49r3rx5ys/PV11dnZ599lktW7ZM77333lUXDwAAIp/LGGOcHDBz5kzdeeed2rp1a1/blClT9OCDD6q0tLRf/1//+tf66KOPdPTo0b62oqIi/fOf/1R1dfWgPtPn88nr9aq9vV0pKSlOyh1Q5/ftSnrphgtfr2xQ0jXeIXlfAABwwWDP345mRrq7u1VbW6uCggK/9oKCAh0+fDjgMdXV1f36z5kzRzU1NTp79mzAY7q6uuTz+fxeAAAgOjkKI62trerp6VFaWppfe1pampqbmwMe09zcHLD/uXPn1NraGvCY0tJSeb3evldGRoaTMgEAQAQJ6gJWl8vl970xpl/blfoHau9VUlKi9vb2vldjY2MwZV5WYtIoda5sUOfKBiUmjRry9wcAAIMT56Tz2LFjFRsb228WpKWlpd/sR69x48YF7B8XF6fU1NSAx3g8Hnk8HielOeaKieE6EQAAhgFHMyNut1vZ2dmqrKz0a6+srFReXl7AY3Jzc/v137dvn3JychQfH++wXAAAEG0cL9MUFxfrzTff1I4dO3T06FGtWLFCDQ0NKioqknRhiWXRokV9/YuKinTixAkVFxfr6NGj2rFjh7Zv366VK1cO3W8BAAAilqNlGkkqLCxUW1ub1q9fr6amJk2bNk0VFRXKzMyUJDU1NfntOZKVlaWKigqtWLFCW7Zs0fjx47V582Y99NBDQ/dbAACAiOV4nxEbQrHPCAAACK2Q7DMCAAAw1AgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKscbwdvQ+8msT6fz3IlAABgsHrP21fa7D0iwkhHR4ckKSMjw3IlAADAqY6ODnm93gF/HhHPpjl//rxOnTqlUaNGyeVyDdn7+nw+ZWRkqLGxkWfehBhjHR6Mc3gwzuHBOIdHKMfZGKOOjg6NHz9eMTEDXxkSETMjMTExmjhxYsjePyUlhT/0MGGsw4NxDg/GOTwY5/AI1ThfbkakFxewAgAAqwgjAADAqhEdRjwej9auXSuPx2O7lKjHWIcH4xwejHN4MM7hMRzGOSIuYAUAANFrRM+MAAAA+wgjAADAKsIIAACwijACAACsivowUlZWpqysLCUkJCg7O1tVVVWX7b9//35lZ2crISFBkyZN0rZt28JUaWRzMs7vv/++7rvvPl133XVKSUlRbm6u9u7dG8ZqI5vTv+lehw4dUlxcnO64447QFhglnI5zV1eXVq9erczMTHk8Ht10003asWNHmKqNXE7HedeuXZo+fbqSkpKUnp6uxx57TG1tbWGqNjIdOHBA8+fP1/jx4+VyufThhx9e8ZiwnwtNFPvd735n4uPjzRtvvGHq6+vN008/bZKTk82JEycC9j927JhJSkoyTz/9tKmvrzdvvPGGiY+PN++++26YK48sTsf56aefNi+88IL529/+Zj777DNTUlJi4uPjzT/+8Y8wVx55nI51r++++85MmjTJFBQUmOnTp4en2AgWzDg/8MADZubMmaaystIcP37c/PWvfzWHDh0KY9WRx+k4V1VVmZiYGPPyyy+bY8eOmaqqKnPbbbeZBx98MMyVR5aKigqzevVq89577xlJ5oMPPrhsfxvnwqgOIzNmzDBFRUV+bbfeeqtZtWpVwP6/+tWvzK233urX9vjjj5u77rorZDVGA6fjHMjUqVPNunXrhrq0qBPsWBcWFprnnnvOrF27ljAyCE7H+Q9/+IPxer2mra0tHOVFDafj/L//+79m0qRJfm2bN282EydODFmN0WYwYcTGuTBql2m6u7tVW1urgoICv/aCggIdPnw44DHV1dX9+s+ZM0c1NTU6e/ZsyGqNZMGM86XOnz+vjo4OjRkzJhQlRo1gx/qtt97Sl19+qbVr14a6xKgQzDh/9NFHysnJ0YsvvqgJEyZo8uTJWrlypc6cOROOkiNSMOOcl5enkydPqqKiQsYYff3113r33Xd1//33h6PkEcPGuTAiHpQXjNbWVvX09CgtLc2vPS0tTc3NzQGPaW5uDtj/3Llzam1tVXp6esjqjVTBjPOlfvvb3+r06dNasGBBKEqMGsGM9eeff65Vq1apqqpKcXFR+5/7kApmnI8dO6aDBw8qISFBH3zwgVpbW/XEE0/om2++4bqRAQQzznl5edq1a5cKCwv173//W+fOndMDDzygV155JRwljxg2zoVROzPSy+Vy+X1vjOnXdqX+gdrhz+k499q9e7eef/55lZeX6/rrrw9VeVFlsGPd09OjhQsXat26dZo8eXK4yosaTv6mz58/L5fLpV27dmnGjBmaN2+eNm7cqJ07dzI7cgVOxrm+vl7Lli3TmjVrVFtbq48//ljHjx9XUVFROEodUcJ9LozafyqNHTtWsbGx/RJ2S0tLv8TXa9y4cQH7x8XFKTU1NWS1RrJgxrlXeXm5lixZoj179mj27NmhLDMqOB3rjo4O1dTUqK6uTk899ZSkCydNY4zi4uK0b98+3XvvvWGpPZIE8zednp6uCRMm+D0qfcqUKTLG6OTJk7r55ptDWnMkCmacS0tLNWvWLD3zzDOSpNtvv13JycnKz8/Xhg0bmL0eIjbOhVE7M+J2u5Wdna3Kykq/9srKSuXl5QU8Jjc3t1//ffv2KScnR/Hx8SGrNZIFM87ShRmRRx99VO+88w7rvYPkdKxTUlL0ySef6MiRI32voqIi3XLLLTpy5IhmzpwZrtIjSjB/07NmzdKpU6f0/fff97V99tlniomJ0cSJE0Nab6QKZpw7OzsVE+N/2oqNjZX033+54+pZOReG7NLYYaD3trHt27eb+vp6s3z5cpOcnGz+9a9/GWOMWbVqlXnkkUf6+vfezrRixQpTX19vtm/fzq29g+B0nN955x0TFxdntmzZYpqamvpe3333na1fIWI4HetLcTfN4Dgd546ODjNx4kTz8MMPm08//dTs37/f3HzzzWbp0qW2foWI4HSc33rrLRMXF2fKysrMl19+aQ4ePGhycnLMjBkzbP0KEaGjo8PU1dWZuro6I8ls3LjR1NXV9d1CPRzOhVEdRowxZsuWLSYzM9O43W5z5513mv379/f9bPHixebuu+/26//nP//Z/PCHPzRut9vceOONZuvWrWGuODI5Gee7777bSOr3Wrx4cfgLj0BO/6YvRhgZPKfjfPToUTN79myTmJhoJk6caIqLi01nZ2eYq448Tsd58+bNZurUqSYxMdGkp6ebn/70p+bkyZNhrjqy/OlPf7rs/3OHw7nQZQxzWwAAwJ6ovWYEAABEBsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsIowAgAArCKMAAAAq/4/0UynGAjOlDsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(fpr,tpr)\n",
    "plt.plot(fpr_rbf,tpr_rbf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the plot of $\\text{Re}(\\text{FPR}^{-1}(r))$.\n",
    "\n",
    "There is also the AUC, which is Area Under the Curve, which is defined as\n",
    "$$\n",
    "    \\int_0^1 \\text{Re}(\\text{FPR}^{-1}(r)) dr = -\\int_0^1 \\text{Re}(\\alpha) \\text{FPR}'(\\alpha) d\\alpha\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the AP (Average Precision) and the (AUC) is used as a single performance metric of a classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $Z = G(X)$, where $G$ is the predicted probability, then $Z$ has density $F_Z$. Let $F_{Z,Y}$ be the joint distribution of $Z$ and $Y$\n",
    "\n",
    "Let\n",
    "$$\n",
    "    \\begin{aligned}\n",
    "        f_1(z) &= f_{Z \\mid Y=1} \\\\\n",
    "        f_0(z) &= f_{Z \\mid Y=0}\n",
    "    \\end{aligned}\n",
    "$$\n",
    "thus we simply get\n",
    "$$\n",
    "    \\begin{aligned}\n",
    "        \\text{FPR}(\\alpha) = \\int_{\\alpha}^1 f_0(z)dz \\\\\n",
    "        \\text{Re}(\\alpha) = \\int_{\\alpha}^1 f_1(z)dz\n",
    "    \\end{aligned}\n",
    "$$\n",
    "As such \n",
    "$$\n",
    "    \\text{Re}'(\\alpha) = -f_1(\\alpha)\n",
    "$$\n",
    "and we can write\n",
    "$$\n",
    "    -\\int_0^1 \\text{Re}(\\alpha) \\text{FPR}'(\\alpha) d\\alpha = \\int_0^1 \\int_z^1 f_1(z') f_0(z) dz dz'\n",
    "$$\n",
    "Consider $Z_1$ be a random variable sampled from $f_1$ and $Z_0$ be sampled from $f_0$. Then we can write the above as\n",
    "$$\n",
    "    \\mathbb{P}(Z_1 > Z_0) = \\int_0^1 \\int_z^1 f_1(z') f_0(z) dz dz'\n",
    "$$\n",
    "It is useful to think about what this probability means. That is, if we take a randomly chosen sample from the positive class and call it $X_1$ and do the same with class $0$ and call that $X_0$, then the AUC is the probability that $G(X_1) > G(X_0)$."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "lx_course_instance": "2023",
  "lx_course_name": "Introduction to Data Science",
  "lx_course_number": "1MS041"
 },
 "nbformat": 4,
 "nbformat_minor": 5
}