DL-with-Python-and-PyTorch2/pytorch-03/pytorch-03_02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 训练模型"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "\t构建模型（假设为model）后，接下来就是训练模型。PyTorch训练模型主要包括加载数据集、损失计算、定义优化算法、反向传播、参数更新等主要步骤。\n",
    "1.加载预处理数据集\n",
    "\t加载和预处理数据集，可以使用PyTorch的数据处理工具，如torch.utils和torchvision等，这些工具将在第4章详细介绍。\n",
    "2.定义损失函数\n",
    "\t定义损失函数可以通过自定义方法或使用PyTorch内置的损失函数，如回归使用的losss_fun=nn.MSELoss()，分类使用的nn.BCELoss等损失函数，更多内容可参考本书5.2.4节。\n",
    "3.定义优化方法\n",
    "\tPytoch常用的优化方法都封装在torch.optim里面，其设计很灵活，可以扩展为自定义的优化方法。所有的优化方法都是继承了基类optim.Optimizer，并实现了自己的优化步骤。\n",
    "最常用的优化算法就是梯度下降法及其各种变种，具体将在5.4节详细介绍，这些优化算法大多使用梯度更新参数。\n",
    "\t如使用SGD优化器时，可设置为optimizer = torch.optim.SGD(params,lr = 0.001)。\n",
    "4.循环训练模型\n",
    "\t 1）设置为训练模式：\n",
    "\tmodel.train()\n",
    "\t\t调用model.train()会把所有的module设置为训练模式。\n",
    "\t 2）梯度清零：\n",
    "\toptimizer. zero_grad()\n",
    "\t\t在默认情况下梯度是累加的，需要手工把梯度初始化或清零，调用optimizer.zero_grad() 即可。\n",
    "\t 3）求损失值：\n",
    "\ty_prev=model(x)\n",
    "\tloss=loss_fun(y_prev,y_true)\n",
    "\t 4）自动求导，实现梯度的反向传播：\n",
    "\tloss.backward()\n",
    "      5）更新参数：\n",
    "\toptimizer.step()\n",
    "5.循环测试或验证模型\n",
    "\t 1）设置为测试或验证模式：\n",
    "\tmodel.eval()\n",
    "\t\t调用model.eval()会把所有的training属性设置为False。\n",
    "\t 2）在不跟踪梯度模式下计算损失值、预测值等：\n",
    "\twith.torch.no_grad():\t\n",
    "6.可视化结果\n",
    "\t下面我们通过实例来说明如何使用nn来构建网络模型、训练模型。\n",
    "【说明】model.train()与model.eval()的使用\n",
    "\t如果模型中有BN (Batch Normalization）层和Dropout，需要在训练时添加model.train()，\n",
    "在测试时添加model.eval()。其中model.train()是保证BN层用每一批数据的均值和方差，而model.eval()是保证BN用全部训练数据的均值和方差；而对于Dropout，model.train()是随机取一部分网络连接来训练更新参数，而model.eval()是利用到了所有网络连接。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5实现神经网络实例\n",
    "前面我们介绍了使用PyTorch构建神经网络的一些组件、常用方法和主要步骤等，本节通过一个构建神经网络的实例把这些内容有机结合起来。"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAADoCAYAAADMkb9iAAAAAXNSR0ICQMB9xQAAAAlwSFlzAAAOxAAADsQBlSsOGwAAABl0RVh0U29mdHdhcmUATWljcm9zb2Z0IE9mZmljZX/tNXEAAGxSSURBVHja7Z0JeBRV1veDIEnITgJkgUCAJBAIBMgCJBC2sEOAsKPsm4BgBJQAgkogoEJAFhUFRwEFVETHcR1xwRkVdHR0RsGZeV9nXrd5Z3FGZ30/Z87X/xtuT3Wlqruqu7p6yfk/z3nSqa7eblfX/dU5554Tcdttt0WwsbGxsbGxsQWzOW+sXbu205AhQ25lY2NjY2NjYwsGW7hwYZ+IiIhmLsBy5U7aunUrGxsbGxsbG1tAbfLkyVRSUvK4A1iuagQs2IHFYrFYLBYr0Pre975H/fv3Z2BhsVgsFosVvGJgYbFYLBaLFfRiYGGxWCwWixX0YmBhsQKsp556ig4cOEDvvPMODwaLxWLpiIHFImGyURqLZVQAlnXr1gloCZQuX74cVr/FZs2akeOkRv/+97/5AGOxwkQMLBYJy62ys7OF4ba/9MUXX/Bgh6EwyQYCWPC627dv9+sxGyhNmTLFdmD59ttvxesClnAuuHTpEh/cLJZFYmCxUJhw/D3pYHJhhZ8YWKxXIIBl/fr1Tg/rsWPHhOeMvTxEBw8e5HEIcjVv3pzuvvtuU98TvterrrqKJk2aRP/617/8/h4ZWCyUGlhwtSVPXnC5S++I3I6/yu0Qtsv/cZ901WPfV155RVy5YZ9wcuGztIEF3zkmPXV+i/I4Ux9zOHbwP8JM6ueWxxZuq48fBhZrtHjxYhevCgCmqU/UONYC8V009fMJQALzxXfffWfoMe+++67XYFlVVcXAEmpSTx74oeKAkVewcBFjwsB23MbVl9wuJ5iysjLncyivfPEY3Mbz4S97WsLvBKMGFhwL+J6xHd+7FCBGHhc4hhYtWiRu49iSxw8eJ48RPDf2w//YVyvBtykAizJcAzt37pzzd4ucl/3797vclr+7nJwcsX9tba14PpkjgzHD7xa3Ma7qEz3uC7cLC+V4rF271mU8Ll68KPbBRInxw32YAGU+EUxuZ/lfgAh8J0bHm4GliUkrJISTmvSYKCcK3IbHBMKJFBMNpJ641BNJOE4sLG1gAZjIsIISWOR98jhSbpPJuzCAsDz54Pkl2GipKQCL9FZp/VYBF/gdQjt27HA+Dh6Tzz//3Pl4pecKE/fZs2c1Xxu/7XD0gg4aNMg5HpjcMCYSTOS5TR7LyrFnD4v9YmBhuZUesCjvVwKL8iqXgaVpS/29S4iVx4k8PqQwcWKyUHrasK8SWJTP5ylHpikAiwxNAOSUnkw53jKUpgQNpXdA7SHQm4TDFVbcgQgDS/BJD1hwbObm5joTwz/77DOxHcACWAeU4rh/8sknnY99+umnnSEm5XblazGwhNiPGZMFTP5wpYseBwhOiLhPeWUs98VtefUrr4TxFydPPF6Z44KJBSfEQC6BZVkruawZ3638XnHMyEkVprxPCicVebWrPN7kY/CcymNT6zlk0i1OROF2TKknSWV+iax9oxTyTTBG6olWjrEM5+o9P6SGlXBLNnXnYZEe4+PHjzcCFjxOho9Y9kgPWJYsWSK2Y07Cb0LuA2DB9/TnP/9Z3Dd48GABIbiNx2C7fLzyvCNfi4ElhIQTndIgnBRlvonMQZH3yas5wAkmE+mOhuTkIh+jdEPL51RuY4W25OSp5xWRydcSdqW08phkUi32lceUfB4tD6C7+0JVyjos0isix1lux0katx9++GHn4zDhqsdYmbOB390333zT6Pll7gbkziMTDsIYKXNYMB7qMceYKnNaIEyEuA8hNPa02CM1sMBLgtuvvvqq8LDg4gYelgsXLjiBRQnYEkKwXXpXpMnHKF+LgSVMpfa2sFhmhEkDsCKvaFnWCMCmvHBgsUJZSmBRwgg8J9JbAq+ich+Zv4ULH+lhwW14YuRjsJ/6d8LAEsZSe1tYLDMCqLCHzTrhhKz2trBYoSz16izYQw89JGAEUIL7SktLxUo5eE8+/vhj4SXDuUUrV0WZwyJXy8nXUXpf1J4Xq8XAwmKxWCwWK+jFwMJisVgsFivoxcDCYrFYLBYr6MXAwmKxWCwWK+jFwMJisVgsFivoxcDCYrFYLBYr6MXAwmKxWCwWK+jFwMJisVgsFivoxcDCYrFYLBYr6MXAwmKxWCwWK+jFwMJisVgsFivoxcDCYrFYLBYr6MXAwmKxWCwWK+jFwMJisVgsFivoZSmwPPLIozSovJyNLaStYtQo+vLLL/36w7vppnU0aPBAyyyvRzfK7NiB2nfMpNh26RSf1l78D+vbt7elr6Vlu/fcxWdTFovlV1kKLHPnzaOFa6rpyNkfsLGFrOXl96JXXnnFrz+8DpntqbZ+Dd1zbKvXdtPWRdR3cBFdHRND0SUVFDFzA0Vcu5Uiap+hiC2PN/zvsOhOuZSYnk5jpgylPffd7NNraln1xnkCWv79738H7ESG76tZs2bkOJFZZzEJDXZ1ZMP/reIa/o+MtvZ1NCwhIZ7++Mc/+mWstm7d4r+xat5Ce+z8aIDyf/3rXzybM7CYB5bb9x+in/7uz2xspuzln39Cm++sp+lLVlBuSZmLLV+/gXYdPkoXPvutLe+lpGyQLcDy1CsH6MInp0zby+8cdYxTFcXl9aGI1Yco4sSvKeLsn9zbgQvUfNbNlNSxE22sW+nV6+oZoCXQwPLggw9S5dQRdOGy95/j9PN7qHD4AGoZG0fRI2Y0QB/gT46vYwzF/8v3UFzv/hSTnCy+h/MfHLd0PGEdMtPpv/7rv/wyVnPnXUtbdl7n01jte6CGMrplU6vUDGpZucwxLt9vGJvTXzWM1a6XGv5fVEcxHbtQm06ZVLNjBb19+ZTlY3XVVc1CBlheffVVcW7RMqO/Hyueg4GFgYXNCzt06gkqGjmeopLbUdTYheIEJ050Spu5gWKGVFFkXAINmlhFZ398sckCy/GndgnoiJx3i2dI0bIHL1NsxQzqO2wgvfHhcQYWh2Ec5q6cTvFdu1FEzXFTY4nvIT4tjfbeX9MkgOW5H91HZWPLKbZoCEXsed34WDlgL3bkTOqUn0enntsd1sDy97//XRMgABpDhw4lx9yqaefOnWu0v6/PwcDCwMJmkUcF8BFXPMIxSZwwdtLD1dvaI44rthyatarabx6XYAUWwEpyQbGYKL2CFYW12HiMOuT3EN6apgws+PyZvXpSy1k3OY6vL72HwOKhNHPFLMs8CMEILDj+Ejt1pogbH3B87q+9G6tdL1F8dnfacWC9Tx4eb4Hl6aefFuGwSZMm0TfffON23y+++IKOHz9u6rjeu3cv1dXVhb2ng4GFrcnY6VfeoIROOQI+vJ1wI+dtpdRu+fTcTz5sEsCy+4FNDbBiJPxj1OrPU6d+fXyGllAFFox/VmEfxzi8bsl4xs5eR4MrKyyBlmADFsBKSp8Sijj+qe9j5bjwiC8dTeu3r7JkrMwAS25uLv35z38WIHLs2DHdY/add95xPO9VtH//flPH9a233kqYNxlYGFjYwsD2nXyCUvqWWTPxHrhA6f3KLIeWYAMWuOGTs3Ms8aw0stpnKKewd5MDFoxpas+elsGKtKgb9lsCLcEELPh+E3sUWAMrCosfOZ3W7/AdWowCy+XLl2nHjh3iOP32229pypQpbo9ZQMuBAwdCDljOnj1L7733HgMLAwubr7CSPGi8tV6Cwx9YDi3BBiwACpHLYzWsXLHoaatp8Y3XNClgyS0qaEgU9cN4xlQupRU1C8ICWAB2KTm5FHH0kvVjdforSizoT0dObbMFWNQAEq7AEs5iYGGzxQAUSTn51sKKwtOC8JBVOS3BBCyrN86nKKzC8BOsyIkjIa+3cPs3BWCZs3ImRc9e730ehuGJuDbkgUWA3bbv++/Yc1xwtHEA0Q8vHrHdw7J48WIGFgYWBhY2VwNIdC0ua1jq6KcTX4s1h2jYtDlhBSxYvYIVKH4JBamt5gT1GzYg7IEFUBaX3Z0invzav+NZf57Su2XT25dPhiywrNk0n6IBy34eq+ZLdlDl3Alej5U/clggrMhhYGFgYWtihhU9SJL196QbWzqO6h86ETbAYot3RTl+Xbt75WUJJWApGzvYp2Rvc8fjaLrj0LqQBZbYlBT/hII0PFIxae3p2TfuDcgqIUCM+vEy6RaF6cwk3oYDsGBZNvJgbrjhBsfvupwmTJhIe/bsoY8++oiBhS287Y1f/oaik9v9p6CUn69qM7rnhw2woDCZLd4VaasP0fBJw8MWWJCPEZuWQRGnvrRnPH3wsgQaWO50gFZc6Sj/e6Is8LIEUx0WeGWefPLJgP4Wqqur6cyZM169h6+//poqRo2mYSPH0g0330qHvvc47dp/lObMX0Y9evaiEydOBPSzMbCw+dWqt9VR9KQV9nkJLPCyBAOwHD1d21DJ1i5YuVJPBJAUrsCycNV0Ue3Xb7krWsdjdh4dO7sz5IDF77krajvxa1EY8u1LoQ0swaCFCxfS0aNHTf8eP/74Y8rvXUAHH3yM3vz55/TWR1+42Ovv/TeNnVhFmzZvCdhvnYGFza8Wm9LOXi9B7TOinH+oA8t/Jtc/2WrehIVCBVhQSt7qZcwePQezNtACx3dpduluIIEFdXki4+Jt865IiysYQIce3sLAEgBgQRioY6csOvPCm41ARW3TZs+ngwcPBuT3zsDC5jdDgbjY7Hx7J93TX4krNV9WDAUDsPh7KbOetaxcLpoZhhuwBGoSbqhzU2A61BFIYLE7HOSEu9kS7k4ysNgMLPX19TRn/lIXz8pdhx6idZu307mLn7gAyw9ef1/AzV//+lcGFrbwsZWbb6WW06ptn3TjCspE1+VQBpbEtFSx5NPusUMvp6q548MOWMQkPND+SRgA3bxlS+E5wDgZ7Zbdtl0KPfLII7pN7mDIlcA52qz1LuhN5SMKacn1UzWte+8cirh2i/1jteVxyi8tCglg+ec//6m5HZ4KWCgBi/SufP/Vnzih5LY7DlBBvxJatuZm8fflC5ddoGXOgmW0e88e23/zDCxsfrNe5SMaOt4GYNJF1+dQBhZMcrYkKqtt7REqHTM4qIDF3aQt7eabb6bS8n66ADB7wTiKGjvP1vwVp+cgKppiYltRn6Lu1DLyavG3X0meW4uMbIkTs26TOxhWuvgDWDrmZlHEmnvsH6sDF6hNVseQAJbOnTtTQUFBo9Uzb775prBQApaTJ0/SmAlT6M2ff+YEEqyQkpByXXUN3Xnwey7el4fPvES9ehfYPu4MLGx+szadc8RJyPZJd/UhqphuriYLOkDDKwPrnt9LuEiNTJTeWtu2bWh7/RrNyXXPfTfR1a1i7B832GbHBNC8uVjOGSzmbtKW1q1bN0ppk6QLAACGiO797fcaXEm8HTy8UIRhxk8Z0hCOCeKQkO0Jtz4m3gYKWJTHKI6/LVu2BMXSX7PAsqGmhlZUb3QCyTOvvUdDK8Y6AeZ7jz1P627Z4QIsSMCNjm5F3333HQMLGyfc+mS3nqEWV7c0NSlmZedQYWmZsPiEBHH1ZGSi9NYioyKpZ0G25uSK7S0dJ+5AAUtUTCuxBDhcQkKYnOHZiJi6NiAeloZk0q0CVuYurRTfsScvQiCBpW1WR4rY/3Zgjr9mV5kCFhynzRzAcu7cOb9cWDz33HOaXqqkpCTdc0mg82ngMYGXx+jvsbJyEu28+4gTSAAonoAF1jUnl37+858zsLCFh4kfcCBOene+TDGJrSm3Zz4de+6HIRcSQoXbgI3dlscpPjmR0jLaGF4tFMzAgom5dEhfSnR8pojluwMyphJY8H4mVA2llDaJdOq53QwsOsBi5kIj2TGWRr1w3tjo0aNDDljMqrikP93/yNMOCDHuYYH1HziIXn75nK2/ewYWNr8YVuk0bxkVmJPegQvUslUM7T76sPCY7Dp8NGSABbBSPqKIro6O9k/fJQPhtOKhxRQXHyM8PXcaqNQarMAiQzAjxw2kYaNKKGLmhoB4WNS1WLr37EJFA/LdelkCCSztc7pQxNYnOCTkRuqQUGJiIi1btoyef/55+tvf/hZSwDJj5izaduchFyDxlMPCHha2sDOcfAIy6dY+Qxk53WnY2PECnEZPrqI1m28NCWAZMaY/1davoTadMgOT/+OY1JF4CWjC+8BfT8ucgxFYJKwgZJDevq3jCnFBwJJuo5LbuJSdFxV341rRwusm69ZnCRSw7HughlJSUxzgykm3noAFkDLPMecpq8ru3LmT6urqQqo0P97zvCXXuwCJepXQj3/2WaMicpzDwhZWlpiRGZiluWuP0KCJVSIkhFoweC8L11RT5cw5huqzBApYevXNpSEOQMBtlMiHt8N2b8DAUcKrgnBQTvdO4r3Mmj+WpsyqCBlgkbCC24AtvP+ALmuOjGw0CQ+pKKaKcQMFHGpBSyCABdWVC/v3oElzxlKzRXX2A0sILWtGCX6t4z0YegmZXVaNvkHDR41zWSUEu+ehJzTrsPAqIbawNFScDUTxs4jle2jC3IWiRD+8LPL94NgsHTZC9DcKNmDBxIV4vMwbwYQSPWKG/YXjYuNFkTW8hwGDC8QVN25vuG2xSBaV9wUrsChhBe8tq0t7kS8SbIXjAFDwXsmlxGposRtYcNwBVn548QjNnDeGIjt04cJxXigYgAVhqffee88U4OTkdjNU5TbQ1W4ZWNj8ZtdcX00t5t5q+6QbM6TKmbei9LLAsGy5V2ERPfeTD4MGWOAFGD2xzOnRkGGDqNYpAZhcezeaVJVggrwWtWcoWIBlQFkfAVly8gdsNfx/MihL86e0TRKhIi1osRNY8H3m98lxhq169OpKV0e34tL8IQos3kjPy6JlgfKuMLCw+dUAB6g6a7uXIC7B6UXZfGc9TZu/0OV9oeaKuxVEdgKL9AggbKHOFek3bABF1JywL9eichmt3jjftUWAA6KUq4VOP79HbEP4IJiAZe3atZSW3obOf3BMEXYpckkaDrbmh/OXT6KVa2cJUFm6eqo4DiRc2QUsOAbhWTl7br/z+4VXSrSG4OaHTQZYIKwWqr/vuGbjQ2XuSsnAQfTss89yLyG28FspJBJv7azYWn+eMrrnu7yH9A6Z9PLPP3F5bwAavRVEdgHL1l0rhAfj3DsPimXE6tonAIG43v3tGbcHL1N8WppYpaQVulD35UF4SBZACzSwoFZGv379aMLkYU6vgUy2VQKMSHZNy6CIU1/adiymd8vWDXEAFvAe5f0SXvG/HcCC7xGwogSq+csnC4iyu59Q8yU7qHLuBNPhIAYW6/T111/T4PIhtGHrTk1oQQ+hPoXFdOLECe7WzBaeVrVkBbVYXGcbsERPWkHV2+pc3sP62jq6ZtkKTaDSWkFkB7C0a9dWhFcACFpQIC09t6uY+ALhXXF6WfI6NarJgveNyXXeskkBBRbACkrUHz582GWVkAyzqD9L2djBIinbFu9K6Wi6w8OycACDrNGihJb2HdL8Ciwba5cK6Dxyapvr8eYAKOltiU1JoYijl2xJTI5Ja++ykoqBxTfw8EVLly13AHNHxwXADNpcu4cWr1xLxQPKqG3bdvTuu+8G9LMxsLD51ZAr0iot0zYvQUJ6ZqOVQHpeFmnqFUT+BhYkxKHS7SPfv1OccAErevVOEHpJ7F3iXy/VgQuUmpPdyLvizssiDVfjhf17UmlZf9tPZBJWUPdCvaxZ5oeo3y/AKy67u/89Bx68K9KwdBx1YpT7AVpiY1v5DVjmXDObuvXIor33b2jk0QNAyfeyZtN8inaArL/HyhfvSrABC37bOHcEclK/7rrrRONMX97Df//3fws4mDtvvgAwVBIOBq8RAwub361o5HhbcjG0vCvS4EUBmOi9R+UKIn8CC04EqKCZlp4qQgIIUyAcpAcLsBU1Cyimconfrm7ju3YTuQvuJgUtL4u0RSurKCExgb744ouAwAqkBBYJWHo1TuasnEnRs9f7L5fFMaaJBf3pyKlaj5MtvnfAFVbnKLe3Tk6iCRMm+KUIWVZWFlXNqmgECKjCq1495Pe+Qoc/oDY5uY0+f6gCSzDIbC+hUBIDC5vfDZ6NlNx8//YV2vUSdS0u062zgvcAL4u7OixyBVGBw/wBLHDVokcRoEUm3co6IZ5Oyn2HDaSoG/ZbvzKjfCLduqfa4+u787LgyrygIF90FzaznNIqWFEDizrZVrfBX61/JuKYyqUCMo1OuOgkLRKuVauE9u3b1+hz+qoFCxbQwNIBjcAE4BSfEOuS8yPzflIcQOGX0JAT7LZ5DSsMLAwsDCxsobVi6MSvKSkn3+1SZbHMetkKkc/ibh+sIGoVEyNqDFgNK/CsyAldAot6FY67K/HBlRXWQYtjsgCs3LRjleGJQc/LInNYpPcIQOEvYfwGDBjQaBKXwNKQyNrGY3gBE3Fqz56WL3PG94PvSc+7o2UY09y8LJf3LJNuMZaTJ0+2BFpkfsW1c69pBCxaoSnl95vYwwF4xz+1dKziR06n9Y7jz8xYMbAwsDCwsNliizfeSrHTqy2/SkseNJ72nXzCkKfHk5cFVth/gPCEPProo5b8yFCYCVfKyokcwIIcAmXtFSOGyTB62mrfWh4cuCDyYox4Vox4WZRJtwAzNIy75557/AIrQ4cOpT/+8Y+N7pPAgiRg5NUY+TyAm6zCPpZBS+zsdaZhxenxcQCLcrWOcpWQFdCyd+9euuGGG8R3pFU4bmB5H8fxWOMWqlL6lFgDLQhDlo6m9dt9h5VgAxYc/74mvTKwMLCwBYnNuXEDJYycbTusmPGyIIflhRdeoJkzZ4q+IL4KE7ja6wBgmTh1qMc+PVq2+MZrKKFjlihlbna8ADtIsFXWUTFjWl4WrVVCy5cvF2YHrEhgwbJm5AOZWW2Cpb2ZvXpSy1k3OcbHy+XOD16m2OKhNHPFLK8nYFQSnqLIK1Eva/YFWnCSnz9/vvP7UQOLCPu0TfLolcL3ntipM0Xc+ID3+T+7XqL47O6048B63eaLoZ50a0dYlIGFgYXNJtu49xDFFY/wrc+Q48SXkJ1vClZkyCcrO8cjsMgclg0bNoiTvdn+HFJ4LCZTtdp3yKC2qcmNaq8YNTwuv7SQWqVmUMvK5e6XPtecoJjySmoZGydgx12Cr9GJ1UilW3hZAGt6kGEVrEhgKR7Y222yrbtw29yV00XycUTNcVOgEjnvFlG/xp13wuh7UOaQaNVhAbQgHGZmPPGYMWPGuHw3amBZtW628EwZWaWD465sbDnFFg2hiD2vm/LqxY6cSZ3y80SrBCtAhUNCDCwMLGy22KFTT1ByVo7jhL/VXHjDMUnEjFtIWQVFAj68eW30F0KfISPAIidDTLxmXb2AHT0PTUqbZOpf1svnk7VM3BX1WiIiqHnLSFFsLi6vj/gfhoq5yFHwBVSUEytyRJSg5a4OCyZM5LV4u0TXCKzI76hN29YuNU3MGlZKFQ4fIMBO9HGCBwu9sOTxie7Z+H/5HjHGMcnJNH1JlQMyjlsy8TpX6bipdGt0PCAcw4AVtVdGDSzqcJQRQ9sDtDpoAOZlDQnMGBu5/N5xQSH+X1RHMR27iO7jNTtWWBICYmBxr5MnT9LHH38clp+NgYUtYIZcklmrqkU1XPT/EQW9tDwFcpIoKKPYlHa04c56n14XvYVQmt8osMiTP1bBILHUiAAqABY9tWrVitbdssDykzeAAgDhbcjHiKlXNnkqHIdJtk+fPvTjH//YNKwAdoxMzjt27KDYuBiva3mow0SY0OHBQon6SAfAAPww6eL/cTNGi89s9bjiO0NPH0+Vbo1Ai7t9lMCilfDrDTDnOsYFY9O8ZUsxVvCk4P+quePp1HN7/HYsMrA0LTGwsAWFoUT+oIlVoqy+9AxIQ9dndF/GSiOrXs+dl0WvDguuWpCM++abb7r9HeBqH6EgPcFT07x5c3rshXq/nsj9ZWovi5FKt/jMmECNJjKbgRVo1KhR1LN3jiXAEkiT3aU9leZ3BySAagCi3tgpgUWW4g/lceOkWwYWBha2sDY0PkTNFTPAIk9ImEj1Jl6EQBA+cifkdsTGxjTqehxKpvSyGC3NjzygWbNmeUxkxoSLydhoGAnPm5iYSGMrh1iWyBmwcd3UMK5GSvNLaFEW7DMydkpg0asIzMDinUK5lxADCxtbEBvCQggPmQEWOUFqrSDCYwArnhJ0EVpCL6FQBhall8VsLyGEyjB+WitezMIKBI9WWVmZS2n+UDXZtDGjfaqhMZB1abAvYBq3P/roIw/n6QZgQR7KwPKCkPdKMbC4CseBt4sEGFjY2ILUEBJCaMgssCgnXrl0V4YwPLmDEVbq1q2bs3BcSHsDrnhZvGl+CA+VOuTjDaxAmKQ3b94cFsACG1JRTMkpSYbHAeNWXl4uwkA/+clPDJynG4AFheKQjB3q48XA4qpXX32Vk24ZWNiaipfFTC8hhHcwWRQWFtKXX35pCHLgmQkHYJFell0H1nrVrRm5QJhk4RHwFlZkQq+6+WEoGzwf0dFRpkJiyOHp1auXocfIbs1apfgZWEIfWMJZDCxs7GVReVnMAAsmWkwURlcQpaamCrAJB2CRXpZho/t7BSxy/Pr27SuSmb1Z+ozOtIDGcAIWWIsWzentt982NAYoKHfmzBnD0AdgqZw2lMZPGRLy4SAGFgYWBha2JmUpbduJsv1mgUXZHwjmqfmfMiE3XIBFdBtuk0gl/Yu8BpZBgwaJsTNbzh+ehaSkJJELE27AkpAQR6tXr/Y4pmhmiM8u98N4IkTmLjQEYOnUOZ0OPbwlLMaKgYWBhYGFrckYSvWjZL8ZYMFkCVhRLnGG5wQT75NPPqn5GCSaytVF4QIssEnTh1OHDhmmT9JqjwC8Jeh3Y1QAHOQQ4XXDDVjS0ttR+/bt3Y5pTU2NCC+q98FxiHHVg5YpVZMpMSkuLLwrDCyNxUm3DCxsYWwoYIemiNLLYmSVkLqZofq++vr6RicRhIPkiSScgAU5LFFRkab63Mi6LGqPFMYN42ek/ooyyTTcgAV1WJAb9eyzz2p+9p07d4p8KL2JUY6vFrT07deXSof0YWDxgwDhge4lBPDn0vwMLGxNxMviCVj0+gMpBU+Bsvmf9AZIhROwYJVQRkZaI0gzCytS8FB5KucPzxZCH/KkHI7Asm/fPuGVU0886maGnsZZDS1JSYl03Y2hXSwuWIElGMS9hBhY2JqAlwW5LG/88jdugQUgYnRils3/MHEgVKQMH/kDWFBSHs0JUVIepdGV1YJRJh09hbCcFftZDSzIYenUqZNHL4snWJGS9UX0whoyd0PKDmBBGXt8Vpi3TSvNAAuWpmZlZbkUhgPMIcnW6GQkx1u2RcAx2KZtG7qlbrlfxwptBuRYWX28MbAwsDCwsAWtWVmS353BwwJPix6w4Pdg9jeBsFFxcTF16dLFZbuVwIKmfaVjBoumfVFj5zU07UPzOXU/ppoToqkf9sP+Vk26sg6LJ5gzCitSyMUAtKhzgvA8aWlpLnDkD2DBuC5cNV30EAL0xXbt3tBY0mFRrVOcIAhItBpgZGn+6upqMaaYfHAsAVbMhN6U447wEsIFJf2LaUvddZaOFY6B8TNHi4aQGBc035Rj1fJKLyYA8y111gMzAwsDCwMLW0AMHZiXrN8gujFL7wCaHsrb2I77ve3U7M6Qw4JclqKBpY2AxVN/IHeCNyAzM9NlorYCWLBCZ/GN11B8124NjSONdryGOfZP6JglHm8VsHz55RfCI6A1oZqFFSnk/GCSVlYVxgQOOFKekK0EFnwvvcoKKSazMzWfdXNDl2a9cXSAICARADN6xhj64cWjlgILvCzdu3cXniajXZr1xnHixInUrl07mjqtyqVbs6/ffdusjgJMIpbvpogHL+mPVc0jDmCeSdHJKbSoeo5lIalg6yVktEEqAwsDC1uIGnr7oPFhTMccxySxobGHQJpjO+7HfoAXPM5qL0vHzp1dgAVX+EgE9VZItv3pT3/qsoLIV2DBVWr+0FKKnraaIk5/ZQ5WpJ34tXh8j4FFAn58BRacILW8LN7CilJYEQNgBAxhAleXn7cKWABwcZlZFLHlMdPj2eK63ZTkgMA7Dq6zDFgg1Kjp2bOn17Ai9dhjj1F6ejoNHz7MZ2DB8Td0ckUDqOx/29xYOY7X6OlrqE2njnTk1DZOumVgYWBhCw1DzsiwaXMoLq+IIurPmzvxOeAFj6taskI8j1VelsioKHrhhRfEMQ03PGDF22WCytoryhVEvgALJotO/fpQi1sf8w5U1BOt43kAP95CixJYEMZRelnwmT3VBTEqQAnCa+gbpD4ZWwEs5ZMqrgDgl96P54OXKXbgaLph61JLgAUTIABt6tSpPk9A8FSdOnVKePtmLxzntYcDx19WYV9qsWa/4zN/7f1YHbhAiQX96a77N/kdWL799luaMmUKNWvWjM6ePet2LHfs2CH2y87OpkuXLoXcpM7AwsDC5gcDHGQVlTlOfId8m3AX14nnsQpakpKTadu2bc5icL7UNFDWXpGCF8Lbbs2YLFJzss3DnR+hRd1LSHpZ3C3/9lYoMpeTk2OphwWfGbASdcN+y8Yzbtw8mrkCK3G8B5b3339fLN1GWEgWyPNWyryfOdfMpm49skQ+idnxQq4OYCViz+vWjNXpr6h12UifoMUIsBw/fpwOHDhA33zzDQ0ePNjt/osXLxbH8uXLl+ngwYMhN/Gjl1Cgw1IhBSw/+/03YTlYnsQQYtye+8mHlJpfZN3E63iejF5F4nl9fW/5ffqKq1DAiqdmhp4mCWXtFaWSWidR3+I80wmIfYcNpBYbj1kKK9IiF9fSxLkTfQYWeFmwYmjChAmWwor03sgVROfOnbMEWIZOHkFRK+6yfDxjKpfSipoFXkELujXj+JOeKdmCwNvJ895773UW2ZO9hFCa30xoSPSOys2xDlYU0JJYXE6HHt7qN2ABhABWIHhQLl68qLvv+vXrxb7wyiA0zFVrGVgYWJq4Z0XAyuEPrD3xHbggnldZZt8bKygsooSEBLrvvvt8Oh7UtVeUQkho664V1K8kz7CnZWPdSoqtmOEXWJEWP3AU1R28ySdgAaB17dqVli1bZunvC4m3slgaYHDMmDHOpc3eAst/xvRr68cTE3FBfzpyqta0x6dVq2iXkvvqujNmhcfKpc2yWzPGakLVUMPQgpwVn8NAbkJpyTm59Owb9/oFWBAOkmMHr8k777yjuy88FNu3b6d169Z5Vb05XL0bTQZYcDCBWHGSCRf7y1/+0uhgZhgxZrklZQ3LbP0w4V698QT1Gznep5ya2Lg4ERJCsqMvUtdeUQMLQAW1PXoWZIu/nlzxManpvuVXGJw44tPSTIWGlMAiw0CHDx+mbt26WXp1Cu+KspgcXgsrsBCC8gZYMOaJPXpTxCk/jqkDyts4JuIfXjxi+H0NLO9DKW1aNyqcp6zsa0ZypZH8LpTAgtczAi3Vty6l+HFzKeLJr/03Vo5zQk5RgencGm88LHq5Kah5I70q3oSEuJdQGADLL37xC3GluWnTJnI8t/hSQ9Xw/m+66Sa6+eabG4ULGEY82+rb6yhm0nV+nXRjxi2kDXfWm35vKB7Xq7BIhIRw0kK1VaNdm9XCVRbCInpSJt0CRgAtdx7SX10yae4Ear54u39h5YpFVS6j1RvnmwYW5Ecoc1aQ5KnXV8ms5EotrYkA+TLo+Dxh8jBTwFI4fICoTePv8Wy+eIf4/oyEhiQ8dMhMawQse/fubbSc25tJVA0snqAF8JqQ7oDlo5f8PlaxpaNNr7KyMocFnhcAjQQWmc8SSsCCc48voewmDyyIFy5ZskQkyn366achbz/60Y9o5cqV9PnnnzOwmAwFxaZler8M14QrHq9jJjQEWBk2djwdOvWEs3CcL8uZPRWZU68SEomfI4qoeuM8N96Vr2wBFrNeFgBLaVn/Rgm2yDWBl8qKk7cn+EFn46TWCYZDCvCuxGV396/HQHE8xqRleHxvsxeMa/j+L7sua5ZSr8AyKrVnSgtYJLQsuX5qIw/Hmk3zKdoBsbaMVf15Su+WbcrL4u0qIXhTlKEiKV9WCQUDsOB3x0m3PgDLu+++K7wr3333XVgM2m9/+1vavHkzA4tJw/JjrOixY9LF6+D1jL630ZOraNfho+K2stItJlxv6irAu+LupKG3rHnW/LE0ZVaFqzveMYm1rFxuD6xcMVTEFZOaQWBJap2omWBrhZcFEzVWuLibBBASGjKipCG8dnZX0HhXnF6WuVtp1pLJul6WpaunXoGFxnVYlJo1a5ZYdWZ0QsRxjBo4yv31gKXhfUxzgRY7vSveelm4W3PTkW3AgmS5v/71r2ExaP/zP/9DGzduZGAJRu+KF16WafMX0mZFCEkJLN54WWQ4yZ3c1WFBqfcBgwucK4hQ0tzOyVXY6kM0fNJwQxPGgEEFlJPbVfMkDdhD7oUvJ3A5CbiTzGHBmOb3cR9ew7hGxsXb4zFQ5LIkpqdpeg4AD1ixo7xPD1gAhUg2NjqeWquL3AGLGlowjnGlo+wdq5pHHMf8QMNeFgYW/wlL64MpvMTAwsBii6FHT8vK62yddFtVzPZYF2j5+g3ClNvUvYTMelmMdHP2VDhu3wM1zhVEkejFcuLX9gILJti0VI+TBSbauUsrXVYJqeWLlwWJteoGgO6ABZMwvAJDKrTDazAxCQ+0eRLG8ZjWns6e2+/yXmrr19DIcY0nZz1ggdQhHndjB8+UukKuJ2BRQkvV3PHUbFGdvWPlONYj4xLo7UsMLIEQznUTJ1ZSQmIide+RT/EJCZTfqxfdcsstAf9cDCwMLLZY0cjxAfESVEyf4xaiUIpfvV0NLLJ3jdEJNtHxQ/dUbM5IpVvkWXTNzaRWHbLsHTeZfNu6jdumfrKOh3pZs9YJ0Fsvi7tkWz1gUYfX1GGYqfMcx+KiOtvHM7piJt2iCLMBSpG3dP6DY25L83s7KZ48eVIULlTvZwRYJLTEp7S2vu6KkbBQdh4dO7uTgcUL+ZJ0i99bn35FtPPuI/TiWx/TWx99IezhMy/RsuvX04iKkfSHP/yBgYWBJbTs/d/9yVRCK66YbAsHKcukp7TTfE84TitnasOMGlgAH8hJQS6FJxltkmi0NP+23aspultBQIAFHXePnq51CytadVi05K2XBeEPI8Xn9JY1I7wGT5VyWXFWrx4UsetF+8d0eT2NmzFaeFMAo/l9cjRhxROwGMnpkWOHDs2Nz9PGgAWequZXt6SIJ/9oPyyPm083O747I2Ehbn7oKm9L8+/ZU08jRo2j137yKyeoqO3u+x+lskGDAwYtDCxBCiyP/vA1YW9++kXIww0aFIpeQQGYdFulZTaqfouVQFgRBJAyAiyQUS+L0aXQRoEFEwsSYAMxdrEDR2nmgihhxSiweONlUdcP8QZY5Pvr2Sebzp5rGO+k9DSKuO+n9o9pzQmRm3Hs7C4q7N/DbW0Wd8DiDkaUUKO3osgosOD4jElrb3voTNisGlqssWIp2IElGOQNsACysnO60avv6sOKtJpb76AlS5cFZMybDLDgy/vNb35Db7/9tkhC27NnD+3evZteeuklseQNZPx///d/tgELQERCibSXf3bZef+wcRMos3MXsT3UgQWAEFc8IjBegoIyOnL2By7wVFhapgsresBixMviqfaKN8Cycu0saj71hoCMndZKITWsuAMWQArGURqaFqIgn3KbngFAUHhv1KhRziXi7qyyspJy8zqLvAuEguBVURp657RqFUU9enWhZs2b+7dYnJvCaFn5De9Hncvirvmh1vhgHDGeaE+A/wHUyvHAuGH81OMEoG6X2o46dU5vNEbScrp3IseEICwyNSMwwHJdPY2dMSZogcWItzWUgKWycpIIA73588+dYPLKO7+g66prXLZJ65HfW8zrdoe+mgSwyAqccEmjT0TLli2dP8i5c+cKkEGRIFkJ0Q5gefqtdwWQ9CkZ4IST6zdtcdmnZuddpoAFEKTlkdHbbpfhmEACbEC8BKXjqP6hE+J9nH7lDQErnpokagEL5Km2iqf7Qw1YWuYVO38n3hiSlTFBSkNxt/j4eJdtenbttdeKXKC1a9eaBhYk2wKilNajd1da4RjLu49uEjU2AjIJH7xIV7Vo4YCFDBcocGcAYK3xQfGzqKgoGjhwoPgf3j/leCBkhNYI6nHCcT1q1EiRKK0eI2my6jJuxxf0D8xYrb6Hhk8aEbTA0rlzZ3F848LXSAJ0IIAFydZGGrfCk9k1J5d+/LPPnECy/paGWjTlw0fTm4rt0mp330vTZsy0fdybBLAALFDsDTUM0OlVCSyLFi0SAPLJJ5/YCixqIAHA+Aos8CRo7a+33S7DKpyImRsCExK6slIIYSHAipHmiHrAIpv66Z0EPNVe8QZYAhoSKhoiEkOVIQJMtOqGjUZCQlJGy8uj1ohWwqie3IWE/rNsuOH/gAFL/Xlqk5VJhx7e4rEVg6eQEFRTUyP6K6nHyFMozWhIiIHFPbCo4Rzwcv78ea/qNvkDWHAOa926tQBX9JHSOx7QGHPa7PmNPCnwsAytGKcJLD94/X1KTU2zvbZakwCWDz/8kE6cOEE9evQQB1cLx1WOOGk5bqMC72effSbaByA0ZDew7Dh0WMDEvJWrXUIX7oBFhpDU2+Ctkc/nabsyLKX0vgCcEJqS94XqkmZnWGPoVJqz7DrKzOosCsNhjD1Z9/xewsWu5YqfOnWqqHKs3o79cdIyEu6AtW3bhrbXr9G9ypUGb0FM9z5Bk3SLnBasbPEWWIyu+kGzPr0+TGaABaucsrq0d8kXCVgOy5bHKb+0yNAkbARY9MBED2TMAssj37+TIhNbByTpFjks46Y0dHD29BvBuVyGxuwyeLD0vGK/+tWvggJY8Fu76qqrXLx1qOKrPq7gnbthw22aoZ+hI7WBBZaYlES///3vGVis1uuvv0533XUX5ebmNgIWeFgCCSwyJARz54GRgCH3x9/Jc+Y2ynmR93vaLkNS8rnwP7YDnGDYBlN7fUItJBTVOY9S0zMor6CP8LAYMdQdUIczpMEFD1c8XPLK7ampqaLZn5FwBywyKlJUZdXLI5DWvWdnimybHlTLmr1JulXKk5fFmwRdPWAZWF7g4iWCZXTLpoj61wPgNWgoxmcVsECoYovJWilMpu7q1hgFltETy6hFZFRAgKVFWSW175jq8fcBw3nc6O/OKouOjtYFlkAnAOOCACFSmJzn1FZeXi6gBknZo0aNpvr7jpsGlj6FxSJyYWceS5MBljvvvNMJLM2RdHfli0O3VxkSCgSwSCDZd+xRj8ACz4fcBo+JGnL0PDJa26VHB9vxV8KP9PYogcdXYMHzI/k1MF6CIpFoa+b96oWElFck8KhIGa29opTRkBAs2ArHYbkrQkOnn9/jFbB48rKox9dbYFGHggJVll/aVUOn05BR/S0FFpzAsYxejqWRSrhGgAWQB9gTx97xTwOQe2a8PH8whITw+5/nmP/OnDkT8OJqp06dEs153QGLtJKSEipznO8YWIIAWP7xj3/Q//7v/1Jtba1wMSOeh/yVtm3b0vDhw2nfvn302muvOVcJ/fOf/wwYsBi5H8AiPSu4zxdgkV4dadKTgv2wv5XAcvbHFymmY05AgOWqq1vS/hOnLAUW2dRPOVkaqb3iLbAUDy2miPnb7B27RXWiyqm7onbwEAFezAILpOdl0avOahZYGkJBGZpLh//Tm8ne3Iy43iXCa3Dqud2WAYt6vHABhrHwBVjwnaa3byuaNRYOH+iAu+P2jtXpryg6pa3bZd/BACzBBClaQm6KGljwnmfMmCG8K9ILpxcSQg5Ln8L+Lsm4HBLyg3Dw4ABGqAfelYkTJzo9K1ilUFRUJDLmZcKQ3N/oQWfVsmZ4MuAp0avDIu+X4RqAhIQPeC2UoRzl/eokW63t6ueSHh68Hl4X7w/PjddQLrf21hIzMsVVu91JjqnZ3Sm3Z76pzs2egAWCh0AWQjNae8UbYMHEm5qeQlGDK4OiBot64scSYm+ARc/LgpIDy5cvNz0BqIFFKxQU0Poiotx8vKje6q5gnFlggWS/ILj3kxyTiKduzp6ApWr2SFFwD56pLbuuo1ZI+rZzrGqfoZyigqDuJfTqq6/q9s4KdNKtFKruAlgQql6zZo2o26P1nrWSbuUqIQk6amjhpFsLhasOeEweeOABEavLyMhwhoHS09NFNjdABh4VfIHSjMqqZc16uStqD4j0fkjIwTY8XkKH+jnVeSd626WXRgKK8nXlqiXc1gpXmTWUyEcM324vwfQlKwSQITfFSmCRXhYztVe8ARbE6Pcf3UTxSPJ78LI94+Z4HbwerrQ9Nj4cXECr1s8xDSyQlpfF6Coid8AiQ0HuQh62J96uPUKlY8rFJFy9qQH03E3IZoBF5vzgqlkZHvIGWACfKGon31sg4K757A20YNV0bn7oo3BcGFm1qLWs2ZPxsmYL9bvf/U6EelasWEExMTEiyRbAkpycLIotYfXBb3/7W68PKi7NHxqJtyhWh6J1ek0OfQEWCB4CuFfN/k6MAgtqisBwe/XG+RRVucyeZFvH6+D1jEwW8AAhzFHSv8j070ntZfGl55AEFnehIKX9ZzztmYjVvXEABVgBYwWwSNArLCxslIBrBlgAqLl5WY1CVraGhU5/JQAJ4Sgjxx8DizaomO0lpFU4zp1x4TgLBWLcvHmz8K5Il9bVV19NpaWldOONNxqut8LAYp2hsmxCeqZ9XoL685TRPd/lPZQO+w/AWAEs2AcrhrzpHeIJWHClC++KcjKxxctiwruiDA0lJCZ4dfJSelRkaMMbSWCBx0cvFKSenBPS0yni6CX/H4tXSvIrPQZay619ARZUvo2NjTX0HegBy/zlk0WhQrVnA/lKcQ7gssPL0nzJDqqcO8Gwd4WBxRpgwZzZ0wEhRkrzb63bR9deO49L81slhHuKi4tF+Afr0OFdQWwX7+H73/++qdUcDCzWWfW2OoqetMKmVQb/qXArDRVukc/iqXicGWDBJOFNzNodsGAywyoc9ZLiW/dUU1z5RD8nhvYXsGR0spBwlZGRZnplDyS9LEgaRfKop/wLd8DSpzDPYyhIaetuXUqtxs71u+dAr/OwrGmjNTmbBRZMlDgWjYyfFrAASuBd0QOFgWOGUIv19/t3rE78mhI6dTblXWFgsU7c/NBGYMEBgl5AOPHh5CU9KwgHpaSkUO/evQWsIFzka6IQA4v3XpbkrBz/J99qeFd87SWkJSSHwjDhWgks8KzoQcOQSSOoxZoD/imyN3s9zVk509RkIYGltKy/SD42U+xNCqBy++23G2ouqae9e/dSYlK86cmu3/CB1GLjMf8VLpx9kxjTt3UgakLV0Ea9mbwBFhSQGzt2LD3yyCOmc1jgbUIisBZUSUN147Y5uX7N+4kvHU07DtxkGDgZWKwXLiD69CsS4aEX3/rYCSoPn3mJll2/nkZUjAwYrIQNsMhVPr/85S/FSW/QoEEuFf4mTJggGh3ifiuymhlYfOvcnNSnTMSq/RUDx/O7q72CyrvXLFvhE7DAS4dicXC9IvnWrJdFD1iUeSt6oYwO+T3ESgpLC3Xd+hj1GFhkGlaUdVgQGsPEadZLAs9MmzZtvEq2lUKjv8KSfF0wCMhEjNUuhe5Xu+jljZgBFkAiyjYgfwWF5MwCy9LV08Qx5ykMg6rHSSjV74fGkZFLttOEuRNNhYIYWPwjnMsmTqykhMRE6t4jXxTSzO/Vi2655ZaAf66wABYcrFgPjh8skmpRZwVLspBw2759e9qyZYvopWA2rsfA4h9bfXsdxUzyT6l+PC+e39N7GD25SpTq9xZYZK8bSIY1fAUWdd6Ku0m2uwMuABmWJNnesF/Aipm8Fb3S/N7UpHnhhRdE5VB31Vk9fRcogKXXS8iTIRyS3rcfRRz+qaWw0qlfX0O1RAAChf3zXJY6mwGW6upqAX0Y/6ysLI+PUwILiv8BmDwts5Z2W301JZdVWAotUTccoMGVI72CFQYWbeDwJq9OS++//75l8yYDiwJYMLCHDx8WeSvSs4LiPpMnTxYnROxj1ZpxBhbfbcjUORS1co/FV2l14nmNhqeQz4Kidt4Ay+jRo0VVUQjeFixtNtNyXg0seo0F3V2ZAzLEShdvq+A6HhdbMcMxWVR4NVHoleYHsABcjApNSdE13ZuQEMYcibsICXkLLHL8BbQceNt3b9XGYw6gLDZc+Ay2at0smrdsktNDZKZwHCBFwt7OnTvFudZo4TiEgo6c2mZqrHY/sKkBWiyogBu1crdPsBJswILJPdATvOwlFMpenrAFlr/85S/ih43nhzsUXpW4uDjKzs4Wqw6eeeYZARiQVV8gA4s1NvqahRQ3x5ouzrHTq2nS4hWmK/ACWtT5LJ6ABZMkwkFK4QrXzISrBBbABzwr6kaDRgzLcxM6Zok6H4bDbA5QaXHdbopLS6eNdSt9ghUtYMEJGxBhZMLFWGLCRe6ZcuI1AzvwsLjr1mwGWtpmdaToaasdY+mFB+HByxRTPon6Dhto2GPhsnxYsdTZKLCoS/HL8XQXlpPAIgv/eQMLgJb49AxqsXq/d4m4By5QXI++VDF9jE+wEmzAEgxiYAlSYMEX8rOf/YxOnjxJPXv2FF4VhIJQKA55K8ePH/fLoDUlYOlVWETT5i9s1EnaKptz4wbR7wcnMK9gZddL4vGLN97qdX2YyplzTAELOuHieFbKrJdFCSyYNDB5eHvCxmqi0jGDqWVsnGPCrGzok4McF7kEGknO+H/1IVHBFv1hRs8YY9ibYxZYIORVIAnXUz6LEvTMQp8My8lQlK/AIm3xjddQTGq6KDxoKEG8/rwo9Z/UsRNt27PGp+9RLnU2CiwS2JRjD68yysW7A5bVN19DnbpkeAVWytDk0MkVoqt3xOqDDhD+1FCoLGrsfGrTKdO0ZydcgKWqqoq2b99O69atE/mWDz/8sPj+0EkZ8xcutlGlHUIuJsLN+E5xn5HPycAShMCCL+7//b//J5o8YaJo1aqVM9G2Y8eOosfDE0884ZcvrSkBS3qHTGeILaVtO7/ACxJkU7vlO05kC8XJ3+hKIOyfVWC+uaHa8Jk231lvGFhQ6hp1C9xNvkaBRS5tteLEDU9Nbf0a6jdsAOUU9qaY5OSG/iFpqeJ/dAr2VG7fKmCBcP7wdA5R5lyowxvuJENBcl8rgUXCA3opYexiu3Zv6D00c0ODJ2v5HnE7esQMapWaQem5XQVwWgGA+P5GjhtI7TukeQQWeLK0loIjpwoTnN65D8DSqXM6HXp4iyVjBc/giMnDBQhjWXzzqTc0jNWWxxugz3EboBzVOkUchzffuthnr0ooA8vBgwfp0qVLtH79elETDP9LYMFf/H/x4kWxL/rcYT/8b7Q5LwNLEAILmhoib2XcuHHOCVUCS35+vpg8vFli6UlIZjp9+jTNmTOHHn/8cTGxScNEHm4GSNHq8mk1vCAsA2jAcuRWaZmOCeI6irh2a4NnQJrjxIftuB/7KSHD19fGUmcJPu6ABccUVqRoyUzXZgDL/Y/eZipvxaw9fKaOHn9xr1+e2wiwQAjT6o0lwhnIBVI+zij0Sc+ClNXAok7KBZBgJU3fwUVUNnKguI2witEGlmYMS52TWid6BBb0gNHruwSQ0QO//gNKqF9JD0uhQXksoPgcxqdr7zwaNWmouA1QVtcVCkdgwYUMjnd3sCABRf1XelhQ4FQCC4S+RQcOHDAMIAwsQQgsaGsNL0ffvn2d9VZkv6CKigpRet/bIlTuhJMplhDCi4O/cHtLw6QXbhYZGeW2PTnABSXv3dU1MWso7Ialx/NWV1NuSZnT8DrY7qnwm7eviXwWFJdzByyYINxVY8XxYaSAWvsOGaLbsTd5K4ZWc9y5yvkd+Rta3AELPCFY6qzVfRleANlAUgl9nrwsylCQHcAiDeMoG8LdesdKv70OPGVXt7xaFMB0J8CgXin+mpoaEbpUfyf4PhITE2j9lgWml4CbOv7uWuUcq8deqHe//+XwARbZx86dtIAFgCI9LE8//TS98847Yt/Lly+L3wK24TYDS4gBC1b6/OMf/xB5KwATgIP0rCAshFwWlN/HgeOvTpJNNSTk79CQ2tCQEQ0a9TpaW22ojDts7HhdYFHWXtETJgSEKD15WeLiYmnRyiq/Tq5DKopp4rRh9Or7DwUMWCCtzszukkPdeVnUoSA7gQXjWDl9mBjXx17wLwS2S20jPHl6F13w9GL89MZc734k6A4fPsxtt2arjr+hI0vE8ffKe9/zsH/TSrqVeSkAOvkXOSu5ubni/IrfC/a5cOGC8++UKVM4hyUUgQWrgn71q19RbW2tWMKckJAgPCswTCZml1QysBgDFrsgRWmAFQlJj/7wNVteEwXlOnburAksytor7uTJy4ITEuqO+COcEAjzBCxaY+IuvwWwh5CGlldGHQqyE1jsNCTdrlmzRremh5F6H2oPDC7yMH7Xzr3G78BizniVkJUC1PvafoaBxSJgQQG4YcOGiavYyMhI0dSwZcuWlJOTQ9OnTxeuNK0THQOLd4YwjJ2QEkgPizRUdtQCDmXtFU8nDD0vC658URk3o316kwIWjAU8Ix999JH4312OhR70aYWCwhlYkMOiF/YxUiAOgCLHS+mZ0mt+yMDCUmvw4MHCswMHQTB4bEIOWJ5//nlq3bq1s6khBhOhILjWdu3aZUvRHq7DEt7Wt7iEunbt6rJEWav2ijvheNTKz5D9djx1aw43YIGQkIhJE0nr6hCRFvQpQ0Z6oaBwBxYALj638iJMluL3NN5KTxU8K7LPEAOL/4Qqs56SbhlYmhCwoGotAEUm2uJvfHw87du3T6wassMVxsAS3oYcFlzdAy6ktGqveDpxwZOilNJr0BSBBULCcocOHejZZ5/1uK9yvPRCQeEOLJDSUwKhICbG0ch4o2z/smXLXFZjMbD4T+HQSyiYFTLAgtwVZM1v2rRJhIGUy5hR3RZLjOFdQW0WBhY2X4EFV0lKSNGrveJOSi+LusdOUwUWeAzwe3VX2ExKelkeeugh3VBQUwAWaMGCBeIzwuOkl9+jJZwz1X2aGFjCG1iwDNqqXkIMLF4IXz7gAM+B/BW5fBnAAniBq/61115z7svAwmYFsEjoQKhRr/aKO0kvC0AHf5Xev6YKLPgN47wCADRSIG7p0qWicrWnfcMdWGS7A3iSPcGbUvDGoAGsshM2A0t4Awsn3QYQWPDFw7uCderDhw8XPz4AC2AFybYAGFS7/eSTTxhY2CwHFkwUKSkpIobrjYqLi0WpbbV3pikCi7LGCsYXITdPj0GtljZt2nisqRTuwAIhdyU5OdmQdwrCGCNp99ChQy4F5hhYwhtYwlkhASx/+tOf6Pz58+IqVS5jlt6VqVOniquO3/zmNwwsbJYDCyZZTBJlZWVeXbUAqFHcUK2mCCzqVT6eSvfL/Y0U42sKwIIr56SkJHHuMQKHKNiH51CX8GdgYWBhYPETsOBA/MMf/iAmELiRo6KiBKzgL1YL3XTTTfTyyy+LfewSA0vTARY5aZptzCcnUeStALQRHmrqwKIu049JFdu0WmgoVwUZ6UDcFIBl79694hhEeBLlHdxJXelW5sDwKiEGFgYWPwMLTlhYVdCuXTtnR2bACmqvGHWPMrCweZvDIhNnzRQlVOatyEqvTRlYMB644lfvo7VsF1KvCsJqF3delqYALFjKDFDRGzMprWXPMjzEwOJf7dy5U7Mlgp1C6Q+zCwQYWCwEFlxl/eAHPxC5BAwsbHYBi7oAHP7KJFp30tpPvcqoqQGLu7CO9ETJx2sViPPkZQl3YFEDH07cyjFTHnuAFWWSrRQej8J9DCz+E8Y/0AmvXJo/wMCCRoYvvviimDzkkmbEcpHM+MQTTzCwsPkFWLTCQForftTS8sSon6spAYu7UvtSMmSBfAtMrFqrgtzVYgl3YNFqZqgM80i5C0nIkBKX5g9vMbAEEFgw6Djhffjhh6IBVJcuXQSwYIUQClChaiYDC5s/gEUr90ROjsqaKkbuwzEM4JbVc5sSsKg9KHpQgzAHVgXphd3wXWCfplSaX0qrlYFc6iz3w/jAu6LnhZJeqtlzZjGwMLAwsPgLWNB1GV4WvNlt27bR4sWLadWqVeKqQ7bhZmBhsxJY7r///kaVapXSCnF48r4ovSxNCVgwiWol1mqND0K97pJrATTqlgfhDizK/BO1lPfJHBd3wvgNGVrOwMLAwsBiR2n+YBADS/gDC5bLu0vyVPYFkv97ym9RelmaCrC484ooJUNBt99+u9vVWHrPF87AohX6UQphINSowjnW0zijeWdG+wy6pW45A4sfhOMTYxxIWGBgYWBhYGlCVjSwVFRPVjY/1JLsvIzJVqvZoZbwO0Augr+BBRCB0Cns2R/dGzBgQcEy9L3xJDkpQ+rlz2ppeVnsABaMIxL+MaaHHt5iC7AAcpGv587rdOnSJdHuACXZjQj7Vm+cS29f9u8xIcfqB2/c06Q8LJx0y8DCwNJE7dEfvkZPv/Wura+Z0z1PFIozIkyc+fn5hmu0yJVH7TtkhD2wGJlsIVyRKpvzeerMrOVlCVdg0VoxpRYAD52Y4aHyNNZQ74LeVD6i0AEsJxlYwlBoxyA7czOweAksmzdvDptBQ5E6fB4GFv/ajkOHnZOundCSlJwscqWMCCGhzMxMw/tDgJvEpMSwDwkZKbantypIDTFqqb0s4RoSGjNmjNvO1vfee6+z7L7eUme1plRNdhx/cX4FFs5hCZzwmwpX2QIsSIxFTsDx48dF3RQsRQ5Vw/uHi3ue47OqQwYMGdYaIGXYuAk0ec5cevPTL2x5zZd//glFRkXRCy+8YOjEgJAQQkPIZ3EXxlAKx02LFi3osRfqwxpY4AXRWmWllDIUpAV2enlEai9LOALL22+/LVYH6QGIVm0aufTbHbSgDkvXnEzae/8GBhaWW+3YsUOszj179mxQeGxsAZavvvqK9u/fTwcPHhRXBKFs9913H919993iBKkOcTFkhL6tr62j1PQMQ/ChzFvB5AF48ZT3IhUbG0PXVc8MW2CB5wkl5N2d5Dx5UdwVQYOUXpZwBBa0HXGXSKvlfVEvddYDlqpZFTRy3MAg8bKED7AgjwjHZLiEY9C3Dx5u5MUEwxjbAixNRTzhh77l9syn/D59PQKLVrhDdiA2otTUdpTVtX3YAounNgbuCsQphVVXmIC1cjOUXpZwBJa8vDxdWHMX/nG3DFoCy8bapZTevi09+8a9DCwWKtx6CeH3iQjJt99+GxTvh4GFgYXtip1+5Q0BLMpeQlqC9wBgorUawGiTRKwS6l/Wi+48tC7sgMVIs0J3oSC13BWeg5cBnppwA5bUtDYCWLQ+s6ekZEirMq4SWFCHZdb8sVS9cZ5fVwsxsNgvhG8Qpg5HMbAwsLBdsWuWrRAhIXfAosxb0ZORJc4AFuQQ5HTvFHbA4gnaPIWCtKRXll82lgw3YImLjxWJ/VpjZCRPRXbC1iokJ4Hl+FO7KDcvKwjCQgwsVoqTbk0Cy4cOYPnr/33X5Iwn/dC1C5/9ltI7ZIqkW3fAYgRGjECNrMNSPqIo5L0samCRTfb0xsZIKEjrcXq5GdjuOGeFFbC0aNFcJN1qARpyd4xMiHrhNGXzw/w+OXTk1DYGljAClnCWX4CFJ0C2ULP6h07QsLHjxW09YDEa7oHchY2UwLLvgRoaMLggbIBF5vHonbDNhIK0xhRJuOrnxiTet2/fsAEWHBPR0VGN4Mwb2NPKdVECS83ti2nKrIoAe1kYWFgMLGxshg2wAmjRAxZPAKIlLH9HjQx3wIITLsJCcM+HA7Do9fqBvAkF6U0IanXs2JGGjCgJgnwM323U+FJqnZzYCFg8lejXE74TlGOQj1MCyxsfHqf4hFg6/8ExBhYGFgYWNrZgN4SBEA5CWEgLWMwuWVZKb7WMElgQEkJoKNSB5csvv9BNtvU2FKQlrdL9q1evprT0NkFUDM07e/mdo5TSNoky2qe6AIunlT/upB57JbDgNatmj6QNty0OIOyFD7Ds3btXeGIDCSwnT54UF1hc6ZaBhS0MDYm2SLiV/6uBxUxROLX0miKqewn56mW5+AvvzdcJQwILVqWgboiWfAkFqYXcIORm/PGPf3Ruw3MnJsXTsbPWeaoufmL/mAIkxk8ZQu07pDmBBccQgMNdbRVPUgKPGliOnq4VuSxew95lH48/H0GJC8e5insJMbCwhbH1KiyiY8/9UBNY8Bsw+ztQC7ACaFGGk9TAUlu/RhTyCmUPCxpGanlQrAgFqSVXBykr3RYP7C08VaEcFirs34MOPbzVpVtzdXW1JVftcqnztXOvcQEWGFYLHTu7MyTHjIGFgYWBhS2k7P3f/cmrx5398UXKys5x2SaBxUwhOE+STez0gAW5BGkZbei5H90XksCS16ObC0BIWRkKUkvptZHLmoWnykIvi52G4wHF3ODpkMCil2jsjWTl4HHjxzUClhs3zRN1WUIxpMbAwsDCwMLWJGzhmmpas/nWRsCCvlHe5q3oSdkbRw0sMBTxwqQRisCSnNxaM9nWU8VbXydghIawhFoCi8wHCkUvy9LVU2nJ9VPFewewyGXJekvEvRGeE99VzbYlLmMEUAYsBTb5loGFgYWBhY1N11LathNJt2pgAax4m7fiboKFxwZXzVrAEqpell0H1lLLllc3OkH6IxSkNQFjUj98+LBzWTPCG6HoZcnq0p5OPbdb3AawrFmzxi8rTvoPKKF+JT0aeVOwOgmhSQYW74V2EYHuJcTAwsDCFoZ26NQTVDpsRKPtGZmZohu3P4SEUcBQRvv0RsBipZfl8Rf30pCKYpo4bRi9+v5D/l2GO6GMOmV1dDlBAs6wYsgfoSC14LUaOXKkE1iw4gW1RayuyYJxrJw+TIzrYy/stfS51YmvKM2v10PJVyHptluPLLrj4FqX94D6LwPLCywJC+H4GzqyRBx/r7z3PfawMLAwsLCx+WKjJ1fRrsNHXbYdOfsDik9IsNy7ohQ8D1FRUZrAgmWtyW0ShbfFl5P4bXeuEl1WYZg8/DVZ4H2mON5vSf8ilxMk6s+gDo1d6tevn0i6fftKbRF/NPbDODZr1kyM6a13rLT0ueXSYvl/y5Yt6ZlnnvHLWAFYkLOSk9ep0Rhh3M6e2+/z57ntrlXOsXrshXoGFhuFJO1HHnmEgYWBhc1OAzzISffln1229Lnf+OVvRDhI1l6BITSE5od9i0v8CixQfEI8zZyn7UmRTel8OYljifSwcYOoU89uFJOc7BzHxLRUyinsTVVzx1tSrA45I737dXMpze+p2q0/hPoXWNYsJ2B/5QMBWnbuv1Hk7Ug7/fwe36GvbRL98OIR8f+qdbNEMTdfljF7AhYk3eK9Y1WS0qOydPW0K3k0vntZHj5TRzv23eAyVv4IdzKwuMpMcUsGFgYWNj8By3M/+VCs6rHiuTffWU/T5i902VZYWiZe01O3ZiuEHJaeBdniJK4+AeOkjlwWb7wsG+tWCiiJ7dqdIhbVUUTtMxTx4GWKOPunBjv8QcM2x33YB/uu3jjf68kCbQVWrZ/jBBYraoZ4IyTdlpb3o34leZZ7WeD1wgTfb9gAcSzGZHamuN79nYb/sR33IwfE7Pf2n0ThkwJ+kIOjLhznD2BByGz2gnEuHZtx7AGevAUWPB6eooxuXcWYxOX1cRmrqNYpFBkbR8MnDac7Dq5jYAlyvfrqq3TgwAG6dOkSAwsbm57B84FwzaCJVZSYkSlOfq3SMimmY464HZvSjopGjhfHm9JLYtTUtVeUq4XsApbvPbFDLMPVuuo062XBRNcpP49iK2Y0QIkEFE/m2Deqcpl4rFmPi3jNzhkupfntDgUpgQU5LPOWTaKVa2e5ell8yGUBzMWlpVP0CMe41pxwP5aO+2PKK6lVcjLV7DAeMkJOjGyAqVWHxZ/AArhC7oyyBsvA8j609/4a016iGUunUnRyCkWNnUsR9a/rj9OJX1PE6nsoduBoatMpU3QttwNYvv32Wzp+/Di98847Hvd9+umn6amnnmrkJbx8+bLbx509e9ar9gnBqubNm4vzLUoWBAMUMrCwBR2oLFy7gSLjEihmSBVFrD2iPQHDa+CYIFpVzBb7zlpVbRhc1LVXlI0P7QQW5LBgsoeXQn1VDohBLouRE/buBzZRfNduFLHrJeOgojbHYxPyeovnMjpRSKiSwBKIUJAaWDCOhf3zRBKrL14WwFtWrzwBcxEnPjU3lo4JObZipoBAT/kgshQ/lhPLZGFlHRZ/A4sET3h15JJm4fGpKDLsZcH3n5CeTpHzNlPEk1+bG6sDFyi2eCgNmTTCGRLzF7BMmTKFtm/fToMGDaKLFy/q7nfw4EFatGiRMHgX5PG8Y8cOsc3d8R1uvYTYw8LG5mbVTmq3fGo5ewNFnP7K+EnPsW/kPJw0M8VzeHodpTcFYSbkrSCnJRDA4i7fAmECeeXtDlZal1aYGy834xhfVE437Vhl6Io6vX3DEmxMWKVl/albt262h4LUwIJJGOOa1SVDTMDeeFkAKyl9ih0Q96Jv4+mAwDY9erttFyAh5ey5A2JZs4QGO4FFXTjODOjVHbyJkooHU8TRSz6NVYs1B6lTv75eQYsRYIF3BcACkICXZP369bpQgf2++eYbcXvw4MHO5wbI4D53rxUMwIJeQupWIOEiBha2oLCNew9RXPEIccXl9YnvwcviOVbfXuf2tWTtFXhkkLeiDA0FAlj04AQTJ0JG7iaLxKLB1sCKwuLKJ9Kte6rdThLKVgIAlrS01ICEgrSABe9J9uR54wNzXhZ4ZgSsHP/UmvF0HJPJfUp0oQXhmCOntonlxPsUYRi7gUUdClLntmjZzXWrKKF0FEWc+tKasao/T50KzUOLEWBBGEjpLZHwogcsyv2Uz11VVRX0wBLOYmBhCwpYSRg527IJN2bSdTTnxg26XhxZewUND9H4UL1PIIAFoQHAiXqps56XBfslZec25ANYCCtOT0vXbm5XvyBhGJO79BAlJCYE9CStBhbY+KohYmI26mXB503M6WYdrChCRK17FjiLwilfD16VW+oa4EoZggkEsMBbhtAQ4E6+N72wEGBZwIrZEJABaOnQq6epirsMLAwsDCxsthi8IVbCitNLMGeDJrTI2ivqvJVAA4u8uscqF2U+i5aXBfd36VfgW86KgdyC1JxszRUvzvd0WVbnTaGi4n5BByyAwNy8To7Jt96jl0V8jtwcx+d+2z/j6fiuuvTr4zIRoxQ/wkFIXFZ7FQIBLOricdL7owXLrf0Bdorw0JBJFYZzaLwJCSEfxUhIiIGFgYWNTRhCMUl9yiwPaTihpaBhmbJ8PVl7BUm36ryVYAAWZShDz5sBW752DrWcusZ/sHLFmi/eTpPmTnC7ggkT7qwF41zqsAQLsDghsH8eLV8zXawg0vOyDJ08wjFRHvDreEbO20LTl1Q5wyyAKHy3tXsal8MPFLAow0HyWFSCA8Cua2Efitj5ol/HKnbkTKqpW2moJ5QvSbdIKlVXY3aXdJuTkyNWAukd6wwsDCxsYWiAhba5+eaW4HqRP5DieA3ZKwi1VypnztHMWwkWYBGhjCkNoQx1nQ45YcSnpbnWVvGXOUAyJjWDnn3jPpcJKz4hhs5/cFzkrsAjpFzWHGzAAsMy5+nXjha1bbRCDfAYxWV3tz68oTWeaRnC04MxQyVY5AFpeRICCSz4jkU/JiQfKwrayWMxzh+hII3fbnx6uqHQkC/LmrWABdJa1gyQAcBoLXdWTqiBXtZ87733in5lVrwHtBJhYGFr8la1ZAW1WFzn90kXr4HXwmui9gpMK28lmIAFEwZCLsocEvyPCQR1QcRSW3/Dio6XRa5qwXvs1CXDuTQ7mIFF1jcZPrpEs7ZN4fABnmusWDaeO8R4Akrj4mN0w1SBBBblUudJ04eL71x6OjK6ZVPEntdtGauoScvpesfx7ik0xIXjXOVrLyGsMJo7bz5lduxE7Ts01MAqdZwTDx06FHDPEQMLm+0Gj0dsWqbfQkHqq1q81oPPPC9+eHp5K8EELHLCQLgAeRhKLwsq0/rVK6VxpRuTnOKcsNB/BuAEaJG9b0IBWJBQ2iUnk9qlJrtctdvmXVF5WWLjWtEtdct1v/9AA4tc6jx6YpmAZUCDbd4V5bGX4jj2LjGw2AUs8Dj1LSymzbV76MyLb9FbH30h7J6HnqBps+fTtGnT6a9//SsDCxt7V/zpZckt6EvpjqsFvbyVYAMWdSgIltW1PUW3TXN8pq/tAxbkE3RtqIIrk21lKEhZOCzYgUWOZ/uO7Vy8LAtXTafms262dUxbTFgm4Mmd5yAYgEV6ptp3TBWVcAuHD6SImuO2jlVcwQA69PAWBhYbgOXo0QepZEAZvfjWR05QUdv2PfdSUXEJ/eEPf2BgYWsaJkrt2+klOPFruqp5czr9yhuG3l+wAIs6wRWTjCgRbyOswFpWLhfvAWEMeFWwqkX53kMFWGCTZw4XjQWllyW3qIAiar9v75jWnKDcwgJRgh9jp7USK1iABZ6ptqnJNGnGcIqMi/PbyiDdENrsDbTAAZXu4I6BxXdg+fLLL0UI6NV3f6ULK9Ju3LiNVq9eE5AxZ2Bhs9WwQkf0A7J50o3LK3KbaBuswILJDN4MrHYZN2M0RSzfY/vYYYItGFwkWgVIaFGXZg8VYMF4tk5OECuGcDsyLt66wmcmALp5dCvxvcJaRl4twpX4K7e1ahXtmBRWixUnjz76qDgeYVZ04jUDLLDb91xPLVteTTFdbAydSat9hnKKCkIGWBBSwfcUyN+CN8CybPl1tGHrLnrz558LKLnr0EPU7KqrxHG5dlOtczvs9ff+m7JzutGvfvUrBha28DYkvLasvM72Sbf5rA20ZP2GkAMWWfcCoZiufXo2dFq2G1h2vUSt01LFe1CGgkIRWGCHH7lNTMB33XsTxffub/8kjOKGHbs0KiQHgMJYwpJTkmjNmgZgmTlzpujRBIuKihKTCP7KbfPnzxf7wZ588kkn3FgFLDDAaouKOfaP1emvqHlkFL11KbiABeXvtVYXBYPMAguSbLvm5NKPf/aZAJJX3vkFFRT2p3MXLovblVNn06kfnHfxstTuvpemzZhp+7gzsLDZahXTHSe91Yfsn3S3PE69ykeEJLBIKIh0XHWjEqjtY3f4A7qqeQvRuXXmvDG05PqpLjZucjllduzgnDTNmJx0fTX0MUppkySACpMrJnUjFtm1Z0CApSE3Y6vXOSzwtEgwAazJ8URXXTkm8jMmJia6wE2v3r2ofEQh3XlwnTiulDV+9AweqWYzNwRkrFrGJbgt1x8IYOncubMY24KCAtqzZ0/AemhpCbVgAK5GgQXLoJFQK70o33vseVq3eYfzfyTcrtu83cXL8oPX36fU1DT67rvvGFjYwtdyS8oC5iXIKigKGWDB6iB5tS0tJinR3twfaVvPOE7OzahsaN9GsOIrsMhJ11e7+eabqbS8nxgn5F0YgcDuPTtTZL+hgfGwDJlE2+rX2JJ0+/XXX7vATe+C3gJYkNANwMNqNAk3gD0ZlsJKMPkdFw0ppojlu21P+Ia1Suvgtut1IIFFaYDmLVu2hFzRuBtuuIFu2HCbE0ieee09KujX4GHBbXhY1MACS0xKot///vcMLGzhaxnd8wPmJYhOSKLl6ze4GJY5o5Cc0uITEsSVk9mr/E6dOhm+svdkqNEhJw5nrkN0VGCApXSSiGfrVR0NtZCQc4XQiskUMW1dQCbh6IqZdIuiOGCwJN3K7tsw5CpJYGnrANKI6w8GBli69qSxDijWgmUYCvABFLwBZmnwPJn5rcvQnJaFWgLwqFGjqf6+4y5Acl11jRjXDh2zaNmamzWBpU9hMf3oRz+y9XfPwMIW3iuElOY4maiBBT2FUL5fad3ze1F9fb3pq3wzVSHNhoRgnfLz/Ns/SKcWRkSreIpv2ybsgAUrn7ACioHFs42YPIIiVgcGWCLbtafJM0b4FVhw/Jj5raeh2rQGrLRq1UqEiELJy6IFLEq7ZuEKOvjgYwwsoWQf/O4bBo5QBpYTv6bIuISQzWGB5ZcWilwcuz1TEYvqKKewt9v8mlAEltr6NRRbXhmQSdjXHBa7gWXp6mkBy2GJSnbfuDLQISHkCM1zzH1nzpxxAlAoAYs6JASTibbIZ8nM6uJMyOWQEANLkzLkkdjuJbgy8QKWQhlYArmsud+wAWEHLHjf8QWBWSUUm50nirGFCrDcuGk+RcIbFYCximh2VdCtEsrLyxOrtx555BGX4z4Ymh9iUjdz7KiTbmG33XGAmjnGHSGhk8+83ghWOOn2ir356Rc0b+VqGjZuAtXsvEtzO/7if0/P9fRb79LkOXPFY5Rde5Xbla/BwOJ/w0od270EV+o5IOE3lIEl0IXjwg1YMP4xae0DtPIl3u3Kl2ADFtvL8isvNNLT3JbnD6Y6LKHYrVm9rNmI8bLmKwaQ2HfsUXr0h69Rn5IBTtDAdsAFtl+/aYuAFk/gI0EFBtedhJzMzl2cr4F9cJuBxR6bvmSFCDHYDiyrD4kl1aEMLEiIjGqdYnsIo1Wq+/caqsACa5vVkSL2v21/MbTCgpAoza+sEROQInvX1dPYGWNCpnBcKAILVFk5iXbefUQ3j0VtPfJ707vvvmv75ww6YAFMyNsAjR2HDovbAAs12Lh7HsCIEmoAOfCsqJ8L+8nXYGDxvyHJNbZ0nP0rDSpmGz42gxVYYOm5Xe1dZYUr3LQ03YTbUAeWqfPG2w7QKGLYUG7+VMgAi7ONwTZ72xjElo6mOw6uc/teGVh813vvvScgxIiX5Y4DD9LEiZVcml+Ga+RtgMTLP7vsEsaRHhO5H7wj0ksCKFHCiPK5cJ+8jX0BM4AV3DYSXmJgscbQfBCFoOyv5ZBJz/3kw5AHltUb51NU5TL7Eh4dr4XX9JQLEqrAsu+BGoorGmJrqMNTwm2wAsui62dQCzSKtLGzdXRKW7ehMwYW63TLlq2O39BMeu0n/6ULKw+cfIZ69uodkLL8QQksSu+KEjhwW3pcACUSMgA0gA/cB/hQ5qoowUcJJdjHaGiJgSUMEm8PXKA2nXMMv79gBha45uOxpBLLjW1Y0ozXOv/B8bAFFlhGt2yKqH/dnmOx/jylO17PXYgjWIEFIcnYtAzbwkLNl+ygyrkTPI5VMAHL2bNnTVWZ9Rc0nTt3zqv3gI7N/QcOarSM+cmX3qa1m7Y5fuflAevUHLTAooYVmBIsACnqkJD0vniCFTxWmWgLyOGQUHiHhaInraDqbXVhASx2elmMeFfCAVhEQulAexJKRYjD8Xqe3lMwAovodj1vIrVYvN3/Y3X6K5EQ7W45M3dr1pY3zQ+VeuuttwSYoL5NUclAat8h03E8dqRV16+mv/3tbwH9bEEHLGpYkRCC3BYJHjKhVv0YNZwo/weoYB94VpSwg8caWSnEwGKt2Vbx9sHLlJCeSRc++23YAAsMdVH82uJg10vUpV+B8OiEO7DAsnr1cHzmFwOebBvswAIvS0pOLkUcveTfYnHzt9C0JVWGxoqBxVpgUQrdp80UxGxSwAKgAJgARmDKVULwhGClD7ZjH7kdYR2ZqAsvDPaR8ILHy+dSAg/2k6+DvzJPhoHFPtt1+CjFDKkKOu9KqAALJo7k7Bz/hIZO/JqSsnMNv79wABZ81viOnf0X7sCY5nRz2xMnFIDFlvo1jguZDr160vkPjhl6Pwws/gOWYFNQAQuAAh4QpanDOept0muivI375b5KU8ORehsDi702bNocarHmkF+vaI3WXgk1YJETR3JBsZgMrZxYW5dW0O4HNpl6H6EOLLCNdSsptmKGX5aNxw2ppK17qg2/l2AGFljlvEqKXFxrPbSc/oris7vTqed2G34vDCwMLFzploHF74YwTWq3fJEU649QUEpuPr3880/CFlhgx5/aZR20OJ4Dz2UGVsIJWGCzVsyi2GnXWwotcePn0UzH875t4r0FO7AgVDhwzBCKuuFu68bKcfwlFpfTXfebO/6CCVgQQsG5I5C/BQYWBhYGFj/Z2R9fpHb5Rdb2F3Kc+FL6ltHpV97w6j2FErBIaEnomEUt1hzwfsxqTojnwHN54+kJF2CBLai+luIrpjmu9r/03VtQMZ3WbF1q+j0EO7BIK580kqJW3OU7tDguMJL7lNChh7eYfg/sYWFgYWBhYLHNUB8lvV+ZNZ4WB/gk5eR7DSuhCCwyp2XIpBEU17u/uWRcx76xRUNEryA8h7ehqXACFthNO1ZRfNduFLHlMe+Owy2PU0xqBq3fvsqr1w8VYIGNnDGW4vL6eF0xGMuXkzplue2txMBiXHv37g24l4eBhS3soQXhoch5W8WVqVcrCxyPTc7KMVwgLpyARQkPWEGEEv5RY+dRxNojDQADDxYMtx3bcB/2wb4onubra4YbsMBOP7+HepUVUiyWPGMcPR2XuN+xHwAQnbWffeM+r187lIAFdvR0rWhzII652u8b8qhELK+n2K55NGnuBI/F4RhYWAwsbEGX0zJrVTUlZOeL3j+G8jKwj2PfWMdj8Fgzy5fDEViUHpcNty2m0jGDBZQkpqUKw21sw33eelSaCrAo67RgzFrGxgkYiZi5ocEAf1duYzvux36+AmAoAou0mtsXU25RAzDHlFc2jA9aH9ScELebT71BeGNikpNF9/FjZ3f5/JrBBix///vfmSwYWNiaUl4LGhVGxiU0FJjDSQ8AgwniiocA27AsulVyO7GvLyGgcAQWOy3cgUWZaAoYWXL9VGGAP3kb243UrAl3YFECc239GjE2VXPHi5Ajbq9cO0t4Y6x8rWACFoRj6urqwjIcw8DCxubBUBV3+foNAkqwRBk2aGKV2IZaLlZ4VBhYGFiCzUIdWOw07iXEwMLAwtZkjYGFgYWBhYElVIEF/YzQfZmBhYGFjYGFgYWBhYGFgSVogSWcZTmwLFxTLcrms7GFquXl97IFWBDjx2Qf6la9cR4DCwMLAwsDS2gBy6FD99Cg8nI2tpC2wQ7zd8Ovm25aJyb5cLHde+4K6IkMwJKb19mZBBvqFp8Y51dgKR9RFDZjha7CDCwMLKaBhcVisQKhr7/+WkwU4WL19fV+m/SQ3yAn1nAwwGqwAAIDi3/FwMJisVgslgVCL6Enn3zSCSyoyXLy5EkXgMG2BQsWkGOO1bQdO3Y02t/Mc3Tr1o3OnDnDlW5ZLBaLxWIZl1Z4+a233hJ5clr28ccf+/wcf/vb38JyLBlYWCwWi8ViBb0YWFgsFovFYgW9GFhYLBYrCPXUU0/RO++847Lt4MGDYlVMWVkZffPNN87tCANgu+NETpcuXTL8GuvXr2+U66D3GixWoMXAwmKxWEEmCR8XL150bgO8DBo0SEAEYGbRokUCNr799lvKyckRoHL58mVx21PCJZ5LvoZySbDea7BYwSAGFhaLxQpCYbWI0sMCzwcgQkqCCfZRekpwWwk67gQ4UQKJ3muwWMEgBhYWi8UKQgEelMAyZcoUFxDB/4AJ7HfgwAEnWCgfp/TU4Pb+/ftdAEQ+h6fXYLGCQQwsLBbLMiEkgTACDLeVwqQ6efJklyt4CPkX2L59+3bDr4MwiLp9AratW7dOPJc6j0PvtYNZVgALxgQel88//7xRPozyOTy9BosVDGJgYTUZ4YStNJzMpTC5qrcp9cUXXxh6DTxe73nwGupJXD631mQSikIIAZ8FMJGdne1M2sT/SOLEffgrYQOfHY85d+6cgA0j0IJ98NyYpNXb8Rx4bjwnJmn52gh9YCLGX7xWKEgNLN6GhHC/Hniot3NIiBXMYmBhNRnBNY6rbBgmTQkhmOQwAWI79lHDCSAD93kS9sPzYl88nxJOMLliG0w5IWAywWvicXgfoS7lZwZAyAkXE6O8T3phMBFiXI4dO+Z8jJEJEo/H49TAgpCHfCyeUz4vXlt6XPDYxYsXh8QkjBwWpRdJnRCL40Ur6VaZl4L/8RzYBzCiXvUjn8/Ta7BYwSAGFlaTEE7cShBRXrkCFqRwklZPhBJCPElWmZTPj0kZwusCVDBpwORtSAk22N/fXaLtlPz8kBImIHllD3hQhz2wHd+DDOEAfHBb+Xg8Rvk9YQyV3gLl/XqvHcySuScwZd7J8ePHNZccK1f9SDiT2/B4eXvSpElOr4zyNZQeGb3XYLECLQYWVpOTvOKUwsQqYQb3KSdQTHoyL0JKhnDwV4aS1JL5FPI5laEO6XnAPkpYwn5qWApFATIAYspx0YMG/NUCFoytDO/g+dQgpwYW/K+EmlAHFhaL1VgMLKwmJzUUYHKUoSJ1uEZChxJYsA+uSqXLXAkd8nGAEuk5UYcvcFvm0SifVz0Jh6ownsqcFUgZegCoSWjAOClhRL3MFh4YLa+TeqzwnMrHKuFP77VZLFZoiYGF1aSkDNtIyYkNk5lydQtuS7BQTr4Q9lHmaCglc2Tk6yCXQgtY1Lkx4QAsyjwcjI/8PNguc0qU46EsTibHQwkd2Ib91UnMWmMl8zggZagJSahar81isUJLDCysJiVMnErI0AotyP+l10Um4yrhAhOoOmEU22RoCbfxGPmcynwOJRTJfSBlomioCt4MOWYIC8mVOjIxVMKcMjcCcIFt2F8Ch1wJBHgBcChzM+R92FZbW+vy3eE1ZAKzBB93r81isUJHDCysJiVMWkpgkQmxsm8LYEIZFpLLlJU5GYAN6UmQV+wSYKTXBdvxGClMlBJIlOCjXIqrTMYNZbnL69Fb8i1zgrSeQ73kXP4vvVRK6S0Rd/faLBYrNMTAwmpSAiBoLVsGqGgVFpNhCmnyOdS3sR8mQ/k/IET5OrgtC6opJ03lY7RqtLBYLBarQQwsLBaLxWKxgl4MLCwWi8VisYJeDCwsFovFYrGCXgwsLBaLxWKxgl4MLCwWi8VisYJeDCwsFovFYrGCXgwsLBaLxWKxgl4MLCwWi8VisYJeDCwsFovFYrGCXgwsLBaLxWKxgl4MLCwWi8VisYJeDCwsFovFYrGCXgwsLBaLxWKxgl5ugWX+/Pmi7T0bGxsbGxsbWyCtpqaGSkpKGgOLA1aG5ObmvsfGxsbGxsbGFgw2bdq0uQ5gaeYCLGxsbGxsbGxswWo8CGxsbGxsbGwMLGxsbGxsbGxsvtr/B7IZXg+g+JMzAAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.1背景说明\n",
    "\t本节将利用神经网络完成对手写数字进行识别的实例，来说明如何借助nn工具箱来实现一个神经网络，并对神经网络有个直观了解。在这个基础上，后续我们将对nn的各模块进行详细介绍。实例环境使用PyTorch1.5+，GPU或CPU,源数据集为MNIST。\n",
    "主要步骤如下。  \n",
    "- 利用PyTorch内置函数mnist下载数据。\n",
    "- 利用torchvision对数据进行预处理，调用torch.utils建立一个数据迭代器。\n",
    "- 可视化源数据。\n",
    "- 利用nn工具箱构建神经网络模型。\n",
    "- 实例化模型，并定义损失函数及优化器。\n",
    "- 训练模型。\n",
    "- 可视化结果。  \n",
    "\t神经网络的结构如图3-5所示。\n",
    " \n",
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用两个隐含层，每层使用ReLU激活函数，输出层使用softmax激活函数，最后使用torch.max(out,1)找出张量out最大值对应索引作为预测值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.2准备数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "# 导入 pytorch 内置的 mnist 数据\n",
    "from torchvision.datasets import mnist \n",
    "#import torchvision\n",
    "#导入预处理模块\n",
    "import torchvision.transforms as transforms\n",
    "from torch.utils.data import DataLoader\n",
    "#导入nn及优化器\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torch import nn\n",
    "\n",
    "from torch.utils.tensorboard import SummaryWriter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义一些超参数\n",
    "train_batch_size = 64\n",
    "test_batch_size = 128\n",
    "learning_rate = 0.01\n",
    "num_epoches = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize([0.5], [0.5])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#定义预处理函数\n",
    "transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize([0.5], [0.5])])\n",
    "#下载数据，并对数据进行预处理\n",
    "train_dataset = mnist.MNIST('../data/', train=True, transform=transform, download=False)\n",
    "test_dataset = mnist.MNIST('../data/', train=False, transform=transform)\n",
    "#得到一个生成器\n",
    "train_loader = DataLoader(train_dataset, batch_size=train_batch_size, shuffle=True)\n",
    "test_loader = DataLoader(test_dataset, batch_size=test_batch_size, shuffle=False)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "【说明】  \n",
    "1)\ttransforms.Compose可以把一些转换函数组合在一起。  \n",
    "2)\tNormalize([0.5], [0.5])对张量进行归一化，这里两个0.5分别表示对张量进行归一化的全局平均值和方差。因图像是灰色的只有一个通道，如果有多个通道，需要有多个数字，如三个通道，应该是Normalize([m1,m2,m3], [n1,n2,n3])。  \n",
    "3)\tdownload参数控制是否需要下载，如果./data目录下已有MNIST，可选择False。  \n",
    "4)\t用DataLoader得到生成器，这可节省内存。  \n",
    "5)\ttorchvision及data的使用第4章将详细介绍。  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = enumerate(test_loader)\n",
    "batch_idx, (example_data, example_targets) = next(examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([128, 1, 28, 28])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.3可视化源数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAELCAYAAAD+9XA2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAcPElEQVR4nO3de7AU1bn38d8D4kYgBxW8IxDwjQkiIGqiiJckHBEERQGh8BijdbxEMVapaBQtr8S3sI45xgiaOmW84DEmIAYRDZ4UlxgvJZTiDfQFi62cgIIIYQOG23r/6KHt1TKzZ3qvuezN91NF1Xr26ul+Zu/FPNOrZ1abc04AADRVq2onAABoGSgoAIAgKCgAgCAoKACAICgoAIAgKCgAgCBadEExs5VmNqiKx19lZmdU6/jIjrGDrPbmsdOkgmJmY83sDTPbbGaf59pXmZmFSrAczOxFM2vI/dtuZtsS8cMZ9znNzO4ImONtiZwazGyrme00swNCHaOaGDvePkOPnXPM7FUz22Bmq83sETPrEGr/1cbY8fYZeuwcYWbP58aNM7MupTw+c0Exs+slPSDpPkmHSjpE0pWSTpG0b57HtM56vJCcc0Occx2ccx0kPSVp8u7YOXdlensz26cKOd6dyKmDpP+Q9Bfn3JeVziU0xk7ZfUvSnZIOk3SMpG9L+r9VyCM4xk7Z7ZI0R9KoTI92zpX8T1JHSZsljWxku8ckTc0luFnSoNxjn5C0VlK9pFsltcptf4ekaYnHd5fkJO2Ti+dLulvS3yRtkjRXUufE9hfl9vmFpImSVkoaVESO96R+Nij32FskrZH0O0n/Lml+Ypt9crl1l3SVpO2StklqkDQzt80qSddJelfSRklPS6rL8Pu23PO6MMvfq5b+MXYqO3Zy+7pA0lvV/tszdprP2JHUNnecLqU8LusZysmS6iT9qYhtx0mapOhd0yuSHlT0x+0h6XRJP5F0SQnHHpfb/mBF70hukCQz66VoEF0k6XBJnSSVdLqW0kVSB0ldFf3h8nLOTZH0jKRfuujdxnmJ7gsk/aui53t8Lj+ZWevclMRJReTyQ0kHSJpZ8rOoPYydhAqMHUk6TdL7pT2FmsTYSajQ2ClJ1oLSWdI659yO3T9IzNluNbPTEtv+yTn3N+fcLkXVdIykm51zm5xzKxVN5VxUwrF/55z7yDm3VdIfJPXL/XyUpNnOuYXOuX9Kuk3R6VtWOyTd4ZzbljtWVv/pnFvjnPtC0uzd+Trndjrn9nfOvV7EPi6W9Afn3JYm5FErGDvFa/LYMbMhil4Mb29CHrWCsVO8EK87JctaUL6Q1Dk5x+ecG+Cc2z/Xl9zvp4l2Z0XVvT7xs3pJR5Rw7DWJ9hZF1VyK3h3Ex3LObc7lktVnzrltTXj8bvnyLYqZtZc0UtLjAXKpBYyd4jV17AyQ9KSk851zKwLkU22MneI1aexklbWgvCbpn5LOLWLb5HLG6xS9W+iW+FlXSf+ba2+W1C7Rd2gJOa2WdOTuwMzaKTr9zCq9DHNjuZVr2eZRkj5TdNreEjB2KjB2zOwESc9J+olzbn7o/VcJY6dyrzuZZCoozrkNij5FMsXMRplZBzNrZWb9JLUv8Lidik4XJ5nZt8ysm6KLR9Nym7wt6TQz62pmHSXdXEJa0yUNM7OBZravpLsU9ns2SyT1MbNjzWw/fXMK4TNF85WhXSzpcZe7UtbcMXbKP3bMrK+iC9JXOefmhNpvtTF2KvO6Y2ZtFV2rkqQ6M6srtH1S5ifunJus6I9yo6TPFT2xRyTdJOnVAg+9RlHV/VjRu+7/lvRobp8vK7rI9I6kxYrm/orN531JV+f2t1rSl4o+7RCEc+4DSb9U9ImPDyUtTG3yX5L6mtmXZja9sf3lLo41mNnJBbbpquiC6pOZE69BjJ2yj50bFL1LfizxPYcl2Z9B7WDslHfs5KYTt0rakPvRckW/t6JYC3njCwCosha99AoAoHIoKACAICgoAIAgKCgAgCAoKACAIEpazdLM+EhYDXLO1fqy3Yyb2rTOOXdQtZMohLFTs/Y4djhDAfZe9Y1vAuzRHscOBQUAEAQFBQAQBAUFABAEBQUAEAQFBQAQBAUFABAEBQUAEAQFBQAQREnflAdaghtuuMGL99tvPy/u06dP3B41alTBfU2dOjVuv/baa17fk0+2qPuiAY3iDAUAEAQFBQAQBAUFABBESfeUZ+XP2sRqw4175pln4nZj10WyWrFihRcPGjTIiz/55JOyHLcJFjvnTqh2EoXUwtiphO985ztevGzZMi++9tpr4/aDDz5YkZwascexwxkKACAICgoAIAg+NowWKTnFJZU2zZWcbvjzn//s9fXo0cOLhw8fHrd79uzp9V144YVefO+99xadA/Yuxx13nBfv2rXLi1etWlXJdDLjDAUAEAQFBQAQBAUFABAE11DQIpxwgv8JxvPOOy/vtu+//74Xn3POOV68bt26uN3Q0OD17bvvvl78+uuvx+2+fft6fZ06dSqQMfC1fv36efHmzZu9eObMmZVMJzPOUAAAQVBQAABB1MSUV/IjnZdddpnX9/e//92Lv/rqq7j91FNPeX1r1qzx4uXLl4dKETXusMMO82Izf/GA5DTX4MGDvb7Vq1cXfZzrr7/ei3v16pV32xdeeKHo/WLv07t377g9fvx4r6+5rlTNGQoAIAgKCgAgCAoKACCImriGMnny5LjdvXv3oh93xRVXePGmTZu8OP3x0EpILpGQfF6StGjRokqns9d4/vnnvfioo47y4uTYWL9+febjjB071ovbtGmTeV/Yu333u9+N2+3bt/f60ksHNRecoQAAgqCgAACCoKAAAIKoiWsoye+e9OnTx+tbunSpF3/ve9+L2/379/f6zjjjDC8+6aST4vann37q9R155JFF57djxw4vXrt2bdxOf/8hKX2HPq6hVE59fX2Q/UyYMMGL03fWS3rjjTcKxkDSjTfeGLfT47W5vlZwhgIACIKCAgAIoiamvP7yl7/ssb0nL730Ut6+Aw44wIuTK3guXrzY6zvxxBOLzi+53IskffTRR3E7PSV34IEHxu0VK1YUfQzUjmHDhsXtu+66y+tLrzb8+eefx+2bb77Z69uyZUsZskNzlf5KRHKF7ORrivTN1YabC85QAABBUFAAAEFQUAAAQdTENZRQvvzySy+eN29e3m0bu1ZTyMiRI+N2+rrNu+++G7eb6/IJe7vk3Hb6mkla8m+8YMGCsuWE5u/000/P25f8KkJzxhkKACAICgoAIAgKCgAgiBZ1DaVcDj74YC+eMmVK3G7Vyq/Jye8tNGWZdFTOc88958Vnnnlm3m2feOIJL7711lvLkhNanmOPPTZvX/pWF80VZygAgCAoKACAIJjyKsLVV1/txQcddFDcTn9U+cMPP6xITsguvUL0gAEDvLiuri5ur1u3zuu75557vLihoSFwdmgpkqudS9Ill1zixW+99VbcfvnllyuSU7lxhgIACIKCAgAIgoICAAiCayh7cMopp3jxL37xi7zbjhgxwovfe++9suSEcGbMmOHFnTp1yrvttGnTvJhbEqBYgwYN8uLkrS0k/1Yc6VtkNFecoQAAgqCgAACCoKAAAILgGsoeDB061IvbtGnjxcml71977bWK5ISmOeecc+J2//79C247f/78uH377beXKyW0cH379vVi55wXT58+vZLpVARnKACAICgoAIAgmPLK2W+//eL2WWed5fVt27bNi5PTINu3by9vYsgk/VHgW265JW6npzDT3n777bjN0iooxaGHHhq3Tz31VK8vvSzTzJkzK5JTJXGGAgAIgoICAAiCggIACIJrKDkTJkyI28cdd5zXl1wiQZJeffXViuSE7K6//novPvHEE/Num75jIx8VRlY//elP43b6Tq8vvvhihbOpPM5QAABBUFAAAEFQUAAAQey111DOPvtsL77tttvi9j/+8Q+v76677qpITgjnuuuuK3rb8ePHezHfPUFW3bp1y9uXvl14S8QZCgAgCAoKACCIvWbKK70Ux69//Wsvbt26ddyeM2eO1/f666+XLzFUXfpOelmX09m4cWPB/SSXfOnYsWPe/ey///5eXMr03c6dO734pptuittbtmwpej/IZtiwYXn7nn/++QpmUh2coQAAgqCgAACCoKAAAIJo0ddQktdF0sunfPvb3/biFStWxO3kR4jR8r3zzjtB9vPHP/7Ri1evXu3FhxxySNweM2ZMkGM2Zs2aNXF70qRJFTnm3mTgwIFenFy+fm/EGQoAIAgKCgAgiBY95dWzZ8+4ffzxxxfcNvnRzOT0F5qn9Ee/zz333LIfc/To0Zkfu2PHjri9a9eugtvOmjUrbi9atKjgtn/9618z54TGnXfeeV6cnGZ/6623vL6FCxdWJKdq4gwFABAEBQUAEAQFBQAQRIu6hpJe6XPu3Ll5t03eoVGSZs+eXZacUB3nn3++F994441xO7kESmOOOeYYLy7l476PPvqoF69cuTLvtjNmzIjby5YtK/oYqKx27dp58dChQ/NuO336dC9OL4vTEnGGAgAIgoICAAiCggIACKJFXUO5/PLLvbhr1655t12wYIEXO+fKkhNqw+TJk4PsZ9y4cUH2g+YpfUuC9F0Yk98ReuCBByqSUy3hDAUAEAQFBQAQRLOe8kqv9HnNNddUKRMAe4P0lNeAAQOqlElt4gwFABAEBQUAEAQFBQAQRLO+hnLqqad6cYcOHfJum16SvqGhoSw5AcDeijMUAEAQFBQAQBAUFABAEM36GkpjlixZErd//OMfe33r16+vdDoA0KJxhgIACIKCAgAIwkpZZdfMWJK3BjnnrNo5FMK4qVmLnXMnVDuJQhg7NWuPY4czFABAEBQUAEAQFBQAQBClfmx4naT6ciSCzLpVO4EiMG5qE2MHWe1x7JR0UR4AgHyY8gIABEFBAQAEQUEBAARBQQEABEFBAQAEQUEBAARBQQEABEFBAQAEQUEBAARBQQEABEFBAQAEQUEBAARBQQEABNGiC4qZrTSzQVU8/iozO6Nax0d2jB1ktTePnSYVFDMba2ZvmNlmM/s8177KzGr9HucvmllD7t92M9uWiB/OuM9pZnZH4Dz/zczqc3k9a2b7h9x/NTF2vH0GHzuJfT9pZs7Mupdj/9XA2PH2GXTsmNkRZva8ma3OjZsupTw+c0Exs+slPSDpPkmHSjpE0pWSTpG0b57HtM56vJCcc0Occx2ccx0kPSVp8u7YOXdlenszK/VGZE1mZn0kTZF0oaLf73ZJv6l0HuXA2KmM3LvU5nATraIxdspul6Q5kkZlerRzruR/kjpK2ixpZCPbPSZpai7BzZIG5R77hKS1iu7EdqukVrnt75A0LfH47pKcpH1y8XxJd0v6m6RNkuZK6pzY/qLcPr+QNFHSSkmDisjxntTPBuUee4ukNZJ+J+nfJc1PbLNPLrfukq5S9IK/TVKDpJm5bVZJuk7Su5I2SnpaUl2Rv+PJkp5IxEdL+qekdln+ZrXyj7FT/rGTe3wbSUsk9d19rGr/7Rk7zWPs5PbRNnecLqU8LusZysmS6iT9qYhtx0maJOlbkl6R9KCiP24PSadL+omkS0o49rjc9gcrekdygySZWS9Fg+giSYdL6iSppNO1lC6SOkjqqugPl5dzboqkZyT90kXvNs5LdF8g6V8VPd/jc/nJzFqb2QYzOynPbo9R9IKw+xgfKnr38H+yPZ2awdhJKNPYkaLn9j+S3s/8LGoPYyehjGMns6wFpbOkdc65Hbt/YGav5hLdamanJbb9k3Pub865XYqq6RhJNzvnNjnnVkr6D+WebJF+55z7yDm3VdIfJPXL/XyUpNnOuYXOuX9Kuk3RC3BWOyTd4ZzbljtWVv/pnFvjnPtC0uzd+Trndjrn9nfOvZ7ncR0UvbtI+oei/yDNGWOneJnGjpl1k3SponfeLQljp3hZX3eaJGtB+UJS5+Qcn3NugHNu/1xfcr+fJtqdFVX3+sTP6iUdUcKx1yTaWxS98ErRu4P4WM65zblcsvrMObetCY/fLV++jWmQ9C+pn/2LolPu5oyxU7ysY+fXkm53zjX3sZLG2Cle1rHTJFkLymuK5vPPLWJbl2ivU/RuIXmhsKuk/821N0tql+g7tIScVks6cndgZu0UnX5m5VJxY7mlt2+q9xXNf0uSzOw7iv5e/y/wcSqNsVP+sfNjSfeb2RpF8+mS9KaZjQl8nEpj7JR/7DRJpoLinNsg6U5JU8xslJl1MLNWZtZPUvsCj9up6HRxkpl9K3dqfp2kablN3pZ0mpl1NbOOkm4uIa3pkoaZ2UAz21fSXQr7PZslkvqY2bFmtp+k21P9nymarwxlmqQRZjbAzNorej5/dM5tCXiMimPsVGTs9FA0xdFP0fy5JA2VNCvgMSqOsVORsSMza6voWpUk1ZlZXaHtkzI/cefcZEV/lBslfa7oiT0i6SZJrxZ46DWKqu7Hii6W/bekR3P7fFnRRaZ3JC1WNPdXbD7vS7o6t7/Vkr7U1+/Omsw594GkXyr6xMeHkhamNvkvSX3N7Eszm97Y/nIXxxrM7OQ8x3tH0nhJv1f0+61T9Ltr9hg7ZR87n+fmz9co+t1K0tomzsnXBMZOecdObjpxq6QNuR8tV/R7K4rlPiIGAECTtOilVwAAlUNBAQAEQUEBAARBQQEABEFBAQAEUdJqlmbGR8JqkHOu1pftZtzUpnXOuYOqnUQhjJ2atcexwxkKsPeqb3wTYI/2OHYoKACAICgoAIAgKCgAgCAoKACAICgoAIAgKCgAgCAoKACAICgoAIAgKCgAgCAoKACAICgoAIAgKCgAgCBKWm24uWnfvn3cvu+++7y+K664wosXL14ct0ePHu311dezhh4ANIYzFABAEBQUAEAQLXrK67DDDovbl112mde3a9cuLz7++OPj9rBhw7y+hx56qAzZoVr69+/vxc8++6wXd+/evew5nHnmmV68dOnSuP3pp5+W/fioLcOHD/fiWbNmefH48ePj9sMPP+z17dy5s3yJlYgzFABAEBQUAEAQFBQAQBAt6hrKQQcd5MWPP/54lTJBLRs8eLAX19XVVTyH9Jz5pZdeGrfHjh1b6XRQBZ06dYrbU6ZMKbjtb37zm7j96KOPen1bt24Nm1gTcIYCAAiCggIACKJZT3n9/Oc/9+IRI0Z48fe///1M+z3ttNO8uFUrv+4uWbIkbi9cuDDTMVBZ++zz9VAfOnRoFTOJJFdmkKTrrrsubidXeJCkzZs3VyQnVFbydaZLly4Ft3366afj9ldffVW2nJqKMxQAQBAUFABAEBQUAEAQzfoayq9+9SsvTi+nktX5559fME6uPjxmzBivLz03jtrwwx/+MG6ffPLJXt/kyZMrnY4OOOAAL+7Vq1fcbteundfHNZSWIf3x9IkTJxb92CeffDJuO+eC5RQaZygAgCAoKACAICgoAIAgrJT5ODOr+uTdnDlz4vaQIUO8vqZcQ/niiy/idkNDg9fXrVu3ovfTunXrzDlk5Zyzih+0BNUYN7179/bi+fPnx+3k31ryb10gffPvXw7JfCRp4MCBcTt52wVJWrt2bbnSWOycO6FcOw+hFl5zQjnhBP9X/eabb+bddseOHV7cpk2bsuTUBHscO5yhAACCoKAAAIKo+Y8Nn3766V589NFHx+30FFcpU17pu57NnTs3bm/cuNHr+9GPfuTFhT7u97Of/SxuT506teh8ENatt97qxcnlTM466yyvrxJTXJJ04IEHxu30uA71kXfUrpEjRxa9bfL1qDnhDAUAEAQFBQAQBAUFABBEzV1D6d69uxf//ve/9+LOnTsXva/kEikzZszw+u68804v3rJlS1H7kaTLL788bqfvEplcxqNt27ZeX/Kua5K0ffv2vMdEaUaNGuXF6SXqly9fHrcXLVpUkZzSktfe0tdMkh8j3rBhQ6VSQgWlb4uRtG3bNi8uZVmWWsIZCgAgCAoKACAICgoAIIiau4aSvFWrVNo1kwULFnjx2LFj4/a6desy55S+hnLvvffG7fvvv9/rSy49nl4WfdasWV68YsWKzDnBN3r0aC9OLwE/ZcqUSqYj6ZvXAy+88MK4vXPnTq/vnnvuidtcW2sZBgwYUDBOSt+i4O233y5LTuXGGQoAIAgKCgAgiJqb8ipF+uOfl156qRc3ZZqrkOTUVXIaQ5JOPPHEshwT39SxY8e4fdJJJxXcthrL4CQ/Xi7507dLly71+ubNm1eRnFA5pbwWtJRlmjhDAQAEQUEBAARBQQEABFHz11Batcpf837wgx9UMJOvmX19g8R0foXyveOOO7z4oosuCprX3qauri5uH3HEEV7f008/Xel0vqFnz555+957770KZoJqSN+hMS25xA7XUAAASKCgAACCoKAAAIKouWsoV155pRfX4q1Rhw8fHrePO+44ry+Zbzr39DUUNM2mTZvidnqpij59+nhx8va769evL0s+Bx98sBenl9RPeuWVV8qSA6pr4MCBcXvcuHEFt03eanzVqlVly6mSOEMBAARBQQEABFFzU17J6aRqSd+FsVevXl58yy23FLWftWvXejGryIa1devWuJ1euXnkyJFe/MILL8Tt9ArRpejdu7cX9+jRI26nVxd2zuXdTy1O5aLpOnXqFLcLfYVAkl5++eVyp1NxnKEAAIKgoAAAgqCgAACCqLlrKLVg4sSJXnz11VcX/diVK1fG7Ysvvtjr++STT5qUF/K7/fbbvTi5PI4knX322XG7KcuypG+JkLxOUsrdRR977LHMOaB2FfqoeHKpFUl65JFHyp1OxXGGAgAIgoICAAiCggIACIJrKDlz5syJ20cffXTm/XzwwQdxm+U1KmfZsmVefMEFF3hxv3794vZRRx2V+TjTp0/P2/f44497cfr20EnJ79Cg+erSpYsXF1puJb28SvoW5i0BZygAgCAoKACAIGpuyiv9cc9CyxcMGTKk4L5++9vfxu3DDz+84LbJ4zRlWYxaWDoG35RcjTi9MnEoH3/8cdHbppdw4Q6OzdOAAQO8uNDr1XPPPVfudKqOMxQAQBAUFABAEBQUAEAQNXcNZerUqV48efLkvNvOnj3biwtd+yjlukgp2z788MNFb4uWLX39Lx0ncc2kZUguV5+WXqbngQceKHc6VccZCgAgCAoKACCImpvyevbZZ714woQJXpy+m2I5pO+0uHTpUi++/PLL4/bq1avLng+ah/QdGgvdsREtw+DBg/P2pVcX37hxY7nTqTrOUAAAQVBQAABBUFAAAEHU3DWU+vp6Lx47dqwXjxgxIm5fe+21Zclh0qRJXvzQQw+V5ThoWdq2bVuwnxWGm782bdp4cc+ePfNu+9VXX3nx9u3by5JTLeEMBQAQBAUFABAEBQUAEETNXUNJW7hwYd547ty5Xl/y+yGSv5T8rFmzvL7k0vaSv0xG8q6LQLEuueQSL96wYYMX33333ZVMB2WQXpYpfdfF5G0Jli9fXpGcaglnKACAICgoAIAgan7Kq5CXXnqpYAxU0ptvvunF999/vxfPmzevkumgDHbu3OnFEydO9OLkcjuLFy+uSE61hDMUAEAQFBQAQBAUFABAEFbKEttmxnrcNcg5l//WgDWAcVOzFjvnTqh2EoUwdmrWHscOZygAgCAoKACAICgoAIAgKCgAgCAoKACAICgoAIAgKCgAgCAoKACAICgoAIAgKCgAgCBKXb5+naT6ciSCzLpVO4EiMG5qE2MHWe1x7JS0lhcAAPkw5QUACIKCAgAIgoICAAiCggIACIKCAgAIgoICAAiCggIACIKCAgAIgoICAAji/wNyT3w8hw7vmQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "examples = enumerate(test_loader)\n",
    "batch_idx, (example_data, example_targets) = next(examples)\n",
    "\n",
    "fig = plt.figure()\n",
    "for i in range(6):\n",
    "  plt.subplot(2,3,i+1)\n",
    "  plt.tight_layout()\n",
    "  plt.imshow(example_data[i][0], cmap='gray', interpolation='none')\n",
    "  plt.title(\"Ground Truth: {}\".format(example_targets[i]))\n",
    "  plt.xticks([])\n",
    "  plt.yticks([])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5.4 构建模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Net(nn.Module):\n",
    "    \"\"\"\n",
    "    使用sequential构建网络，Sequential()函数的功能是将网络的层组合到一起\n",
    "    \"\"\"\n",
    "    def __init__(self, in_dim, n_hidden_1, n_hidden_2, out_dim):\n",
    "        super(Net, self).__init__()\n",
    "        self.flatten = nn.Flatten()\n",
    "        self.layer1 = nn.Sequential(nn.Linear(in_dim, n_hidden_1),nn.BatchNorm1d(n_hidden_1))\n",
    "        self.layer2 = nn.Sequential(nn.Linear(n_hidden_1, n_hidden_2),nn.BatchNorm1d(n_hidden_2))\n",
    "        self.out = nn.Sequential(nn.Linear(n_hidden_2, out_dim))\n",
    "        \n",
    " \n",
    "    def forward(self, x):\n",
    "        x=self.flatten(x)\n",
    "        x = F.relu(self.layer1(x))\n",
    "        x = F.relu(self.layer2(x))\n",
    "        x = F.softmax(self.out(x),dim=1)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "F.softmax函数中参数dim指维度，dim=0表示按列计算；dim=1表示按行计算。默认dim的方法已经弃用了，最好声明dim，否则会警告."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 0.01\n",
    "momentum = 0.9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#实例化模型\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "model = Net(28 * 28, 300, 100, 10)\n",
    "model.to(device)\n",
    "\n",
    "# 定义损失函数和优化器\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=lr, momentum=momentum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5.5 训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "学习率:0.009000\n",
      "epoch: 0, Train Loss: 1.6156, Train Acc: 0.8931, Test Loss: 1.5135, Test Acc: 0.9598\n",
      "epoch: 1, Train Loss: 1.5100, Train Acc: 0.9624, Test Loss: 1.4960, Test Acc: 0.9728\n",
      "epoch: 2, Train Loss: 1.4960, Train Acc: 0.9730, Test Loss: 1.4914, Test Acc: 0.9760\n",
      "epoch: 3, Train Loss: 1.4888, Train Acc: 0.9787, Test Loss: 1.4867, Test Acc: 0.9786\n",
      "epoch: 4, Train Loss: 1.4841, Train Acc: 0.9822, Test Loss: 1.4864, Test Acc: 0.9787\n",
      "学习率:0.008100\n",
      "epoch: 5, Train Loss: 1.4801, Train Acc: 0.9855, Test Loss: 1.4853, Test Acc: 0.9787\n",
      "epoch: 6, Train Loss: 1.4779, Train Acc: 0.9875, Test Loss: 1.4835, Test Acc: 0.9814\n",
      "epoch: 7, Train Loss: 1.4755, Train Acc: 0.9892, Test Loss: 1.4847, Test Acc: 0.9789\n",
      "epoch: 8, Train Loss: 1.4747, Train Acc: 0.9900, Test Loss: 1.4819, Test Acc: 0.9824\n",
      "epoch: 9, Train Loss: 1.4728, Train Acc: 0.9913, Test Loss: 1.4813, Test Acc: 0.9834\n",
      "学习率:0.007290\n",
      "epoch: 10, Train Loss: 1.4719, Train Acc: 0.9921, Test Loss: 1.4808, Test Acc: 0.9825\n",
      "epoch: 11, Train Loss: 1.4710, Train Acc: 0.9929, Test Loss: 1.4817, Test Acc: 0.9823\n",
      "epoch: 12, Train Loss: 1.4700, Train Acc: 0.9938, Test Loss: 1.4813, Test Acc: 0.9819\n",
      "epoch: 13, Train Loss: 1.4697, Train Acc: 0.9937, Test Loss: 1.4803, Test Acc: 0.9832\n",
      "epoch: 14, Train Loss: 1.4689, Train Acc: 0.9945, Test Loss: 1.4807, Test Acc: 0.9824\n",
      "学习率:0.006561\n",
      "epoch: 15, Train Loss: 1.4681, Train Acc: 0.9950, Test Loss: 1.4801, Test Acc: 0.9830\n",
      "epoch: 16, Train Loss: 1.4681, Train Acc: 0.9950, Test Loss: 1.4801, Test Acc: 0.9833\n",
      "epoch: 17, Train Loss: 1.4673, Train Acc: 0.9956, Test Loss: 1.4804, Test Acc: 0.9826\n",
      "epoch: 18, Train Loss: 1.4668, Train Acc: 0.9960, Test Loss: 1.4798, Test Acc: 0.9835\n",
      "epoch: 19, Train Loss: 1.4666, Train Acc: 0.9962, Test Loss: 1.4795, Test Acc: 0.9835\n"
     ]
    }
   ],
   "source": [
    "# 开始训练\n",
    "losses = []\n",
    "acces = []\n",
    "eval_losses = []\n",
    "eval_acces = []\n",
    "writer = SummaryWriter(log_dir='logs',comment='train-loss')\n",
    "\n",
    "for epoch in range(num_epoches):\n",
    "    train_loss = 0\n",
    "    train_acc = 0\n",
    "    model.train()\n",
    "    #动态修改参数学习率\n",
    "    if epoch%5==0:\n",
    "        optimizer.param_groups[0]['lr']*=0.9\n",
    "        print(\"学习率:{:.6f}\".format(optimizer.param_groups[0]['lr']))\n",
    "    for img, label in train_loader:\n",
    "        img=img.to(device)\n",
    "        label = label.to(device)\n",
    "        # 正向传播\n",
    "        out = model(img)\n",
    "        loss = criterion(out, label)\n",
    "        # 反向传播\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        # 记录误差\n",
    "        train_loss += loss.item()\n",
    "        # 保存loss的数据与epoch数值\n",
    "        writer.add_scalar('Train', train_loss/len(train_loader), epoch)\n",
    "        # 计算分类的准确率\n",
    "        _, pred = out.max(1)\n",
    "        num_correct = (pred == label).sum().item()\n",
    "        acc = num_correct / img.shape[0]\n",
    "        train_acc += acc\n",
    "        \n",
    "    losses.append(train_loss / len(train_loader))\n",
    "    acces.append(train_acc / len(train_loader))\n",
    "    # 在测试集上检验效果\n",
    "    eval_loss = 0\n",
    "    eval_acc = 0\n",
    "    #net.eval() # 将模型改为预测模式\n",
    "    model.eval()\n",
    "    for img, label in test_loader:\n",
    "        img=img.to(device)\n",
    "        label = label.to(device)\n",
    "        img = img.view(img.size(0), -1)\n",
    "        out = model(img)\n",
    "        loss = criterion(out, label)\n",
    "        # 记录误差\n",
    "        eval_loss += loss.item()\n",
    "        # 记录准确率\n",
    "        _, pred = out.max(1)\n",
    "        num_correct = (pred == label).sum().item()\n",
    "        acc = num_correct / img.shape[0]\n",
    "        eval_acc += acc\n",
    "        \n",
    "    eval_losses.append(eval_loss / len(test_loader))\n",
    "    eval_acces.append(eval_acc / len(test_loader))\n",
    "    print('epoch: {}, Train Loss: {:.4f}, Train Acc: {:.4f}, Test Loss: {:.4f}, Test Acc: {:.4f}'\n",
    "          .format(epoch, train_loss / len(train_loader), train_acc / len(train_loader), \n",
    "                     eval_loss / len(test_loader), eval_acc / len(test_loader)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x25db5b2cd48>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3xc5X3n8c93dBtZkiWwhCXZgM012CYmRlBIUnCAbUjIpdmQBii5lV1C23Rhm2xDu1lIeLG7SdhtUkK3rAsOJWFNtkBJQiCUTQlOXgWCbYwxOIBjboplW7axJNuSrJF++8ccCSEkS5ZGF89836/XvObMOc8556fj8e8885znnEcRgZmZ5a/UdAdgZmaTy4nezCzPOdGbmeU5J3ozszznRG9mluec6M3M8pwTvRkg6VZJ/2Wc6/5c0r/LdUxmuVI83QGYTZSkV4B/FxH/b7zbiIircheR2cziGr3lPUmu0FhBc6K3w5qk7wHHAD+WtFfSX0haICkkXSHpNeBfkrL/KGmbpDZJqyUtHrSdOyTdmEwvl9Qs6YuSdkhqkfS5McaTkvQVSa8m694pqTpZlpb0fUm7JO2R9JSkucmyz0raIqlD0suS/jDHh8oKmBO9HdYi4lPAa8CHI6IyIr45aPG5wCnA+5PPDwEnAkcB64C7DrLpeqAamAdcAfytpCPGENJnk9f7gOOASuCWZNlnkm0eDcwBrgI6JVUANwMfiIgq4N3A+jHsy2xMnOgtn301IvZFRCdARKyMiI6I6Aa+Ciztr20Powe4ISJ6IuJBYC9w8hj2+YfAX0fElojYC/wlcEnSfNRDNsGfEBG9EbE2ItqT9fqAJZLKI6IlIp4b7x9tNpQTveWz1/snJBVJ+rqk30hqB15JFtWOsO6uiMgM+ryfbO18NI3Aq4M+v0q208Nc4HvAw8DdkrZK+qakkojYB3ySbA2/RdJPJL1jDPsyGxMnessHIz2CdfD8y4CPAheQbT5ZkMxXjmPZChw76PMxQAbYnvw6+FpELCLbPPMh4NMAEfFwRPwboAH4NfD3OY7LCpgTveWD7WTbww+mCugGdgGzgP82SbGsAv6jpIWSKpP9/CAiMpLeJ+lUSUVAO9mmnF5JcyV9JGmr7ybbTNQ7SfFZAXKit3zw34GvJD1ZvjRCmTvJNqP8FngeeGKSYllJtolmNfAy0AX8WbKsHriHbJLfBDwGfJ/s/8Mvkv01sJvsReQ/maT4rADJA4+YmeU31+jNzPKcE72ZWZ5zojczy3NO9GZmeW5GPuyptrY2FixYMN1hmJkdNtauXbszIuqGWzYjE/2CBQtYs2bNdIdhZnbYkPTqSMvcdGNmluec6M3M8pwTvZlZnpuRbfRmlh96enpobm6mq6trukPJG+l0mvnz51NSUjLmdZzozWzSNDc3U1VVxYIFC5By/aDQwhMR7Nq1i+bmZhYuXDjm9UZtupG0MhkSbeNByiyXtF7Sc5IeS+YdLelRSZuS+VePOSozywtdXV3MmTPHST5HJDFnzpxD/oU0ljb6O4ALD7LjGuB/AR+JiMXAJ5JFGeCLEXEKcBbwp5IWHVJ0ZnbYc5LPrfEcz1ETfUSsJvvo1JFcBtwXEa8l5Xck7y0RsS6Z7iD7WNZ5hxzhGPX1Bbf8y0usfrF1snZhZnZYykWvm5OAIyT9XNJaSZ8eWkDSAuBdwJMjbUTSlZLWSFrT2nroyTqVEitWb+Fnm7Yf8rpmlp927drFaaedxmmnnUZ9fT3z5s0b+HzgwIExbeNzn/scL7zwwpj3edttt3HNNdeMN+RJkYuLscXA6cD5QDnwuKQnIuJFgGSUnXuBawYNhPw2EbECWAHQ1NQ0rofkN1SXs7XNV/fNLGvOnDmsX78egK9+9atUVlbypS+9dWyaiCAiSKWGr/d+97vfnfQ4J1suavTNwE8jYl9E7CQ7ss5SAEklZJP8XRFxXw72dVANNWla2jonezdmdpjbvHkzS5Ys4aqrrmLZsmW0tLRw5ZVX0tTUxOLFi7nhhhsGyr73ve9l/fr1ZDIZampquPbaa1m6dClnn302O3bsGPM+v//973PqqaeyZMkS/uqv/gqATCbDpz71qYH5N998MwDf+ta3WLRoEUuXLuXyyy+f8N+bixr9D4FbJBUDpcDvAN9S9orB7cCmiPjrHOxnVA3V5Wz8bdtU7MrMDtHXfvwcz28d8Uf9uCxqnM31H148rnWff/55vvvd73LrrbcC8PWvf50jjzySTCbD+973Pi6++GIWLXpr/5G2tjbOPfdcvv71r/Pnf/7nrFy5kmuvvXbUfTU3N/OVr3yFNWvWUF1dzQUXXMADDzxAXV0dO3fu5NlnnwVgz549AHzzm9/k1VdfpbS0dGDeRIyle+Uq4HHgZEnNkq6QdJWkqwAiYhPwU2AD8CvgtojYCLwH+BRwXtL1cr2kD0444oNorE6zc+8BujMeV9nMDu7444/njDPOGPi8atUqli1bxrJly9i0aRPPP//829YpLy/nAx/4AACnn346r7zyypj29eSTT3LeeedRW1tLSUkJl112GatXr+aEE07ghRde4Oqrr+bhhx+muroagMWLF3P55Zdz1113HdKNUSMZtUYfEZeOocxNwE1D5v0SmNJ+VfXVaQC2tXVx7JyKqdy1mY1ivDXvyVJR8WaOeOmll/ibv/kbfvWrX1FTU8Pll18+bF/10tLSgemioiIymcyY9jXS2Nxz5sxhw4YNPPTQQ9x8883ce++9rFixgocffpjHHnuMH/7wh9x4441s3LiRoqKiQ/wL35RXz7pprCkHYOseX5A1s7Frb2+nqqqK2bNn09LSwsMPP5zT7Z911lk8+uij7Nq1i0wmw9133825555La2srEcEnPvEJvva1r7Fu3Tp6e3tpbm7mvPPO46abbqK1tZX9+/dPaP959QiEhv4afbsvyJrZ2C1btoxFixaxZMkSjjvuON7znvdMaHu3334799xzz8DnNWvWcMMNN7B8+XIigg9/+MNcdNFFrFu3jiuuuIKIQBLf+MY3yGQyXHbZZXR0dNDX18eXv/xlqqqqJhSPRvpJMZ2amppiPAOPdB7o5ZTrfsp/ev/J/On7TpiEyMzsUGzatIlTTjllusPIO8MdV0lrI6JpuPJ51XRTXlpEzawSd7E0MxskrxI9ZLtYtriN3sxsQN4l+sbqtO+ONZtBZmLz8OFsPMcz7xJ9Q02abW66MZsR0uk0u3btcrLPkf7n0afT6UNaL6963UC26eaN/T10HuilvHT8/U7NbOLmz59Pc3Mz43lQoQ2vf4SpQ5GHiT57pmtp6+S4usppjsassJWUlBzSSEg2OfKv6aY6e9NUi9vpzcyAPEz0jTX9NXonejMzyMNEP3d2kuj3+IKsmRnkYaJPlxQxp6LUXSzNzBJ5l+jBA5CYmQ2Wn4m+upxtrtGbmQF5m+jTbHUbvZkZkLeJvpz2rgz7usc2KICZWT4by1CCKyXtkLTxIGWWJ0MFPifpsUHzL5T0gqTNkkYfWDFH3uxi6Vq9mdlYavR3ABeOtFBSDfC/gI9ExGLgE8n8IuBvgQ8Ai4BLJS0aaTu55JumzMzeNGqij4jVwO6DFLkMuC8iXkvK70jmnwlsjogtEXEAuBv46ATjHZOBxyD4ccVmZjlpoz8JOELSzyWtlfTpZP484PVB5ZqTecOSdKWkNZLWTPQBSHNnp5Fgq5tuzMxy8lCzYuB04HygHHhc0hOAhik74rNKI2IFsAKyQwlOJKDS4hS1lWWu0ZuZkZtE3wzsjIh9wD5Jq4GlyfyjB5WbD2zNwf7GJDsAiWv0Zma5aLr5IfC7koolzQJ+B9gEPAWcKGmhpFLgEuBHOdjfmPimKTOzrFFr9JJWAcuBWknNwPVACUBE3BoRmyT9FNgA9AG3RcTGZN0vAA8DRcDKiHhuUv6KYdRXp/nl5p1TtTszsxlr1EQfEZeOocxNwE3DzH8QeHB8oU1MY02avd0Z2rt6mJ0umY4QzMxmhLy8MxYG9aX3BVkzK3B5m+h9d6yZWVbeJvp63x1rZgbkcaKfW1VGSh5pyswsbxN9cVGKo6rSHmnKzApe3iZ6yI405b70Zlbo8jvR++5YM7N8T/TltOzpImJCj84xMzus5XmiT9PZ00tbZ890h2JmNm3yOtE31riLpZlZXif6+mrfNGVmlteJvjG5aWqrH4NgZgUsrxN9XVUZxSm5Rm9mBS2vE31RSsydnfaDzcysoOV1oodszxtfjDWzQpb3ib6+Ou2mGzMraHmf6Btrymlp801TZla4Rk30klZK2iFp4wjLl0tqk7Q+eV03aNl/lPScpI2SVklK5zL4sWioTtOd6WP3vgNTvWszsxlhLDX6O4ALRynzi4g4LXndACBpHvAfgKaIWEJ23NhLJhLseDT4ufRmVuBGTfQRsRrYPc7tFwPlkoqBWcDWcW5n3BoGbppyojezwpSrNvqzJT0j6SFJiwEi4rfA/wBeA1qAtoj455E2IOlKSWskrWltbc1RWNlHFYPvjjWzwpWLRL8OODYilgLfAe4HkHQE8FFgIdAIVEi6fKSNRMSKiGiKiKa6urochJVVW1FGSZF8d6yZFawJJ/qIaI+Ivcn0g0CJpFrgAuDliGiNiB7gPuDdE93foUqlRH11mm2u0ZtZgZpwopdUL0nJ9JnJNneRbbI5S9KsZPn5wKaJ7m88GmaXe0hBMytYxaMVkLQKWA7USmoGrgdKACLiVuBi4I8lZYBO4JLIdlp/UtI9ZJt2MsDTwIrJ+CNG01CTZt1rb0zHrs3Mpt2oiT4iLh1l+S3ALSMsu57siWFaNVSXs62thb6+IJXSdIdjZjal8v7OWIDGmjQ9vcEu3zRlZgWoIBJ9/Wx3sTSzwlUQib5/SEF3sTSzQlQQib7BQwqaWQEriER/ZEUppcUpPwbBzApSQSR6SR6AxMwKVkEkekhGmtrjphszKzwFk+gbq8tdozezglQwib6hJs229i56+zzSlJkVloJJ9PXV5fT2BTv3dk93KGZmU6pgEn1j0sVyq9vpzazAFEyi95CCZlaoCibRN9a4Rm9mhalgEn11eQnlJUVsc43ezApMwSR63zRlZoWqYBI9ZLtYbvXzbsyswIya6CWtlLRD0sYRli+X1CZpffK6btCyGkn3SPq1pE2Szs5l8IeqobqcFj/B0swKzKgjTAF3kB1B6s6DlPlFRHxomPl/A/w0Ii6WVArMOvQQc6exOs2Oji4yvX0UFxXUjxkzK2CjZruIWA3sPtQNS5oNnAPcnmznQETsOeQIc6i+upy+gB0dvmnKzApHrqq1Z0t6RtJDkhYn844DWoHvSnpa0m2SKnK0v3FpqPFz6c2s8OQi0a8Djo2IpcB3gPuT+cXAMuDvIuJdwD7g2pE2IulKSWskrWltbc1BWG/XWO2Rpsys8Ew40UdEe0TsTaYfBEok1QLNQHNEPJkUvYds4h9pOysioikimurq6iYa1rBcozezQjThRC+pXpKS6TOTbe6KiG3A65JOToqeDzw/0f1NRFVZMRWlRe5Lb2YFZdReN5JWAcuBWknNwPVACUBE3ApcDPyxpAzQCVwSEf3PAv4z4K6kx80W4HM5/wsOgSQaatzF0swKy6iJPiIuHWX5LWS7Xw63bD3QNL7QJkf27lg33ZhZ4Si4zuSN1eVsddONmRWQgkv09dVpdu7t5kCmb7pDMTObEgWX6Btr0kTA9nbX6s2sMBRcovcAJGZWaAou0Te6L72ZFZiCS/T1rtGbWYEpuERfWVZMVbqYFg8paGYFouASPbiLpZkVloJM9A01vmnKzApHYSb66nIPEm5mBaNAE32anXsP0J3pne5QzMwmXcEmesC1ejMrCAWZ6BtrPACJmRWOgkz0AzX6dl+QNbP8V6CJ3jV6MyscBZnoy0uLqJlV4i6WZlYQCjLRQ7ZW75GmzKwQjJroJa2UtEPSxhGWL5fUJml98rpuyPIiSU9LeiBXQedCY3Xad8eaWUEYS43+DuDCUcr8IiJOS143DFl2NbBpPMFNpvrqNNvcdGNmBWDURB8Rq4Hd49m4pPnARcBt41l/MjXWlPPG/h46D/imKTPLb7lqoz9b0jOSHpK0eND8bwN/AYw6bp+kKyWtkbSmtbU1R2GNrL+LpS/Imlm+y0WiXwccGxFLge8A9wNI+hCwIyLWjmUjEbEiIpoioqmuri4HYR2cR5oys0Ix4UQfEe0RsTeZfhAokVQLvAf4iKRXgLuB8yR9f6L7y5U3a/RO9GaW3yac6CXVS1IyfWayzV0R8ZcRMT8iFgCXAP8SEZdPdH+5Ut+f6D0AiZnlueLRCkhaBSwHaiU1A9cDJQARcStwMfDHkjJAJ3BJRMSkRZwj6ZIi5lSUuoulmeW9URN9RFw6yvJbgFtGKfNz4OeHEthU8AAkZlYICvbOWID62R6AxMzyX0En+saaNFvdRm9mea6gE31DdTntXRn2dWemOxQzs0lT0Im+scY3TZlZ/ivoRF8/233pzSz/FXSi7x9S0I8rNrN8VtCJfu7sNBJsddONmeWxgk70pcUpaivLXKM3s7xW0Ike+gcgcY3ezPJXwSf67AAkrtGbWf4q+ETfUF3uXjdmltcKPtE31qTZ252hvatnukMxM5sUBZ/oBwYg8QVZM8tTTvQeUtDM8pwTfY2HFDSz/FbwiX5uVRkpeaQpM8tfBZ/oi4tSHFWV9khTZpa3Rk30klZK2iFp4wjLl0tqk7Q+eV2XzD9a0qOSNkl6TtLVuQ4+V9yX3szy2ahDCQJ3kB0q8M6DlPlFRHxoyLwM8MWIWCepClgr6ZGIeH58oU6expo0v97WMd1hmJlNilFr9BGxGth9qBuOiJaIWJdMdwCbgHmHHOEUaKgup2VPF4fBmOZmZocsV230Z0t6RtJDkhYPXShpAfAu4MmRNiDpSklrJK1pbW3NUVhj01CdprOnl7ZO3zRlZvknF4l+HXBsRCwFvgPcP3ihpErgXuCaiGgfaSMRsSIimiKiqa6uLgdhjd3ATVNupzezPDThRB8R7RGxN5l+ECiRVAsgqYRskr8rIu6b6L4mS4OHFDSzPDbhRC+pXpKS6TOTbe5K5t0ObIqIv57ofiZTY1Kj3+rHIJhZHhq1142kVcByoFZSM3A9UAIQEbcCFwN/LCkDdAKXRERIei/wKeBZSeuTzf1VUuufUeqqyihKyTV6M8tLoyb6iLh0lOW3kO1+OXT+LwGNP7SpU5QSc6s80pSZ5aeCvzO2X0ONn0tvZvnJiT7RUJ12042Z5SUn+kRjUqP3TVNmlm+c6BMN1Wm6M33s3ndgukMxM8spJ/rEmwOQuJ3ezPKLE33Cd8eaWb5yok/47lgzy1dO9InaijJKi1JsaG6b7lDMzHLKiT6RSolPnnE096xt5p61zdMdjplZzjjRD3Ldhxfx7uPn8Jf3beBXLx/yI/jNzGYkJ/pBSopS/N0fns7RR8zi899bwys79013SGZmE+ZEP0T1rBJWfvYMAvijf3iKtv0ejMTMDm9O9MNYUFvBrZefzuu79/On/2cdPb190x2Smdm4OdGP4Kzj5vBfP3Yqv9y8k+t/9JwfjWBmh61RH1NcyP6g6Wi2tO7j1sd+w/F1lVzx3oXTHZKZ2SFzoh/FX7z/ZF7euZcbf/I8C+bM4vxT5k53SGZmh8RNN6NIpcS3Pnkaixtn8x9WPc2mlhHHNzczm5FGTfSSVkraIWnjCMuXS2qTtD55XTdo2YWSXpC0WdK1uQx8Ks0qLea2T59BZbqYK+54ih0dfh6OmR0+xlKjvwO4cJQyv4iI05LXDQCSioC/BT4ALAIulbRoIsFOp/rqNLd/5gze2N/Dv79zLV09vdMdkpnZmIya6CNiNTCe20TPBDZHxJaIOADcDXx0HNuZMZbMq+bbl5zGhuY9fPEfn6Gvzz1xzGzmy1Ub/dmSnpH0kKTFybx5wOuDyjQn84Yl6UpJayStaW1tzVFYuff+xfV8+cJ38JMNLXz7Zy9NdzhmZqPKRaJfBxwbEUuB7wD3J/M1TNkRq8ARsSIimiKiqa6uLgdhTZ7Pn3Mcf9A0n5t/9hL3P/3b6Q7HzOygJpzoI6I9IvYm0w8CJZJqydbgjx5UdD6wdaL7mwkkcePvn8rvLDySv7hnA2te8QPQzGzmmnCil1QvScn0mck2dwFPASdKWiipFLgE+NFE9zdTlBanuPXy02msSfP5763l9d37pzskM7NhjaV75SrgceBkSc2SrpB0laSrkiIXAxslPQPcDFwSWRngC8DDwCbg/0bEc5PzZ0yPIypKuf2zZ9DT28cf3fEU7V1+AJqZzTyaic9waWpqijVr1kx3GGP2r5t38umVv+LdJ9Sy8jNNFBf5PjQzm1qS1kZE03DLnJFy4N0n1HLj7y9h9YutfOr2X7H21TemOyQzswF+1k2OXHLmMfT0Bd9+5EU+/nf/yjkn1XH1+Sdy+rFHTHdoZlbg3HSTY/sPZPje46/yv1dvYfe+A5xzUh3XXHAiy45xwjezyXOwphsn+kmyrzvD9554lRVJwj83SfjvcsI3s0ngRD+Nhib85SfXcc0FJ3Ha0TXTHZqZ5REn+hlgX3eGOx9/lRWrf8Mb+3t438l1XO2Eb2Y54kQ/g+ztznDn46/w96u3DCT8ay44iaVO+GY2AU70M1B/wl+xegt79vdw3juO4urzT3TCN7NxcaKfwfZ2Z/iHf32Fv/9FNuH/zsIj+fjp8/ngqQ1Ulrn3q5mNjRP9YaCjq4fvP/EaP3jqNV7ZtZ/ykiIuXFLPx5fN5+zj51CUGu5hoGZmWU70h5GIYN1rb3DP2t/ywIatdHRlaKhO87F3zePjp8/n+LrK6Q7RzGYgJ/rDVFdPL488v5371jXz2Iut9AUsPbqGi5fN48NLG6mZVTrdIZrZDOFEnwd2dHTxw6e3cu+6Zn69rYPSohTnn3IUH182n3NPrqPED1IzK2hO9HkkInhuazv3rmvmR+u3smvfAeZUlPKR0xr5+LL5LG6cTTI8gJkVECf6PNXT28djL7Ry77pmfrZpBwd6+zjxqEo+9M5GLnpnAycc5fZ8s0LhRF8A9uw/wI83tPDjZ7by1Cu7iYB31FfxoXc2cNE7G1lYWzHdIZrZJHKiLzDb27t48NkWfrKhhTXJs/EXN87O1vRPbeCYObOmOUIzy7UJJ3pJK4EPATsiYslByp0BPAF8MiLuSeZ9E7iI7CAnjwBXxyg7daLPna17Onnw2RYe2NDC+tf3ALB0fjUXJTX9eTXl0xyhmeVCLhL9OcBe4M6REr2kIrKJvAtYGRH3SHo3cBNwTlLsl8BfRsTPD7Y/J/rJ8fru/dma/rMtbGhuA+Bdx9Rw0akNXPTOBhqqnfTNDlcHS/Rjusc+IlZLWjBKsT8D7gXOGLwqkAZKAQElwPax7NNy7+gjZ/H5c4/n8+cez6u79vGTZ1t44JkWbvzJJm78ySaajj2Cdx8/h4V1FSysrWRhbQXV5SXTHbaZTVBOHqYiaR7wMeA8BiX6iHhc0qNAC9lEf0tEbBphG1cCVwIcc8wxuQjLDuLYORX8yfIT+JPlJ7CldW9S09/GLY9upm/Qj7w5FaUsrK3IvuoqOK42exI4ds4s0iVF0/cHmNmY5eqpWd8GvhwRvYP7cEs6ATgFmJ/MekTSORGxeugGImIFsAKyTTc5isvG4Li6Sr5w3ol84bwTOZDp47Xd+3l55z5e3rmXl3fuY0vrPh57sZV/XNs8sI4EjdXlHFdXMXAiWFBbwQl1lcyrKSflZ/OYzRi5SvRNwN1Jkq8FPigpA5wIPBERewEkPQScBbwt0dvMUFqc4oSjKpM++HPfsmxvd4ZXdu5jy859vNz65ongn57+LR1dmYFys0qLOHFuFScdVclJc6s4qb6Kk+ZWUj877Zu5zKZBThJ9RCzsn5Z0B/BARNwv6ZPAv5f038k23ZxLtvZvh6HKsmKWzKtmybzqt8yPCHbtO8CW1n1s3rGXF7d38NKODh594a2/AqrSxdnEP7eSE4+q4uT6Kk6cW0ldZZlPAGaTaEyJXtIqYDlQK6kZuJ7shVUi4taDrHoP2Xb7Z8lemP1pRPx4IgHbzCOJ2soyaivLOHPhkW9Z9sa+A7y4vSN57eWF7R38dOM2Vu1/faBMzaySgRPAgjkVNFSXU1+dpqE6zVFVZRT7OT5mE+IbpmzKRQQ79771BPDS9g5e2N7xliYggJTgqKr0QOLvf2+oLh/4PHd22g91s4I34e6VZrkkibqqMuqqynjPCbUD8yOC9s4MLe2dtLR1sa2ti5a2Llr2dLKtvYuXduxl9Yut7DvQO2R7UFtZRuPAieDNXwT1s7Of51aXUVbsXkJWmJzobcaQRPWsEqpnlfCO+tkjluvo6smeANq62Nb25klha1sXr+zcz+O/2UX7kF8GkO0q+tZfBuXJieDNz+WlPhlY/nGit8NOVbqEqnS2XX8k+7ozbGt/66+CluTzb/d0sfbVN3hjf8/b1pudLh5oDsq+ygZNZ38h1FaW+rqBHVac6C0vVZQVc3xd5UGHXuzq6R04EWxr72Trni52tHexvb2bbe1d/GbHTnZ0dJPpe+t1rP6movohJ4L62WnqZpdRV1nGUbPLmFNR5rF+bUZworeClS4pYkFyo9dI+vqyXUe3t3exvb2LbcmJYHtbF9s7umh+o5N1r+1h974Db1s3JTiyooyjkusRdVVvnc6eENLUVZVRUVrkLqY2aZzozQ4ilXrzwvHQ+wcG6870sqO9mx0d3bR2dNPa0ZV939vNjvbs+4vbO2gd5hcCQHlJEUfNLuOIWaVUl5cwu7yE2eniQdMlzC5PPqffXD67vMQ9jmxUTvRmOVBWXMTRR87i6CMP/qz/vr5gT2cPrR3d7Og/GXRkTxA7OrrZs/8Ae/Yf4LXd+2nv7KGts2fYE8Ngs0qLBk4E5aXFlKREcZEoKUpRnBLFRSlKikRxKpWd3/9elKKov2wyr7Q4Rbq4iLKSt76nS4abl6Is+VxWnPIvkhnMid5sCqVS4siKUo6sKOXk+pEvJveLCDp7emnvzNDW2UN7Vw/tyXvb/h7auzJvfu7sobOnj0xvH5neYG8mQ6Y36OntI9MXZHr76OkNMn19Q+Zn541yPhlVWRF4r0MAAAgKSURBVHGKqnTJ25upKt/edFVZVuwTwxRyojebwSQxq7SYWaXZ3kCTqa8vONDbR3dPH12ZXrp6eunO9L31fWBZH93DvLcnv1ZaO7rZvL2D1r3d9PS+/QySLkkliT/9lhNBzawSSotSlBSlKC1+81U25HNJUYrSouwviYH5RSn3hhqBE72ZAdlfG+lUtpmmmtyMQ9DXF7R19tC6t3vY5qrWjm5+07qXJ17exZ5huruOR1FKFEmkUpBS/7QoSomUREoMTBcNzM+WLS5KUZo0a/WfULLTGjgBlSQnlexyUVpUREmyvLQ4RbokewzLS7LNW+XJ5/Qwn6eqV5YTvZlNmlRKHFFRyhEVpQe97wGyF7Q7ujIcyPRlX71D3g82P5nO9PbRG0FvX7bZq7cv6I2gL3kffj70JdP9TVsHMn309Pax70DvwHRP75vz+/fZ05vd1niVFqVIl2RPDuWlRcytSvN/rzp73NsbiRO9mc0IZcVFlFUefncm9/Zlr3cMNHv19NKd6aXzQLaZq/NAthmsM2n66ux583NXUr7/c/kkDebjRG9mNgHZ5p9sUwyTexll3HzlwswszznRm5nlOSd6M7M8N2qil7RS0g5JG0cpd4akXkkXD5p3jKR/lrRJ0vOSFkw8ZDMzOxRjqdHfAVx4sAKSioBvAA8PWXQncFNEnAKcCewYR4xmZjYBoyb6iFgN7B6l2J8B9zIokUtaBBRHxCPJdvZGxP4JxGpmZuMw4TZ6SfOAjwFDBwk/Cdgj6T5JT0u6Kan5m5nZFMrFxdhvA1+OiN4h84uB3wW+BJwBHAd8dqSNSLpS0hpJa1pbW3MQlpmZQW5umGoC7k6eRFcLfFBSBmgGno6ILQCS7gfOAm4fbiMRsQJYkZRtlfTqOOOpBXaOc92p4PgmxvFNjOObmJkc37EjLZhwoo+Ihf3Tku4AHoiI+5NmmiMk1UVEK3AesGaM26wbbzyS1kRE03jXn2yOb2Ic38Q4vomZ6fGNZNREL2kVsByoldQMXA/ZR9tFxNB2+QER0SvpS8DPlK3urwX+PhdBm5nZ2I2a6CPi0rFuLCI+O+TzI8A7Dz0sMzPLlXy8M3bFdAcwCsc3MY5vYhzfxMz0+IaliAmOH2ZmZjNaPtbozcxsECd6M7M8d9gmekkXSnpB0mZJ1w6zvEzSD5LlT07lA9UkHS3p0eRhbs9JunqYMssltUlan7yum6r4kv2/IunZZN9v6/aqrJuT47dB0rIpjO3kQcdlvaR2SdcMKTOlx2+4h/tJOlLSI5JeSt6PGGHdzyRlXpL0mSmM7yZJv07+/f5JUs0I6x70uzCJ8X1V0m8H/Rt+cIR1D/p/fRLj+8Gg2F6RtH6EdSf9+E1YRBx2L6AI+A3Zu21LgWeARUPK/AlwazJ9CfCDKYyvAViWTFcBLw4T33Ky9xxM1zF8Bag9yPIPAg8BInuj25PT+G+9DTh2Oo8fcA6wDNg4aN43gWuT6WuBbwyz3pHAluT9iGT6iCmK7/fIPm8Ksg8dfFt8Y/kuTGJ8XwW+NIZ//4P+X5+s+IYs/5/AddN1/Cb6Olxr9GcCmyNiS0QcAO4GPjqkzEeBf0im7wHOT/rzT7qIaImIdcl0B7AJmDcV+86hjwJ3RtYTQI2khmmI43zgNxEx3julcyKGf7jf4O/YPwC/P8yq7wceiYjdEfEG8AijPA02V/FFxD9HRCb5+AQwP9f7HasRjt9YjOX/+oQdLL4kb/wBsCrX+50qh2uinwe8PuhzM29PpANlki97GzBnSqIbJGkyehfw5DCLz5b0jKSHJC2e0sAggH+WtFbSlcMsH8sxngqXMPJ/sOk8fgBzI6IFsid34KhhysyU4/hHZH+hDWe078Jk+kLStLRyhKavmXD8fhfYHhEvjbB8Oo/fmByuiX64mvnQfqJjKTOpJFWSfXzzNRHRPmTxOrLNEUuB7wD3T2VswHsiYhnwAeBPJZ0zZPlMOH6lwEeAfxxm8XQfv7GaCcfxPwMZ4K4Rioz2XZgsfwccD5wGtJBtHhlq2o8fcCkHr81P1/Ebs8M10TcDRw/6PB/YOlIZScVANeP76TgukkrIJvm7IuK+ocsjoj0i9ibTDwIlkmqnKr6I2Jq87wD+iexP5MHGcown2weAdRGxfeiC6T5+ie39zVnJ+3AD60zrcUwu/n4I+MNIGpSHGsN3YVJExPaI6I2IPrKPRxluv9N9/IqBfwv8YKQy03X8DsXhmuifAk6UtDCp9V0C/GhImR8B/T0cLgb+ZaQveq4lbXq3A5si4q9HKFPff81A0plk/y12TVF8FZKq+qfJXrQbOlTkj4BPJ71vzgLa+pspptCINanpPH6DDP6OfQb44TBlHgZ+T9IRSdPE7/H2kdgmhaQLgS8DH4kRBv0Z43dhsuIbfM3nYyPsdyz/1yfTBcCvI6J5uIXTefwOyXRfDR7vi2yvkBfJXpH/z8m8G8h+qQHSZH/ybwZ+BRw3hbG9l+zPyw3A+uT1QeAq4KqkzBeA58j2IngCePcUxndcst9nkhj6j9/g+AT8bXJ8nwWapvjfdxbZxF09aN60HT+yJ5wWoIdsLfMKstd8fga8lLwfmZRtAm4btO4fJd/DzcDnpjC+zWTbt/u/g/290BqBBw/2XZii+L6XfLc2kE3eDUPjSz6/7f/6VMSXzL+j/zs3qOyUH7+JvvwIBDOzPHe4Nt2YmdkYOdGbmeU5J3ozszznRG9mluec6M3M8pwTvZlZnnOiNzPLc/8fqp4RDweFcGMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title('train loss')\n",
    "plt.plot(np.arange(len(losses)), losses)\n",
    "plt.legend(['Train Loss'], loc='upper right')\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}