![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "V30P_YpIJI4p",
"colab_type": "text"
},
"source": [
"# FastAI-深度学习实战-宠物分类\n",
"\n",
"- 【2019-04-29】王奇文，wqw547243068@163.com，[School of AI-Beijing](https://wqw547243068.github.io/school-of-ai-beijing/)\n",
"\n",
"\n",
"## 第一课 宠物图像分类 Lesson 1-What's your pets ?\n",
"\n",
"- 内容：从零开始，完成图像分类任务，并达到世界领先水平\n",
"- 数据集：[Oxford-IIIT Pet Dataset](http://www.robots.ox.ac.uk/~vgg/data/pets/)是一个宠物图像数据集，包含12种猫和25种狗,每种宠物200张左右宠物图片，并同时包含宠物轮廓标注信息.共753M\n",
" - \n",
" - \n",
" - \n",
"- 任务：区分所有物种，2分类、37分类\n",
" - \n",
"- 效果：2012年，[O. M. Parkhi et al](http://www.robots.ox.ac.uk/~vgg/publications/2012/parkhi12a/parkhi12a.pdf).论文里用了一个复杂模型，先区分图像，再分清头和身体，费了好大劲，最后取得成绩是：分类准确率59.21%\n",
"- 作为刚入门的新人，我们能做到什么程度呢？拭目以待！"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b-4k6JSGKEQz",
"colab_type": "text"
},
"source": [
"## 环境准备\n",
"- 按照指定格式准备目录结构\n",
" - 在/root目录下创建.torch和.fastai目录，里面分别包含models和data子目录\n",
" - 简历软连接，指向/content\n",
"- curl命令从远程下载bash脚本,，并通过管道(|)命令执行，内容是：\n",
"```bash\n",
"#!/bin/bash\n",
"# 主要功能：fastai本地环境部署\n",
"if [ ! -e /content/models ]; then\n",
" mkdir -p /root/.torch/models\n",
" mkdir -p /root/.fastai/data\n",
" ln -s /root/.torch/models /content\n",
" ln -s /root/.fastai/data /content\n",
" rm -rf /content/sample_data/\n",
"fi\n",
"```"
]
},
{
"cell_type": "code",
"metadata": {
"id": "tB-xy8GUGEmm",
"colab_type": "code",
"outputId": "473e5234-a2c9-48c9-f9b1-35dc8983b3ab",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 50
}
},
"source": [
"!curl -s https://course.fast.ai/setup/colab | bash"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Updating fastai...\n",
"Done.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QcFig3LaISBc",
"colab_type": "text"
},
"source": [
"### kaggle环境准备\n",
"- kaggle的云环境目录结构不同于colab\n",
"- /kaggle底下有如下目录：\n",
" - config：配置信息，默认只有jupyter的配置文件jupyter.json\n",
" - input：数据集存放位置（注意：尽量去dataset区域寻找数据集，不要自己上传）\n",
" - 比如：点页面右上角“Add Dataset”，搜索oxfordiiit-pet-dataset，添加到自己的空间\n",
" - 查看是否添加成功，执行以下命令：\n",
"```shell\n",
" !ls /kaggle/input/the-oxfordiiit-pet-dataset\n",
"```\n",
" - lib：库，不用管\n",
" - working：默认工作目录，写操作限制在这个目录下\n",
"- 注意：\n",
" - 请开启右下角Settings区域里的internet（联网下载数据）+GPU选项（训练加速）\n",
" - 请开启右下角Settings区域里的internet（联网下载数据）+GPU选项（训练加速）"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "_UodEwRhmGxg",
"colab_type": "text"
},
"source": [
"### Google Drive\n",
"- 如果数据保存在谷歌云上，需要挂载"
]
},
{
"cell_type": "code",
"metadata": {
"id": "TGGaiIywmFgW",
"colab_type": "code",
"outputId": "92031382-7299-4745-f5eb-3acc9ede9321",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 121
}
},
"source": [
"from google.colab import drive\n",
"# 将google硬盘挂载在/comtent/drive/目录上面\n",
"drive.mount('/content/drive/') \n",
"# 点击提示的链接，将认证码输入并按enter"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
"\n",
"Enter your authorization code:\n",
"··········\n",
"Mounted at /content/drive/\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "fylHY0ddntQW",
"colab_type": "code",
"outputId": "02dc35f0-e872-4dc2-da08-0980d5025cb7",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 336
}
},
"source": [
"ls /content/drive/My\\ Drive"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
" \u001b[0m\u001b[01;34m20160200-college-de-france\u001b[0m/\n",
"\u001b[01;34m'Colab Notebooks'\u001b[0m/\n",
"'Copy of How neural networks work.gslides'\n",
"\u001b[01;34m'DeepBayes2018 presentations'\u001b[0m/\n",
"\u001b[01;34m'deep learning'\u001b[0m/\n",
"'How neural networks work.gslides'\n",
"'Idle SMS Recall Analysis.gsheet'\n",
" ijcai_ecai_tutorial.pdf\n",
" \u001b[01;34mPyTorchZeroToAll\u001b[0m/\n",
"\u001b[01;34m'School of AI'\u001b[0m/\n",
"'SMS 测试.gsheet'\n",
" \u001b[01;34mSoAI\u001b[0m/\n",
" \u001b[01;34mTantan\u001b[0m/\n",
" \u001b[01;34mtensorflow\u001b[0m/\n",
"'Untitled document.gdoc'\n",
"'Untitled drawing.gdraw'\n",
"'已从 Google 笔记本导入 - My first notebook.gdoc'\n",
"'已从 Google 笔记本导入 - Notebook 1.gdoc'\n",
" 手机测试文档.docx\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0Qf48X9nmWFX",
"colab_type": "text"
},
"source": [
"### 其他\n",
"- 可视化：绘图工具嵌入jupyter页面\n",
"- 自动刷新功能"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3qO-4awHKdJ_",
"colab_type": "text"
},
"source": [
"- 以下3行代码是每个jupyter笔记都需要添加的，作用是：\n",
" - 确保调用的库自动更新——autoreload\n",
" - 加载画图程序，直接嵌入笔记本——matplotlib"
]
},
{
"cell_type": "code",
"metadata": {
"id": "gILGnWR8GTRb",
"colab_type": "code",
"colab": {}
},
"source": [
"%reload_ext autoreload\n",
"%autoreload 2 # 每隔2s自动刷新库\n",
"%matplotlib inline # 开启matplotlib嵌入功能，直接显示在笔记内"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "8JPDQen4LAPh",
"colab_type": "text"
},
"source": [
"## fastai包导入\n",
"- fastai是在pytorch 1.0基础上构建的深度学习工具包\n",
"- 特点：开发速度快，直接使用最先进的模型"
]
},
{
"cell_type": "code",
"metadata": {
"id": "N6A0yd05HfbN",
"colab_type": "code",
"colab": {}
},
"source": [
"from fastai.vision import * # 导入所有子包\n",
"from fastai.metrics import error_rate # 评估函数：错误率"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "FhV-Hh-9Q8E-",
"colab_type": "text"
},
"source": [
"### fastai模块\n",
"- fastai在pytorch上高度封装\n",
" - fastai是一种超级简洁的深度学习工具，代码上甚至比Keras还要简洁。\n",
" - 底层是pytorch，能够提供高速的处理性能\n",
"- 模块架构\n",
" - \n",
"- 常用方法：从左到右分别是核心代码、训练代码和应用代码\n",
"- \n",
"- 参考：[使用fastai进行图像分类之120种小狗](https://www.jianshu.com/p/ab35ed21df87)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ptim84wfQCr9",
"colab_type": "text"
},
"source": [
"## GPU验证"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XvrLpnG2a9HI",
"colab_type": "text"
},
"source": [
"- ① NVIDIA自带\n",
" - NVIDIA自带的驱动检测方法，能看到GPU的配置\n",
" - 可以看到colab的GPU是Tesla T4，显存15G，强于上一版本K80 "
]
},
{
"cell_type": "code",
"metadata": {
"id": "KM1SZawJa3nk",
"colab_type": "code",
"outputId": "1b7cb838-55cf-4faa-fdcc-529e80372122",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 286
}
},
"source": [
"!nvidia-smi"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Wed May 1 03:24:04 2019 \n",
"+-----------------------------------------------------------------------------+\n",
"| NVIDIA-SMI 418.56 Driver Version: 410.79 CUDA Version: 10.0 |\n",
"|-------------------------------+----------------------+----------------------+\n",
"| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n",
"| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n",
"|===============================+======================+======================|\n",
"| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |\n",
"| N/A 71C P0 31W / 70W | 1323MiB / 15079MiB | 0% Default |\n",
"+-------------------------------+----------------------+----------------------+\n",
" \n",
"+-----------------------------------------------------------------------------+\n",
"| Processes: GPU Memory |\n",
"| GPU PID Type Process name Usage |\n",
"|=============================================================================|\n",
"+-----------------------------------------------------------------------------+\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "NnZESzAEcV99",
"colab_type": "text"
},
"source": [
"- ② GPU监控工具-[gpustat](https://www.ctolib.com/wookayin-gpustat.html)\n",
" - 仅适用于N卡（NVIDIA ），A卡不行（AMD）\n",
" - 效果示例："
]
},
{
"cell_type": "code",
"metadata": {
"id": "4Aws4gkRbTq4",
"colab_type": "code",
"outputId": "de3c4a83-9601-4d9a-8ee2-21ca8d4a06a9",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 269
}
},
"source": [
"#!pip install gpustat\n",
"!gpustat -cp"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Collecting gpustat\n",
" Downloading https://files.pythonhosted.org/packages/12/18/7b3b08a7ba21d637c159c8920b183043c5ba1fcdaed6c7dc06e51312af11/gpustat-0.5.0.tar.gz\n",
"Requirement already satisfied: six>=1.7 in /usr/local/lib/python3.6/dist-packages (from gpustat) (1.12.0)\n",
"Requirement already satisfied: nvidia-ml-py3>=7.352.0 in /usr/local/lib/python3.6/dist-packages (from gpustat) (7.352.0)\n",
"Requirement already satisfied: psutil in /usr/local/lib/python3.6/dist-packages (from gpustat) (5.4.8)\n",
"Collecting blessings>=1.6 (from gpustat)\n",
" Downloading https://files.pythonhosted.org/packages/03/74/489f85a78247609c6b4f13733cbf3ba0d864b11aa565617b645d6fdf2a4a/blessings-1.7-py3-none-any.whl\n",
"Building wheels for collected packages: gpustat\n",
" Building wheel for gpustat (setup.py) ... \u001b[?25ldone\n",
"\u001b[?25h Stored in directory: /root/.cache/pip/wheels/2e/7a/86/ae2c167876a53a7519aed8ae8543455f50825384268e1d7245\n",
"Successfully built gpustat\n",
"Installing collected packages: blessings, gpustat\n",
"Successfully installed blessings-1.7 gpustat-0.5.0\n",
"\u001b[1m\u001b[37m54b5f69d9cf8\u001b[m Wed May 1 03:30:04 2019\n",
"\u001b[36m[0]\u001b[m \u001b[34mTesla T4 \u001b[m |\u001b[1m\u001b[31m 71'C\u001b[m, \u001b[32m 0 %\u001b[m | \u001b[36m\u001b[1m\u001b[33m 1323\u001b[m / \u001b[33m15079\u001b[m MB |\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ClsnG0TVap8c",
"colab_type": "text"
},
"source": [
"- ③ Pytorch的GPU测试方法"
]
},
{
"cell_type": "code",
"metadata": {
"id": "g5QovkWjQGad",
"colab_type": "code",
"outputId": "c456c23b-645b-4f0c-a36d-a44ae3b92f99",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"torch.cuda.is_available()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"True"
]
},
"metadata": {
"tags": []
},
"execution_count": 15
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "p_a6lolmTLdk",
"colab_type": "code",
"outputId": "70d6967b-1e7d-4390-a93f-de67a2e061b6",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"torch.backends.cudnn.enabled"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"True"
]
},
"metadata": {
"tags": []
},
"execution_count": 14
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "fsKkL7BJaQ6x",
"colab_type": "text"
},
"source": [
"- ④ Tensorflow的GPU测试代码"
]
},
{
"cell_type": "code",
"metadata": {
"id": "kkC2mSdcaMoI",
"colab_type": "code",
"outputId": "9e5b8d96-1f10-4cdb-a22b-86d0969952ab",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"import tensorflow as tf\n",
"device_name = tf.test.gpu_device_name()\n",
"if device_name != '/device:GPU:0':\n",
" raise SystemError('GPU device not found')\n",
"print('Found GPU at: {}'.format(device_name))"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Found GPU at: /device:GPU:0\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UpN903wpL6W2",
"colab_type": "text"
},
"source": [
"## 设置batch_size\n",
"- bs是每个喂给模型的样本量，数字越大，越吃内存\n",
"- 如果运行过程中，内存溢出，请将bs值调小\n",
"- bs数值符合2^n，为什么？\n",
" - 计算机底层设计有关，电脑的CPU/GPU处理的都是二进制，不管32位还是64位，为了最大限度利用起来，需要设置成2的幂级数形式\n",
" - 计算机底层设计有关，电脑的CPU/GPU处理的都是二进制，不管32位还是64位，为了最大限度利用起来，需要设置成2的幂级数形式"
]
},
{
"cell_type": "code",
"metadata": {
"id": "9s7k7ujpHtqZ",
"colab_type": "code",
"colab": {}
},
"source": [
"# bs = 64\n",
"# bs = 16\n",
"bs = 8"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "3DypJ_MTM85E",
"colab_type": "text"
},
"source": [
"## 数据准备\n",
"- 下载图像数据集到本地\n",
"- 数据拆分：特征+标签"
]
},
{
"cell_type": "code",
"metadata": {
"id": "5pvy8ouMNQSU",
"colab_type": "code",
"outputId": "aea7e61b-59a7-4d9a-ba67-8a933ef70e55",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"# URLs包含了很多有用的数据集地址，如mnist\n",
"#URLs.MNIST\n",
"URLs.PETS"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"'https://s3.amazonaws.com/fast-ai-imageclas/oxford-iiit-pet'"
]
},
"metadata": {
"tags": []
},
"execution_count": 17
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "4REKdAg_H5iU",
"colab_type": "code",
"outputId": "1c82f167-d7f6-4400-daaf-057f8d0bff54",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"# 从亚马逊云上下载数据，宠物数据集地址：https://s3.amazonaws.com/fast-ai-imageclas/oxford-iiit-pet'\n",
"# 如果无法访问，直接去牛津大学数据集地址：http://www.robots.ox.ac.uk/~vgg/data/pets/\n",
"PET_URL = 'http://www.robots.ox.ac.uk/~vgg/data/pets' \n",
"path = untar_data(URLs.PETS) # 解压缩\n",
"#path = untar_data(PET_URL)\n",
"path"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"PosixPath('/root/.fastai/data/oxford-iiit-pet')"
]
},
"metadata": {
"tags": []
},
"execution_count": 18
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TZgZeqywPFfi",
"colab_type": "text"
},
"source": [
"- path包包含了大量linux本地操作命令，如：ls、mkdir等"
]
},
{
"cell_type": "code",
"metadata": {
"id": "12uvDAObIM5z",
"colab_type": "code",
"outputId": "06e9e396-ca47-4b94-c2ce-13f97790c877",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 50
}
},
"source": [
"path.ls() # 查看当前目录下的文件\n",
"#!pwd # 除了调用path子包，还可以加!直接调用linux命令"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"[PosixPath('/root/.fastai/data/oxford-iiit-pet/images'),\n",
" PosixPath('/root/.fastai/data/oxford-iiit-pet/annotations')]"
]
},
"metadata": {
"tags": []
},
"execution_count": 19
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jXJFfD_pPkPq",
"colab_type": "text"
},
"source": [
"## 数据读取\n",
"- 数据集分成图片及标签，分别对应images和annotation目录\n",
"- 此处可以通过linux命令计算图片数目\n",
"- 用python的PIL和matplotlib显示图片内容"
]
},
{
"cell_type": "code",
"metadata": {
"id": "9J6_JbJfIzwc",
"colab_type": "code",
"outputId": "883aad61-4b26-4e7a-d431-7d461039159f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 84
}
},
"source": [
"path_anno = path/'annotations'\n",
"path_img = path/'images'\n",
"#!ls $path_img # 显示目录下的图片文件: beagle_194.jpg和beagle_194.mat\n",
"!ls $path_img | wc -l # 显示文件数目7393\n",
"!ls $path_img | awk -F'.' '{print $2}'|sort|uniq -c\n",
"!echo \"显示annotation目录内容：\"\n",
"!ls $path_anno"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"7393\n",
" 1 \n",
"显示annotation目录内容：\n",
"list.txt README test.txt trainval.txt trimaps xmls\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "THpEtxjb7YTd",
"colab_type": "code",
"outputId": "4b8ae457-e47a-4774-b9b2-5c162c31376e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 168
}
},
"source": [
"!wc -l $path_anno/*"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
" 7355 /root/.fastai/data/oxford-iiit-pet/annotations/list.txt\n",
" 45 /root/.fastai/data/oxford-iiit-pet/annotations/README\n",
" 3669 /root/.fastai/data/oxford-iiit-pet/annotations/test.txt\n",
" 3680 /root/.fastai/data/oxford-iiit-pet/annotations/trainval.txt\n",
"wc: /root/.fastai/data/oxford-iiit-pet/annotations/trimaps: Is a directory\n",
" 0 /root/.fastai/data/oxford-iiit-pet/annotations/trimaps\n",
"wc: /root/.fastai/data/oxford-iiit-pet/annotations/xmls: Is a directory\n",
" 0 /root/.fastai/data/oxford-iiit-pet/annotations/xmls\n",
" 14749 total\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "8N8sR5f3sjg9",
"colab_type": "code",
"outputId": "720bd879-3fcb-49f1-c401-f0a9f43ba856",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 185
}
},
"source": [
"!head $path_anno/list.txt"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"#Image CLASS-ID SPECIES BREED ID\n",
"#ID: 1:37 Class ids\n",
"#SPECIES: 1:Cat 2:Dog\n",
"#BREED ID: 1-25:Cat 1:12:Dog\n",
"#All images with 1st letter as captial are cat images\n",
"#images with small first letter are dog images\n",
"Abyssinian_100 1 1 1\n",
"Abyssinian_101 1 1 1\n",
"Abyssinian_102 1 1 1\n",
"Abyssinian_103 1 1 1\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "dD_w_Wmv7HOj",
"colab_type": "code",
"outputId": "ace8a0ef-a05b-4d7a-c0f7-389d58812b65",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 353
}
},
"source": [
"!head $path_anno/trainval.txt\n",
"!head $path_anno/test.txt"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Abyssinian_100 1 1 1\n",
"Abyssinian_101 1 1 1\n",
"Abyssinian_102 1 1 1\n",
"Abyssinian_103 1 1 1\n",
"Abyssinian_104 1 1 1\n",
"Abyssinian_105 1 1 1\n",
"Abyssinian_106 1 1 1\n",
"Abyssinian_107 1 1 1\n",
"Abyssinian_108 1 1 1\n",
"Abyssinian_109 1 1 1\n",
"Abyssinian_201 1 1 1\n",
"Abyssinian_202 1 1 1\n",
"Abyssinian_204 1 1 1\n",
"Abyssinian_205 1 1 1\n",
"Abyssinian_206 1 1 1\n",
"Abyssinian_207 1 1 1\n",
"Abyssinian_20 1 1 1\n",
"Abyssinian_210 1 1 1\n",
"Abyssinian_212 1 1 1\n",
"Abyssinian_213 1 1 1\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FUAwkn8KrNmr",
"colab_type": "text"
},
"source": [
"- 文件说明\n",
" - list.txt——所有数据集文件名列表\n",
" - trainval.txt——训练集、验证集文件名列表\n",
" - test.txt——测试集文件名列表\n",
" - list = trainval + test\n",
"- label文件trainval.txt说明：\n",
"\n",
"|Name-图片文件名|Class ID-类目ID|SPECIES-物种ID|BREED ID-物种类目ID|\n",
"|---|---|---|---|\n",
"|Abyssinian_100|1|1|1|\n",
"|Abyssinian_101|1|1|1|\n",
"|...|...|...|...|\n",
"|Siamese_157|33|1|11|\n",
"|Siamese_158|33|1|11|\n",
"|yorkshire_terrier_189|37|2|25|\n",
"|yorkshire_terrier_18|37|2|25|\n",
"|yorkshire_terrier_190|37|2|25|\n",
"\n",
"备注：\n",
"- Class ID 是对应于pet_label_map.pbtxt的ID值\n",
"- SPECIES是总分类:1:猫 2:狗\n",
"- BREED ID :在分类下面的子分类序号,对于总分类1猫其序号为1-25;对于总分类2狗，其序号为1-12。\n",
"- [采用Oxford-IIIT Pets数据训练Object-Detect](https://www.jianshu.com/p/3b472539b88a)\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "5eDQNhrARy5h",
"colab_type": "code",
"outputId": "6de18e7b-5366-41a1-adad-0e836b6f464d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 286
}
},
"source": [
"#!ls $path_img | head\n",
"from matplotlib import pyplot as plt\n",
"from PIL import Image\n",
"\n",
"img_file = '%s/beagle_194.jpg'%(path_img) # 随便选一张图片\n",
"plt.imshow(Image.open(img_file))"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"| epoch | \n", "train_loss | \n", "valid_loss | \n", "accuracy | \n", "time | \n", "
|---|---|---|---|---|
| 0 | \n", "0.140596 | \n", "0.082088 | \n", "0.973013 | \n", "00:08 | \n", "
| epoch | \n", "train_loss | \n", "valid_loss | \n", "error_rate | \n", "time | \n", "
|---|---|---|---|---|
| 0 | \n", "0.888307 | \n", "0.403361 | \n", "0.129229 | \n", "02:01 | \n", "
| 1 | \n", "0.796944 | \n", "0.339743 | \n", "0.099459 | \n", "02:02 | \n", "
| 2 | \n", "0.616858 | \n", "0.263689 | \n", "0.080514 | \n", "02:02 | \n", "
| 3 | \n", "0.471726 | \n", "0.245238 | \n", "0.078484 | \n", "02:02 | \n", "
_cl_int_plot_top_losses[source][test]\n", "\n", "
_cl_int_plot_top_losses(k,largest=True,figsize=(12, 12),heatmap:bool=True,heatmap_thresh:int=16,return_fig:bool=None) →Optional[Figure]
No tests found for _cl_int_plot_top_losses. To contribute a test please refer to this guide and this discussion.
Show images in top_losses along with their prediction, actual, loss, and probability of actual class.
| epoch | \n", "train_loss | \n", "valid_loss | \n", "error_rate | \n", "time | \n", "
|---|---|---|---|---|
| 0 | \n", "0.982449 | \n", "0.445620 | \n", "0.133288 | \n", "02:27 | \n", "
| epoch | \n", "train_loss | \n", "valid_loss | \n", "error_rate | \n", "time | \n", "
|---|---|---|---|---|
| 0 | \n", "0.435095 | \n", "0.238926 | \n", "0.072395 | \n", "02:26 | \n", "
| epoch | \n", "train_loss | \n", "valid_loss | \n", "error_rate | \n", "time | \n", "
|---|---|---|---|---|
| 0 | \n", "1.061286 | \n", "0.276833 | \n", "0.086604 | \n", "04:34 | \n", "
| epoch | \n", "train_loss | \n", "valid_loss | \n", "error_rate | \n", "time | \n", "
|---|---|---|---|---|
| 0 | \n", "0.745421 | \n", "0.249251 | \n", "0.081191 | \n", "06:07 | \n", "