前言
最近根据制定的“Deep Learning”学习计划，11月份的主要任务是：熟悉各大DL网络模型，主要以分类和检测为主；看论文；熟悉病理数据等。我们有一个2人组的小分队，我这个月的主要工作集中在学习目标检测的经典算法以及基于tensorflow或者keras跑一些经典的案例，主要有R-CNN，SPP-Net，Fast-RCNN，Faster-RCNN，YOLO等；另一名成员主要学习分类相关的经典网络模型，主要是google-net一系列的模型（inception-v1，inception-v2，inception-v3，resnet 等）。我们分别要整理出一份关于检测和分类的详细报告，然后不断完善、互相交流讨论、分享，发挥小分队的优势。
本文主要结合实践对R-CNN 进行整理。其中 R 表示的是候选区（Region Proposal）， CNN 部分使用经典的 AlexNet 网络模型，先来看个关于 AlexNet 的图示感受一下。
2012 年，Hinton 的学生 Alex Krizhevsky 提出了深度卷积神经网络模型AlexNet，它可以算是 LeNet 的一种更深更宽的版本，几个关键点整理如下：
1）首次在CNN 中成功应用了 ReLU（非线性激活函数）、Dropout （防止过拟合）和 LRN 等Trick。
2）使用了GPU 进行运算加速，作者开源了他们在 GPU 上训练卷积神经网络 CUDA 代码。
3）包含了6 亿 3000 万个连接，6000 万个参数和65 万个神经元，拥有 5 个卷积层，其中 3 个卷积层后面连接了最大池化层(MaxPooling Layer)，最后还有 3 个全连接层。
4）AlexNet 以显著的优势赢得了竞争激烈的ILSVRC 2012 比赛，top-5 的错误率降低至了16.4%，相比第二名的成绩 26.2%错误率有了巨大的提升。
5）AlexNet 可以说是神经网络在低谷期后的第一次发声，确立了深度学习（深度卷积网络）在计算机视觉的统治地位，同时也推动了深度学习在语音识别、自然语言处理、强化学习等领域的拓展。
本文不过多的介绍理论知识，待后续有更深刻的理解，再针对性的整理相关的理论篇。
AlexNet 模型结构
再来看看R-CNN的图示：
RossB.Girshick（RBG）大神使用 Region Proposal + CNN 代替传统目标检测使用的滑动窗口 + 手工设计特征，设计了R-CNN框架。与 R-CNN 相关的几个 关键词 如下所示：
1）Region proposals：
2）Selective Search：选择性搜索，找出可能含有物体的框，这些框之间是可以互相重叠互相包含，这样就可以避免暴力枚举出所有框。
3）Warp and Crop：修正区域大小，以适合CNN的输入（AlexNet 的 input layer 的图像大小为 224 x 224 x3，3表示 RGB 3个颜色通道）。
4）Supervisedpre-training：有监督预训练也称迁移学习
5）IOU：交并比IOU=(A∩B)/(A∪B)
6）NMS：非极大值抑制（参考：http://blog.csdn.net/H2008066215019910120/article/details/25917609）
7）DPM：使用判别训练的部件模型进行目标检测
关于选择性搜索算法，如下图所示，有这么多相关的算法，“Selective Search”只是其中一种。
R-CNN的缺点：
1）要求输入固定大小的图片，需要对原始图片进行crop（裁剪）或者wrap（缩放），在一定程度上导致图片信息的丢失和变形，限制了识别精度。
2）重复计算：R-CNN虽然不再是穷举，但依然有两千个左右的候选框，这些候选框都需要进行CNN操作，计算量依然很大，其中有不少其实是重复计算；
3）SVM模型：线性模型，在标注数据不缺的时候显然不是最好的选择；
4）训练测试分为多步：区域提名、特征提取、分类、回归都是断开的训练的过程，中间数据还需要单独保存；训练的空间和时间代价很高。
本文主要介绍R-CNN的实践，不过多介绍理论知识。具体的理论知识网上有很多相关的教程，我也贴出一些参考网址放在 Reference 部分。
Reference
github 源码（1）：https://github.com/edwardbi/DeepLearningModels/tree/master/RCNN
github源码（2）本文使用的代码：http://download.csdn.net/download/houchaoqun_xmu/10138539
selectivesearch：https://github.com/AlpacaDB/selectivesearch
本文实践部分主要参考：https://www.cnblogs.com/edwardbi/p/5647522.html
我看AlexNet【简书】：http://www.jianshu.com/p/58168fec534d
【卷积神经网络-进化史】从LeNet到AlexNet：http://blog.csdn.net/cyh_24/article/details/51440344
深度学习（十八）基于R-CNN的物体检测：http://blog.csdn.net/hjimce/article/details/50187029
深度学习与计算机视觉，看这一篇就够了：https://www.leiphone.com/news/201605/zZqsZiVpcBBPqcGG.html#rd
tflearn官网：http://tflearn.org/installation/
Ubuntu 常用软件安装：http://blog.csdn.net/houchaoqun_xmu/article/details/72461592
准备工作
1）本文实践环境：python3 + tensorflow-1.2.0 + tflearn
tensorflow.__version__ = 1.2.0
tflearn：先装好 tensorflow，然后直接使用 pip install tflearn 命令安装 tflearn2）下载 github 源码：
git clone https://github.com/Houchaoqun/MachineLearning_DeepLearning.git3）下载数据集并放在该 project 的根目录下（参考下文的代码结构）：
- 链接: https://pan.baidu.com/s/1hrSKz56
- 密码: gput
4）安装 python 提供的 SelectiveSearch 插件，输入如下命令：
pip install selectivesearch
flower 数据集
1）2-flowers：2种花朵的类别（不同时期，不同姿态，不同颜色），每个类别下有30张图片
该目录结构如下所示：
hcq@hcq-home:~/document/deepLearning/github/rcnn-tflearn/2flowers$ tree -L 2
.
└── jpg
├── 0
└── 12）17-flowers：17种花朵的类别，每个类别下有80张图片
该目录结构如下所示：
hcq@hcq-home:~/document/deepLearning/github/rcnn-tflearn/17flowers$ tree -L 2
.
└── jpg
├── 0
├── 1
├── 10
├── 11
├── 12
├── 13
├── 14
├── 15
├── 16
├── 2
├── 3
├── 4
├── 5
├── 6
├── 7
├── 8
├── 9
└── files.txttflearn-rcnn 代码结构
hcq@hcq-home:~/document/deepLearning/github/rcnn-tflearn$ tree -L 2
.
├── 17flowers
│ └── jpg
├── 2flowers
│ └── jpg
├── fine_tune_RCNN.py
├── output
│ └── alexnet_oxflowers17
├── preprocessing_RCNN.py
├── RCNN.md
├── RCNN_output.py
├── refine_backup.txt
├── refine_list.txt
├── svm_train
│ ├── 1.txt
│ └── 2.txt
├── testimg7.jpg
├── train_alexnet.py
└── train_list.txttflearn 构建 AlexNet 的核心代码
def create_alexnet(num_classes):
# Building 'AlexNet'
network = input_data(shape=[None, 224, 224, 3])
network = conv_2d(network, 96, 11, strides=4, activation='relu')
network = max_pool_2d(network, 3, strides=2)
network = local_response_normalization(network)
network = conv_2d(network, 256, 5, activation='relu')
network = max_pool_2d(network, 3, strides=2)
network = local_response_normalization(network)
network = conv_2d(network, 384, 3, activation='relu')
network = conv_2d(network, 384, 3, activation='relu')
network = conv_2d(network, 256, 3, activation='relu')
network = max_pool_2d(network, 3, strides=2)
network = local_response_normalization(network)
network = fully_connected(network, 4096, activation='tanh')
network = dropout(network, 0.5)
network = fully_connected(network, 4096, activation='tanh')
network = dropout(network, 0.5)
network = fully_connected(network, num_classes, activation='softmax')
network = regression(network, optimizer='momentum',
loss='categorical_crossentropy',
learning_rate=0.001)
return network实践步骤
1）做好准备工作；
2）执行如下命令，基于 flower17 数据集（标签只有类别）做预训练，得到分类器
python train_alexnet.pyepoch = 200，训练效果如下：
hcq@hcq-home:~/document/deepLearning/github/rcnn-tflearn$ python train_alexnet.py
2017-11-27 20:07:56.694124: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-27 20:07:56.694144: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-27 20:07:56.694162: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-11-27 20:07:56.694166: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-27 20:07:56.694170: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-11-27 20:07:57.024534: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-11-27 20:07:57.025026: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.582
pciBusID 0000:01:00.0
Total memory: 10.90GiB
Free memory: 10.29GiB
2017-11-27 20:07:57.025053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0
2017-11-27 20:07:57.025063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0: Y
2017-11-27 20:07:57.025081: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0)
2017-11-27 20:08:05.490085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0)
loading previous parameters
2017-11-27 20:08:05.569326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0)
2017-11-27 20:08:06.435150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0)
---------------------------------
Run id: alexnet_oxflowers17
Log directory: output/
---------------------------------
Training samples: 1224
Validation samples: 136
--
Training Step: 200 | total loss: 3.37247 | time: 9.586ss
| Momentum | epoch: 004 | loss: 3.37247 - acc: 0.0630 | val_loss: 2.98178 - val_acc: 0.0515 -- iter: 0160/1224
--
Training Step: 400 | total loss: 3.32241 | time: 17.929s
| Momentum | epoch: 009 | loss: 3.32241 - acc: 0.0576 | val_loss: 2.77432 - val_acc: 0.0588 -- iter: 0320/1224
--
Training Step: 600 | total loss: 3.00749 | time: 26.467s
| Momentum | epoch: 014 | loss: 3.00749 - acc: 0.0976 | val_loss: 2.46418 - val_acc: 0.1544 -- iter: 0480/1224
-- ... ... ...
Training Step: 7600 | total loss: 0.00295 | time: 66.383s
| Momentum | epoch: 193 | loss: 0.00295 - acc: 0.9999 | val_loss: 0.78864 - val_acc: 0.8088 -- iter: 1088/1224
--
Training Step: 7800 | total loss: 0.00561 | time: 76.434s
| Momentum | epoch: 198 | loss: 0.00561 - acc: 0.9998 | val_loss: 0.73804 - val_acc: 0.8088 -- iter: 1224/1224
--
Training Step: 7878 | total loss: 0.00451 | time: 75.050s
| Momentum | epoch: 200 | loss: 0.00451 - acc: 0.9998 -- iter: 1224/1224由上述提示信息可见，经过200轮训练后，模型的准确率已经从 0.0630 达到 0.9998。而此时还未涉及到 R-CNN 的核心部分 - 候选框的提取。本文使用的是 Selective Search 算法，直接使用 pip install selectivesearch 进行下载即可，有兴趣的读者也可以自己编写python脚本。
3）执行如下命令，基于 flower2 数据集（标签既有类别，又有位置信息 [x,y,w,h]）做 fine-tuning
python fine_tune_RCNN.pyfine_tune_Alexnet 函数如下所示：
def fine_tune_Alexnet(network, X, Y):
# Training
model = tflearn.DNN(network, checkpoint_path='rcnn_model_alexnet',
max_checkpoints=1, tensorboard_verbose=2, tensorboard_dir='output_RCNN')
if os.path.isfile('fine_tune_model_save.model'):
print("Loading the fine tuned model")
model.load('fine_tune_model_save.model')
# saver.restore(sess, './alexnet-cnn.model')
elif os.path.isfile('model_save.model.meta'):
print("Loading the alexnet")
# saver = tf.train.Saver()
try:
model.load('model_save.model')
print("successful loaded [model_save.model]...")
except Exception as e:
print(e)
pass
# saver.restore(model.session,'./model_save.model')
else:
print("No file to load, error")
return False
model.fit(X, Y, n_epoch=10, validation_set=0.1, shuffle=True,
show_metric=True, batch_size=32, snapshot_step=200,
snapshot_epoch=False, run_id='alexnet_rcnnflowers2') # epoch = 1000
# Save the model
model.save('fine_tune_model_save.model')fine-tuning 结束后，提示如下所示（会生成 fine_tune_model_save.model 的相关文件）：
successful loaded [model_save.model]...
---------------------------------
Run id: alexnet_rcnnflowers2
Log directory: output_RCNN/
---------------------------------
Training samples: 2034
Validation samples: 226
--
Training Step: 8000 | total loss: 0.18893 | time: 112.817s
| Momentum | epoch: 002 | loss: 0.18893 - acc: 0.9366 | val_loss: 0.20319 - val_acc: 0.9336 -- iter: 1856/2034
--
Training Step: 8200 | total loss: 0.18021 | time: 4.906s8s
| Momentum | epoch: 006 | loss: 0.18021 - acc: 0.9413 | val_loss: 0.19566 - val_acc: 0.9336 -- iter: 0064/2034
--
Training Step: 8400 | total loss: 0.18054 | time: 20.381ss
| Momentum | epoch: 009 | loss: 0.18054 - acc: 0.9347 | val_loss: 0.16186 - val_acc: 0.9336 -- iter: 0320/2034
--
Training Step: 8518 | total loss: 0.11383 | time: 122.717s
| Momentum | epoch: 010 | loss: 0.11383 - acc: 0.9564 -- iter: 2034/2034
由上述提示信息可知，训练10轮后，模型的准确率为0.9564。适当增加epoch的值，再进行训练可以得到更优的效果。本人准备将epoch的值设置成1000再训练一次模型，感受一下效果。
至此，模型就已经训练好了。此处有点不同的是，该网络模型不是直接使用CNN后接softmax做分类，而是换成SVM。因为SVM适用于小样本训练，这里这么做可以提高准确率。详细的解释可以参考博文：http://blog.csdn.net/hjimce/article/details/50187029
训练SVM的代码如下所示：
# Construct cascade svms
def train_svms(train_file_folder, model):
listings = os.listdir(train_file_folder)
svms = []
for train_file in listings:
if "pkl" in train_file:
continue
X, Y = generate_single_svm_train(train_file_folder+train_file)
train_features = []
for i in X:
feats = model.predict([i])
train_features.append(feats[0])
print("feature dimension")
print(np.shape(train_features))
clf = svm.LinearSVC()
print("fit svm")
clf.fit(train_features, Y)
svms.append(clf)
return svms
# Load training images
def generate_single_svm_train(one_class_train_file):
trainfile = one_class_train_file
savepath = one_class_train_file.replace('txt', 'pkl')
images = []
Y = []
if os.path.isfile(savepath):
print("restoring svm dataset " + savepath)
images, Y = prep.load_from_pkl(savepath)
else:
print("loading svm dataset " + savepath)
images, Y = prep.load_train_proposals(trainfile, 2, threshold=0.3, svm=True, save=True, save_path=savepath)
return images, Ygenerate_single_svm_train 函数分别根据 ./svm_train 目录下的 ”1.txt“和”2.txt“生成对应的”1.pkl“和”2.pkl“，都是很大的文件（> 1GB）。
测试结果
执行如下命令，作用包括：
- 训练 SVM
- 使用”testimg7.jpg“测试模型
python RCNN_output.py此时，你可能会遇到如下问题：
Traceback (most recent call last):
File "RCNN_output.py", line 151, in <module>
pred = i.predict(f)
File "/home/hcq/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/base.py", line 324, in predict
scores = self.decision_function(X)
File "/home/hcq/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/base.py", line 300, in decision_function
X = check_array(X, accept_sparse='csr')
File "/home/hcq/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py", line 441, in check_array
"if it contains a single sample.".format(array))
ValueError: Expected 2D array, got 1D array instead:
array=[-0.62386084 0.64800894 0.54052156 ..., 0.29537463 0.49037218
0.40998983].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.你需要在 RCNN_output.py 脚本里加一条语句 f = f.reshape(1, -1) ，如下所示：
if __name__ == '__main__':
train_file_folder = 'svm_train/'
img_path = 'testimg7.jpg'
imgs, verts = image_proposal(img_path)
net = create_alexnet(3)
model = tflearn.DNN(net)
model.load('fine_tune_model_save.model')
svms = train_svms(train_file_folder, model)
print("Done fitting svms")
features = model.predict(imgs) #
print("predict image:")
print(np.shape(features)) # (107, 4096)
results = []
results_label = []
count = 0
for f in features:
f = f.reshape(1, -1) # add by hcq 20171128
for i in svms:
pred = i.predict(f)
print(pred)
if pred[0] != 0:
results.append(verts[count])
results_label.append(pred[0])
count += 1
print("result:")
print(results)
print("result label:")
print(results_label)
img = skimage.io.imread(img_path)
fig, ax = plt.subplots(ncols=1, nrows=1, figsize=(6, 6))
ax.imshow(img)
for x, y, w, h in results:
rect = mpatches.Rectangle(
(x, y), w, h, fill=False, edgecolor='red', linewidth=1)
ax.add_patch(rect)
plt.show()调试成功后，效果如下所示（此处使用的是经过200轮fine-tuning后的模型）：
执行如下命令，加入非极大值抑制（nms）后的效果如下所示：
python RCNN_output_nms.py
实践效果解释（后续补上）
实践过程中遇到的问题：
1）使用python2（建议使用python3）：
restoring svm dataset svm_train/2.pkl
Traceback (most recent call last):
File "RCNN_output.py", line 139, in <module>
svms = train_svms(train_file_folder, model)
File "RCNN_output.py", line 119, in train_svms
X, Y = generate_single_svm_train(train_file_folder + train_file)
File "RCNN_output.py", line 80, in generate_single_svm_train
images, Y = prep.load_from_pkl(savepath)
File "/home/hcq/document/deepLearning/github/rcnn-tflearn/preprocessing_RCNN.py", line 133, in load_from_pkl
X, Y = pickle.load(open(dataset_file, 'rb'))
File "/home/hcq/anaconda2/lib/python2.7/pickle.py", line 1384, in load
return Unpickler(file).load()
File "/home/hcq/anaconda2/lib/python2.7/pickle.py", line 864, in load
dispatch[key](self)
File "/home/hcq/anaconda2/lib/python2.7/pickle.py", line 892, in load_proto
raise ValueError, "unsupported pickle protocol: %d" % proto
ValueError: unsupported pickle protocol: 3解决方案：
http://blog.csdn.net/u013828589/article/details/72848192
https://stackoverflow.com/questions/25843698/valueerror-unsupported-pickle-protocol-3-python2-pickle-can-not-load-the-file
2）sklearn 工具包版本造成的问题：ValueError: Expected 2D array, got 1D array instead
Traceback (most recent call last):
File "RCNN_output.py", line 149, in <module>
pred = i.predict(f)
File "/home/hcq/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/base.py", line 324, in predict
scores = self.decision_function(X)
File "/home/hcq/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/base.py", line 300, in decision_function
X = check_array(X, accept_sparse='csr')
File "/home/hcq/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py", line 441, in check_array
"if it contains a single sample.".format(array))
ValueError: Expected 2D array, got 1D array instead:
array=[-0.62386084 0.64800894 0.54052156 ..., 0.29537463 0.49037218
0.40998983].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.解决方案：
http://blog.csdn.net/llx1026/article/details/77940880