Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun Microsoft Research kahe, v-xiangz, v-shren, jiansun @microsoft.com
Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun Microsoft Research kahe, v-xiangz, v-shren, jiansun @microsoft.com
原文全文
Page 1
<a id="S0001"></a> Source: p.1 S0001
Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun Microsoft Research kahe, v-xiangz, v-shren, jiansun @microsoft.com { } Abstract 20 Deeper neural networks are more difficult to train.
<a id="S0002"></a> Source: p.1 S0002
We present a residual learning framework to ease the training 10 of networks that are substantially deeper than those used previously.
<a id="S0003"></a> Source: p.1 S0003
We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, in- 0 0 1 2 iter. 3 (1e4) 4 5 6 stead of learning unreferenced functions.
<a id="S0004"></a> Source: p.1 S0004
We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
<a id="S0005"></a> Source: p.1 S0005
On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8 × deeper than VGG nets [41] but still having lower complexity.
<a id="S0006"></a> Source: p.1 S0006
An ensemble of these residual nets achieves 3.57% error on the ImageNet test set.
<a id="S0007"></a> Source: p.1 S0007
This result won the 1st place on the ILSVRC 2015 classification task.
<a id="S0008"></a> Source: p.1 S0008
We also present analysis on CIFAR-10 with 100 and 1000 layers.
<a id="S0009"></a> Source: p.1 S0009
The depth of representations is of central importance for many visual recognition tasks.
<a id="S0010"></a> Source: p.1 S0010
Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset.
<a id="S0011"></a> Source: p.1 S0011
Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. 1.
<a id="S0012"></a> Source: p.1 S0012
Introduction Deep convolutional neural networks [22, 21] have led to a series of breakthroughs for image classification [21, 50, 40].
<a id="S0013"></a> Source: p.1 S0013
Deep networks naturally integrate low/mid/highlevel features [50] and classifiers in an end-to-end multilayer fashion, and the “levels” of features can be enriched by the number of stacked layers (depth).
<a id="S0014"></a> Source: p.1 S0014
Recent evidence [41, 44] reveals that network depth is of crucial importance, and the leading results [41, 44, 13, 16] on the challenging ImageNet dataset [36] all exploit “very deep” [41] models, with a depth of sixteen [41] to thirty [16].
<a id="S0015"></a> Source: p.1 S0015
Many other nontrivial visual recognition tasks [8, 12, 7, 32, 27] have also 1http://image-net.org/challenges/LSVRC/2015/ and http://mscoco.org/dataset/#detections-challenge2015. )%( rorre gniniart 20 10 00 1 2 3 4 5 6 iter. (1e4) )%( rorre tset 56-layer 20-layer 56-layer 20-layer Figure 1.
<a id="S0016"></a> Source: p.1 S0016
Training error (left) and test error (right) on CIFAR-10 with 20-layer and 56-layer “plain” networks.
<a id="S0017"></a> Source: p.1 S0017
The deeper network has higher training error, and thus test error.
<a id="S0018"></a> Source: p.1 S0018
Similar phenomena on ImageNet is presented in Fig. 4. greatly benefited from very deep models.
<a id="S0019"></a> Source: p.1 S0019
Driven by the significance of depth, a question arises: Is learning better networks as easy as stacking more layers?
<a id="S0020"></a> Source: p.1 S0020
An obstacle to answering this question was the notorious problem of vanishing/exploding gradients [1, 9], which hamper convergence from the beginning.
<a id="S0021"></a> Source: p.1 S0021
This problem, however, has been largely addressed by normalized initialization [23, 9, 37, 13] and intermediate normalization layers [16], which enable networks with tens of layers to start converging for stochastic gradient descent (SGD) with backpropagation [22].
<a id="S0022"></a> Source: p.1 S0022
When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly.
<a id="S0023"></a> Source: p.1 S0023
Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error, as reported in [11, 42] and thoroughly verified by our experiments.
<a id="S0024"></a> Source: p.1 S0024
The degradation (of training accuracy) indicates that not all systems are similarly easy to optimize.
<a id="S0025"></a> Source: p.1 S0025
Let us consider a shallower architecture and its deeper counterpart that adds more layers onto it.
<a id="S0026"></a> Source: p.1 S0026
There exists a solution by construction to the deeper model: the added layers are identity mapping, and the other layers are copied from the learned shallower model.
<a id="S0027"></a> Source: p.1 S0027
The existence of this constructed solution indicates that a deeper model should produce no higher training error than its shallower counterpart.
<a id="S0028"></a> Source: p.1 S0028
But experiments show that our current solvers on hand are unable to find solutions that 1 5102 ceD 01 ]VC.sc[ 1v58330.2151:viXra
Page 2
<a id="S0029"></a> Source: p.2 S0029
ImageNet test set, and won the 1st place in the ILSVRC x 2015 classification competition.
<a id="S0030"></a> Source: p.2 S0030
The extremely deep repweight layer resentations also have excellent generalization performance F (x) relu x on other recognition tasks, and lead us to further win the weight layer identity 1st places on: ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation in ILSVRC & (x)(cid:1)+(cid:1)x F relu COCO 2015 competitions.
<a id="S0031"></a> Source: p.2 S0031
This strong evidence shows that Figure 2.
<a id="S0032"></a> Source: p.2 S0032
Residual learning: a building block. the residual learning principle is generic, and we expect that it is applicable in other vision and non-vision problems. are comparably good or better than the constructed solution (or unable to do so in feasible time). 2.
<a id="S0033"></a> Source: p.2 S0033
Related Work In this paper, we address the degradation problem by introducing a deep residual learning framework.
<a id="S0034"></a> Source: p.2 S0034
In image recognition, VLAD stead of hoping each few stacked layers directly fit a [18] is a representation that encodes by the residual vectors desired underlying mapping, we explicitly let these lay- with respect to a dictionary, and Fisher Vector [30] can be ers fit a residual mapping.
<a id="S0035"></a> Source: p.2 S0035
Formally, denoting the desired formulated as a probabilistic version [18] of VLAD.
<a id="S0036"></a> Source: p.2 S0036
Both underlying mapping as (x), we let the stacked nonlinear of them are powerful shallow representations for image relayers fit another mappin H g of (x) := (x) x.
<a id="S0037"></a> Source: p.2 S0037
The orig- trieval and classification [4, 48].
<a id="S0038"></a> Source: p.2 S0038
For vector quantization, inal mapping is recast into ( F x)+x. W H e hyp − othesize that it encoding residual vectors [17] is shown to be more effecis easier to optimize the re F sidual mapping than to optimize tive than encoding original vectors. the original, unreferenced mapping.
<a id="S0039"></a> Source: p.2 S0039
To the extreme, if an In low-level vision and computer graphics, for solvidentity mapping were optimal, it would be easier to push ing Partial Differential Equations (PDEs), the widely used the residual to zero than to fit an identity mapping by a stack Multigrid method [3] reformulates the system as subprobof nonlinear layers. lems at multiple scales, where each subproblem is responsible for the residual solution between a coarser and a finer The formulation of (x) + x can be realized by feedfor- F scale.
<a id="S0040"></a> Source: p.2 S0040
An alternative to Multigrid is hierarchical basis preward neural networks with “shortcut connections” (Fig. 2). conditioning [45, 46], which relies on variables that repre- Shortcut connections [2, 34, 49] are those skipping one or sent residual vectors between two scales.
<a id="S0041"></a> Source: p.2 S0041
In our case, the shortcut connections simply [3, 45, 46] that these solvers converge much faster than stanperform identity mapping, and their outputs are added to dard solvers that are unaware of the residual nature of the the outputs of the stacked layers (Fig. 2).
<a id="S0042"></a> Source: p.2 S0042
These methods suggest that a good reformulation cut connections add neither extra parameter nor computaor preconditioning can simplify the optimization. tional complexity.
<a id="S0043"></a> Source: p.2 S0043
The entire network can still be trained end-to-end by SGD with backpropagation, and can be eas- Shortcut Connections.
<a id="S0044"></a> Source: p.2 S0044
Practices and theories that lead to ily implemented using common libraries (e.g., Caffe [19]) shortcut connections [2, 34, 49] have been studied for a long without modifying the solvers. time.
<a id="S0045"></a> Source: p.2 S0045
An early practice of training multi-layer perceptrons We present comprehensive experiments on ImageNet (MLPs) is to add a linear layer connected from the network [36] to show the degradation problem and evaluate our input to the output [34, 49].
<a id="S0046"></a> Source: p.2 S0046
We show that: 1) Our extremely deep residual nets diate layers are directly connected to auxiliary classifiers are easy to optimize, but the counterpart “plain” nets (that for addressing vanishing/exploding gradients.
<a id="S0047"></a> Source: p.2 S0047
The papers simply stack layers) exhibit higher training error when the of [39, 38, 31, 47] propose methods for centering layer redepth increases; 2) Our deep residual nets can easily enjoy sponses, gradients, and propagated errors, implemented by accuracy gains from greatly increased depth, producing re- shortcut connections.
<a id="S0048"></a> Source: p.2 S0048
In [44], an “inception” layer is comsults substantially better than previous networks. posed of a shortcut branch and a few deeper branches.
<a id="S0049"></a> Source: p.2 S0049
Similar phenomena are also shown on the CIFAR-10 set Concurrent with our work, “highway networks” [42, 43] [20], suggesting that the optimization difficulties and the present shortcut connections with gating functions [15]. effects of our method are not just akin to a particular dataset.
<a id="S0050"></a> Source: p.2 S0050
These gates are data-dependent and have parameters, in We present successfully trained models on this dataset with contrast to our identity shortcuts that are parameter-free. over 100 layers, and explore models with over 1000 layers.
<a id="S0051"></a> Source: p.2 S0051
When a gated shortcut is “closed” (approaching zero), the On the ImageNet classification dataset [36], we obtain layers in highway networks represent non-residual funcexcellent results by extremely deep residual nets.
<a id="S0052"></a> Source: p.2 S0052
On the contrary, our formulation always learns layer residual net is the deepest network ever presented on residual functions; our identity shortcuts are never closed, ImageNet, while still having lower complexity than VGG and all information is always passed through, with addinets [41].
<a id="S0053"></a> Source: p.2 S0053
Our ensemble has 3.57% top-5 error on the tional residual functions to be learned.
Page 3
<a id="S0054"></a> Source: p.3 S0054
way networks have not demonstrated accuracy gains with ReLU [29] and the biases are omitted for simplifying noextremely increased depth (e.g., over 100 layers). tations.
<a id="S0055"></a> Source: p.3 S0055
The operation + x is performed by a shortcut F connection and element-wise addition.
<a id="S0056"></a> Source: p.3 S0056
Deep Residual Learning ond nonlinearity after the addition (i.e., σ(y), see Fig. 2).
<a id="S0057"></a> Source: p.3 S0057
The shortcut connections in Eqn.(1) introduce neither ex- 3.1.
<a id="S0058"></a> Source: p.3 S0058
Residual Learning tra parameter nor computation complexity.
<a id="S0059"></a> Source: p.3 S0059
This is not only Let us consider (x) as an underlying mapping to be attractive in practice but also important in our comparisons H fit by a few stacked layers (not necessarily the entire net), between plain and residual networks.
<a id="S0060"></a> Source: p.3 S0060
We can fairly comwith x denoting the inputs to the first of these layers.
<a id="S0061"></a> Source: p.3 S0061
If one pare plain/residual networks that simultaneously have the hypothesizes that multiple nonlinear layers can asymptoti- same number of parameters, depth, width, and computacally approximate complicated functions2, then it is equiv- tional cost (except for the negligible element-wise addition). alent to hypothesize that they can asymptotically approxi- The dimensions of x and must be equal in Eqn.(1). F mate the residual functions, i.e., (x) x (assuming that If this is not the case (e.g., when changing the input/output H − the input and output are of the same dimensions).
<a id="S0062"></a> Source: p.3 S0062
So channels), we can perform a linear projection W s by the rather than expect stacked layers to approximate (x), we shortcut connections to match the dimensions: H explicitly let these layers approximate a residual function (x) := (x) x.
<a id="S0063"></a> Source: p.3 S0063
The original function thus becomes y = (x, W i ) + W s x. (2) F H − F { } (x)+x.
<a id="S0064"></a> Source: p.3 S0064
Although both forms should be able to asymptot- F We can also use a square matrix W in Eqn.(1).
<a id="S0065"></a> Source: p.3 S0065
But we will ically approximate the desired functions (as hypothesized), s show by experiments that the identity mapping is sufficient the ease of learning might be different. for addressing the degradation problem and is economical, This reformulation is motivated by the counterintuitive and thus W is only used when matching dimensions. phenomena about the degradation problem (Fig. 1, left).
<a id="S0066"></a> Source: p.3 S0066
As s The form of the residual function is flexible.
<a id="S0067"></a> Source: p.3 S0067
Experwe discussed in the introduction, if the added layers can F iments in this paper involve a function that has two or be constructed as identity mappings, a deeper model should F three layers (Fig. 5), while more layers are possible.
<a id="S0068"></a> Source: p.3 S0068
But if have training error no greater than its shallower counterhas only a single layer, Eqn.(1) is similar to a linear layer: part.
<a id="S0069"></a> Source: p.3 S0069
The degradation problem suggests that the solvers F y = W x + x, for which we have not observed advantages. might have difficulties in approximating identity mappings 1 We also note that although the above notations are about by multiple nonlinear layers.
<a id="S0070"></a> Source: p.3 S0070
With the residual learning refully-connected layers for simplicity, they are applicable to formulation, if identity mappings are optimal, the solvers convolutional layers.
<a id="S0071"></a> Source: p.3 S0071
The function (x, W ) can repremay simply drive the weights of the multiple nonlinear lay- i F { } sent multiple convolutional layers.
<a id="S0072"></a> Source: p.3 S0072
The element-wise addiers toward zero to approach identity mappings. tion is performed on two feature maps, channel by channel.
<a id="S0073"></a> Source: p.3 S0073
In real cases, it is unlikely that identity mappings are optimal, but our reformulation may help to precondition the 3.3.
<a id="S0074"></a> Source: p.3 S0074
If the optimal function is closer to an identity mapping than to a zero mapping, it should be easier for the We have tested various plain/residual nets, and have obsolver to find the perturbations with reference to an identity served consistent phenomena.
<a id="S0075"></a> Source: p.3 S0075
To provide instances for dismapping, than to learn the function as a new one.
<a id="S0076"></a> Source: p.3 S0076
We show cussion, we describe two models for ImageNet as follows. by experiments (Fig. 7) that the learned residual functions in Plain Network.
<a id="S0077"></a> Source: p.3 S0077
Our plain baselines (Fig. 3, middle) are general have small responses, suggesting that identity mapmainly inspired by the philosophy of VGG nets [41] (Fig. 3, pings provide reasonable preconditioning. left).
<a id="S0078"></a> Source: p.3 S0078
The convolutional layers mostly have 3 3 filters and × 3.2.
<a id="S0079"></a> Source: p.3 S0079
Identity Mapping by Shortcuts follow two simple design rules: (i) for the same output feature map size, the layers have the same number of fil- We adopt residual learning to every few stacked layers. ters; and (ii) if the feature map size is halved, the num- A building block is shown in Fig. 2.
<a id="S0080"></a> Source: p.3 S0080
Formally, in this paper ber of filters is doubled so as to preserve the time comwe consider a building block defined as: plexity per layer.
<a id="S0081"></a> Source: p.3 S0081
We perform downsampling directly by convolutional layers that have a stride of 2.
<a id="S0082"></a> Source: p.3 S0082
The network y = (x, W ) + x. (1) F { i } ends with a global average pooling layer and a 1000-way fully-connected layer with softmax.
<a id="S0083"></a> Source: p.3 S0083
The total number of Here x and y are the input and output vectors of the layweighted layers is 34 in Fig. 3 (middle). ers considered.
<a id="S0084"></a> Source: p.3 S0084
The function (x, W ) represents the F { i } It is worth noticing that our model has fewer filters and residual mapping to be learned.
<a id="S0085"></a> Source: p.3 S0085
For the example in Fig. 2 lower complexity than VGG nets [41] (Fig. 3, left).
<a id="S0086"></a> Source: p.3 S0086
Our 34that has two layers, = W σ(W x) in which σ denotes 2 1 F layer baseline has 3.6 billion FLOPs (multiply-adds), which 2This hypothesis, however, is still an open question.
<a id="S0087"></a> Source: p.3 S0087
See [28]. is only 18% of VGG-19 (19.6 billion FLOPs). 3
Page 4
<a id="S0088"></a> Source: p.4 S0088
VGG-19 34-layer plain 34-layer residual Residual Network.
<a id="S0089"></a> Source: p.4 S0089
Based on the above plain network, we insert shortcut connections (Fig. 3, right) which turn the image image image s o iz u e t : p 2 u 2 t 4 3x3 conv, 64 network into its counterpart residual version.
<a id="S0090"></a> Source: p.4 S0090
The identity shortcuts (Eqn.(1)) can be directly used when the input and 3x3 conv, 64 output are of the same dimensions (solid line shortcuts in pool, /2 output Fig. 3).
<a id="S0091"></a> Source: p.4 S0091
When the dimensions increase (dotted line shortcuts size: 112 3x3 conv, 128 in Fig. 3), we consider two options: (A) The shortcut still 3x3 conv, 128 7x7 conv, 64, /2 7x7 conv, 64, /2 performs identity mapping, with extra zero entries padded pool, /2 pool, /2 pool, /2 output for increasing dimensions.
<a id="S0092"></a> Source: p.4 S0092
This option introduces no extra size: 56 3x3 conv, 256 3x3 conv, 64 3x3 conv, 64 parameter; (B) The projection shortcut in Eqn.(2) is used to 3x3 conv, 256 3x3 conv, 64 3x3 conv, 64 match dimensions (done by 1 1 convolutions).
<a id="S0093"></a> Source: p.4 S0093
For both × 3x3 conv, 256 3x3 conv, 64 3x3 conv, 64 options, when the shortcuts go across feature maps of two 3x3 conv, 256 3x3 conv, 64 3x3 conv, 64 sizes, they are performed with a stride of 2. 3x3 conv, 64 3x3 conv, 64 3.4.
<a id="S0094"></a> Source: p.4 S0094
Implementation 3x3 conv, 64 3x3 conv, 64 pool, /2 3x3 conv, 128, /2 3x3 conv, 128, /2 Our implementation for ImageNet follows the practice output size: 28 3x3 conv, 512 3x3 conv, 128 3x3 conv, 128 in [21, 41].
<a id="S0095"></a> Source: p.4 S0095
The image is resized with its shorter side ran- 3x3 conv, 512 3x3 conv, 128 3x3 conv, 128 domly sampled in [256, 480] for scale augmentation [41]. 3x3 conv, 512 3x3 conv, 128 3x3 conv, 128 A 224 224 crop is randomly sampled from an image or its × horizontal flip, with the per-pixel mean subtracted [21].
<a id="S0096"></a> Source: p.4 S0096
The 3x3 conv, 512 3x3 conv, 128 3x3 conv, 128 standard color augmentation in [21] is used.
<a id="S0097"></a> Source: p.4 S0097
We adopt batch 3x3 conv, 128 3x3 conv, 128 normalization (BN) [16] right after each convolution and 3x3 conv, 128 3x3 conv, 128 before activation, following [16].
<a id="S0098"></a> Source: p.4 S0098
We initialize the weights 3x3 conv, 128 3x3 conv, 128 as in [13] and train all plain/residual nets from scratch.
<a id="S0099"></a> Source: p.4 S0099
We s o iz u e t : p 1 u 4 t pool, /2 3x3 conv, 256, /2 3x3 conv, 256, /2 use SGD with a mini-batch size of 256.
<a id="S0100"></a> Source: p.4 S0100
The learning rate 3x3 conv, 512 3x3 conv, 256 3x3 conv, 256 starts from 0.1 and is divided by 10 when the error plateaus, 3x3 conv, 512 3x3 conv, 256 3x3 conv, 256 and the models are trained for up to 60 104 iterations.
<a id="S0101"></a> Source: p.4 S0101
We 3x3 conv, 512 3x3 conv, 256 3x3 conv, 256 × use a weight decay of 0.0001 and a momentum of 0.9.
<a id="S0102"></a> Source: p.4 S0102
We 3x3 conv, 512 3x3 conv, 256 3x3 conv, 256 do not use dropout [14], following the practice in [16]. 3x3 conv, 256 3x3 conv, 256 In testing, for comparison studies we adopt the standard 3x3 conv, 256 3x3 conv, 256 10-crop testing [21].
<a id="S0103"></a> Source: p.4 S0103
For best results, we adopt the fully- 3x3 conv, 256 3x3 conv, 256 convolutional form as in [41, 13], and average the scores 3x3 conv, 256 3x3 conv, 256 at multiple scales (images are resized such that the shorter 3x3 conv, 256 3x3 conv, 256 side is in 224, 256, 384, 480, 640 ). { } 3x3 conv, 256 3x3 conv, 256 4.
<a id="S0104"></a> Source: p.4 S0104
Experiments 3x3 conv, 256 3x3 conv, 256 o si u z t e p : u 7 t pool, /2 3x3 conv, 512, /2 3x3 conv, 512, /2 4.1.
<a id="S0105"></a> Source: p.4 S0105
ImageNet Classification 3x3 conv, 512 3x3 conv, 512 We evaluate our method on the ImageNet 2012 classifi- 3x3 conv, 512 3x3 conv, 512 cation dataset [36] that consists of 1000 classes.
<a id="S0106"></a> Source: p.4 S0106
The models 3x3 conv, 512 3x3 conv, 512 are trained on the 1.28 million training images, and evalu- 3x3 conv, 512 3x3 conv, 512 ated on the 50k validation images.
<a id="S0107"></a> Source: p.4 S0107
We also obtain a final 3x3 conv, 512 3x3 conv, 512 result on the 100k test images, reported by the test server. o si u z t e p : u 1 t fc 4096 avg pool avg pool We evaluate both top-1 and top-5 error rates. fc 4096 fc 1000 fc 1000 fc 1000 Plain Networks.
<a id="S0108"></a> Source: p.4 S0108
We first evaluate 18-layer and 34-layer plain nets.
<a id="S0109"></a> Source: p.4 S0109
The 34-layer plain net is in Fig. 3 (middle).
<a id="S0110"></a> Source: p.4 S0110
Example network architectures for ImageNet.
<a id="S0111"></a> Source: p.4 S0111
Left: the 18-layer plain net is of a similar form.
<a id="S0112"></a> Source: p.4 S0112
See Table 1 for de- VGG-19 model [41] (19.6 billion FLOPs) as a reference.
<a id="S0113"></a> Source: p.4 S0113
Mid- tailed architectures. dle: a plain network with 34 parameter layers (3.6 billion FLOPs).
<a id="S0114"></a> Source: p.4 S0114
The results in Table 2 show that the deeper 34-layer plain Right: a residual network with 34 parameter layers (3.6 billion net has higher validation error than the shallower 18-layer FLOPs).
<a id="S0115"></a> Source: p.4 S0115
The dotted shortcuts increase dimensions.
<a id="S0116"></a> Source: p.4 S0116
To reveal the reasons, in Fig. 4 (left) we commore details and other variants. pare their training/validation errors during the training procedure.
<a id="S0117"></a> Source: p.4 S0117
We have observed the degradation problem - the 4
Page 5
<a id="S0118"></a> Source: p.5 S0118
layer name output size 18-layer 34-layer 50-layer 101-layer 152-layer conv1 112 112 7 7, 64, stride 2 × × 3 3 max pool, stride 2 conv2 x 56 × 56 (cid:20) 3 3 × 3 3 , , 6 6 4 4 (cid:21) × 2 (cid:20) 3 3 × 3 3 , , 6 6 4 4 (cid:21) × 3 × 1 3 × × 1 3 , , 6 6 4 4 × 3 1 3 × × 1 3 , , 6 6 4 4 × 3 1 3 × × 1 3 , , 6 6 4 4 × 3 × × 1 1, 256 1 1, 256 1 1, 256 conv3 x 28 × 28 (cid:20) 3 3 × 3 3 , , 1 1 2 2 8 8 (cid:21) × 2 (cid:20) 3 3 × 3 3 , , 1 1 2 2 8 8 (cid:21) × 4 1 3 × × × 1 3 , , 1 1 2 2 8 8 × 4 1 3 × × × 1 3 , , 1 1 2 2 8 8 × 4 1 3 × × × 1 3 , , 1 1 2 2 8 8 × 8 × × 1 1, 512 1 1, 512 1 1, 512 conv4 x 14 × 14 (cid:20) 3 3 × 3 3 , , 2 2 5 5 6 6 (cid:21) × 2 (cid:20) 3 3 × 3 3 , , 2 2 5 5 6 6 (cid:21) × 6 1 3 × × × 1 3 , , 2 2 5 5 6 6 × 6 1 3 × × × 1 3 , , 2 2 5 5 6 6 × 23 1 3 × × × 1 3 , , 2 2 5 5 6 6 × 36 × × 1 1, 1024 1 1, 1024 1 1, 1024 conv5 x 7 × 7 (cid:20) 3 3 × 3 3 , , 5 5 1 1 2 2 (cid:21) × 2 (cid:20) 3 3 × 3 3 , , 5 5 1 1 2 2 (cid:21) × 3 1 3 × × × 1 3 , , 5 5 1 1 2 2 × 3 1 3 × × × 1 3 , , 5 5 1 1 2 2 × 3 1 3 × × × 1 3 , , 5 5 1 1 2 2 × 3 × × 1 1, 2048 1 1, 2048 1 1, 2048 × × × 1 1 average pool, 1000-d fc, softmax × FLOPs 1.8 109 3.6 109 3.8 109 7.6 109 11.3 109 × × × × × Table 1.
<a id="S0119"></a> Source: p.5 S0119
Building blocks are shown in brackets (see also Fig. 5), with the numbers of blocks stacked.
<a id="S0120"></a> Source: p.5 S0120
Downsampling is performed by conv3 1, conv4 1, and conv5 1 with a stride of 2. 60 50 40 30 20 0 10 20 30 40 50 iter. (1e4) )%( rorre 60 50 40 30 plain-18 plain-34 20 0 10 20 30 40 50 iter. (1e4) )%( rorre 34-layer 18-layer 18-layer ResNet-18 ResNet-34 34-layer Figure 4.
<a id="S0121"></a> Source: p.5 S0121
Thin curves denote training error, and bold curves denote validation error of the center crops.
<a id="S0122"></a> Source: p.5 S0122
Left: plain networks of 18 and 34 layers.
<a id="S0123"></a> Source: p.5 S0123
In this plot, the residual networks have no extra parameter compared to their plain counterparts. plain ResNet reducing of the training error3.
<a id="S0124"></a> Source: p.5 S0124
The reason for such opti- 18 layers 27.94 27.88 mization difficulties will be studied in the future. 34 layers 28.54 25.03 Residual Networks.
<a id="S0125"></a> Source: p.5 S0125
Next we evaluate 18-layer and 34- Table 2.
<a id="S0126"></a> Source: p.5 S0126
Top-1 error (%, 10-crop testing) on ImageNet validation. layer residual nets (ResNets).
<a id="S0127"></a> Source: p.5 S0127
The baseline architectures Here the ResNets have no extra parameter compared to their plain are the same as the above plain nets, expect that a shortcut counterparts.
<a id="S0128"></a> Source: p.5 S0128
Fig. 4 shows the training procedures. connection is added to each pair of 3 3 filters as in Fig. 3 × (right).
<a id="S0129"></a> Source: p.5 S0129
In the first comparison (Table 2 and Fig. 4 right), we use identity mapping for all shortcuts and zero-padding 34-layer plain net has higher training error throughout the for increasing dimensions (option A).
<a id="S0130"></a> Source: p.5 S0130
So they have no extra whole training procedure, even though the solution space parameter compared to the plain counterparts. of the 18-layer plain network is a subspace of that of the We have three major observations from Table 2 and 34-layer one.
<a id="S0131"></a> Source: p.5 S0131
First, the situation is reversed with residual learn- We argue that this optimization difficulty is unlikely to ing – the 34-layer ResNet is better than the 18-layer ResNet be caused by vanishing gradients.
<a id="S0132"></a> Source: p.5 S0132
More importantly, the 34-layer ResNet exhibits trained with BN [16], which ensures forward propagated considerably lower training error and is generalizable to the signals to have non-zero variances.
<a id="S0133"></a> Source: p.5 S0133
We also verify that the validation data.
<a id="S0134"></a> Source: p.5 S0134
This indicates that the degradation problem backward propagated gradients exhibit healthy norms with is well addressed in this setting and we manage to obtain BN.
<a id="S0135"></a> Source: p.5 S0135
So neither forward nor backward signals vanish.
<a id="S0136"></a> Source: p.5 S0136
In accuracy gains from increased depth. fact, the 34-layer plain net is still able to achieve compet- Second, compared to its plain counterpart, the 34-layer itive accuracy (Table 3), suggesting that the solver works 3We have experimented with more training iterations (3×) and still obto some extent.
<a id="S0137"></a> Source: p.5 S0137
We conjecture that the deep plain nets may served the degradation problem, suggesting that this problem cannot be have exponentially low convergence rates, which impact the feasibly addressed by simply using more iterations. 5
Page 6
<a id="S0138"></a> Source: p.6 S0138
VGG-16 [41] 28.07 9.33 3x3, 64 1x1, 64 relu GoogLeNet [44] - 9.15 relu 3x3, 64 PReLU-net [13] 24.27 7.38 3x3, 64 relu 1x1, 256 plain-34 28.54 10.02 relu relu ResNet-34 A 25.03 7.76 ResNet-34 B 24.52 7.46 Figure 5. A deeper residual function F for ImageNet.
<a id="S0139"></a> Source: p.6 S0139
Left: a ResNet-34 C 24.19 7.40 building block (on 56×56 feature maps) as in Fig. 3 for ResNet- ResNet-50 22.85 6.71 34.
<a id="S0140"></a> Source: p.6 S0140
Right: a “bottleneck” building block for ResNet-50/101/152.
<a id="S0141"></a> Source: p.6 S0141
ResNet-101 21.75 6.05 ResNet-152 21.43 5.71 parameter-free, identity shortcuts help with training.
<a id="S0142"></a> Source: p.6 S0142
Error rates (%, 10-crop testing) on ImageNet validation. we investigate projection shortcuts (Eqn.(2)).
<a id="S0143"></a> Source: p.6 S0143
In Table 3 we VGG-16 is based on our test.
<a id="S0144"></a> Source: p.6 S0144
ResNet-50/101/152 are of option B compare three options: (A) zero-padding shortcuts are used that only uses projections for increasing dimensions. for increasing dimensions, and all shortcuts are parameterfree (the same as Table 2 and Fig. 4 right); (B) projecmethod top-1 err. top-5 err.
<a id="S0145"></a> Source: p.6 S0145
VGG [41] (ILSVRC’14) - 8.43† tion shortcuts are used for increasing dimensions, and other shortcuts are identity; and (C) all shortcuts are projections.
<a id="S0146"></a> Source: p.6 S0146
GoogLeNet [44] (ILSVRC’14) - 7.89 Table 3 shows that all three options are considerably bet- VGG [41] (v5) 24.4 7.1 ter than the plain counterpart. B is slightly better than A.
<a id="S0147"></a> Source: p.6 S0147
We PReLU-net [13] 21.59 5.71 argue that this is because the zero-padded dimensions in A BN-inception [16] 21.99 5.81 indeed have no residual learning. C is marginally better than ResNet-34 B 21.84 5.71 B, and we attribute this to the extra parameters introduced ResNet-34 C 21.53 5.60 by many (thirteen) projection shortcuts.
<a id="S0148"></a> Source: p.6 S0148
But the small dif- ResNet-50 20.74 5.25 ferences among A/B/C indicate that projection shortcuts are ResNet-101 19.87 4.60 not essential for addressing the degradation problem.
<a id="S0149"></a> Source: p.6 S0149
So we ResNet-152 19.38 4.49 do not use option C in the rest of this paper, to reduce mem- Table 4.
<a id="S0150"></a> Source: p.6 S0150
Error rates (%) of single-model results on the ImageNet ory/time complexity and model sizes.
<a id="S0151"></a> Source: p.6 S0151
Identity shortcuts are validation set (except † reported on the test set). particularly important for not increasing the complexity of the bottleneck architectures that are introduced below. method top-5 err. (test) VGG [41] (ILSVRC’14) 7.32 Deeper Bottleneck Architectures.
<a id="S0152"></a> Source: p.6 S0152
Next we describe our GoogLeNet [44] (ILSVRC’14) 6.66 deeper nets for ImageNet.
<a id="S0153"></a> Source: p.6 S0153
Because of concerns on the train- VGG [41] (v5) 6.8 ing time that we can afford, we modify the building block PReLU-net [13] 4.94 as a bottleneck design4.
<a id="S0154"></a> Source: p.6 S0154
For each residual function , we F BN-inception [16] 4.82 use a stack of 3 layers instead of 2 (Fig. 5).
<a id="S0155"></a> Source: p.6 S0155
The three layers ResNet (ILSVRC’15) 3.57 are 1 1, 3 3, and 1 1 convolutions, where the 1 1 layers × × × × are responsible for reducing and then increasing (restoring) Table 5.
<a id="S0156"></a> Source: p.6 S0156
The top-5 error is on the dimensions, leaving the 3 3 layer a bottleneck with smaller test set of ImageNet and reported by the test server. × input/output dimensions.
<a id="S0157"></a> Source: p.6 S0157
Fig. 5 shows an example, where both designs have similar time complexity.
<a id="S0158"></a> Source: p.6 S0158
ResNet reduces the top-1 error by 3.5% (Table 2), resulting The parameter-free identity shortcuts are particularly imfrom the successfully reduced training error (Fig. 4 right vs. portant for the bottleneck architectures.
<a id="S0159"></a> Source: p.6 S0159
This comparison verifies the effectiveness of residual cut in Fig. 5 (right) is replaced with projection, one can learning on extremely deep systems. show that the time complexity and model size are doubled, as the shortcut is connected to the two high-dimensional Last, we also note that the 18-layer plain/residual nets ends.
<a id="S0160"></a> Source: p.6 S0160
So identity shortcuts lead to more efficient models are comparably accurate (Table 2), but the 18-layer ResNet for the bottleneck designs. converges faster (Fig. 4 right vs. left).
<a id="S0161"></a> Source: p.6 S0161
When the net is “not 50-layer ResNet: We replace each 2-layer block in the overly deep” (18 layers here), the current SGD solver is still able to find good solutions to the plain net.
<a id="S0162"></a> Source: p.6 S0162
In this case, the 4Deeper non-bottleneck ResNets (e.g., Fig. 5 left) also gain accuracy ResNet eases the optimization by providing faster converfrom increased depth (as shown on CIFAR-10), but are not as economical gence at the early stage. as the bottleneck ResNets.
<a id="S0163"></a> Source: p.6 S0163
So the usage of bottleneck designs is mainly due to practical considerations.
<a id="S0164"></a> Source: p.6 S0164
We further note that the degradation problem Identity vs.
<a id="S0165"></a> Source: p.6 S0165
We have shown that of plain nets is also witnessed for the bottleneck designs. 6
Page 7
<a id="S0166"></a> Source: p.7 S0166
34-layer net with this 3-layer bottleneck block, resulting in method error (%) a 50-layer ResNet (Table 1).
<a id="S0167"></a> Source: p.7 S0167
We use option B for increasing Maxout [10] 9.38 dimensions.
<a id="S0168"></a> Source: p.7 S0168
NIN [25] 8.81 101-layer and 152-layer ResNets: We construct 101- DSN [24] 8.22 layer and 152-layer ResNets by using more 3-layer blocks # layers # params (Table 1).
<a id="S0169"></a> Source: p.7 S0169
Remarkably, although the depth is significantly FitNet [35] 19 2.5M 8.39 increased, the 152-layer ResNet (11.3 billion FLOPs) still Highway [42, 43] 19 2.3M 7.54 (7.72±0.16) has lower complexity than VGG-16/19 nets (15.3/19.6 bil- Highway [42, 43] 32 1.25M 8.80 lion FLOPs).
<a id="S0170"></a> Source: p.7 S0170
ResNet 20 0.27M 8.75 The 50/101/152-layer ResNets are more accurate than ResNet 32 0.46M 7.51 the 34-layer ones by considerable margins (Table 3 and 4).
<a id="S0171"></a> Source: p.7 S0171
ResNet 44 0.66M 7.17 We do not observe the degradation problem and thus en- ResNet 56 0.85M 6.97 joy significant accuracy gains from considerably increased ResNet 110 1.7M 6.43 (6.61±0.16) depth.
<a id="S0172"></a> Source: p.7 S0172
The benefits of depth are witnessed for all evaluation ResNet 1202 19.4M 7.93 metrics (Table 3 and 4).
<a id="S0173"></a> Source: p.7 S0173
Classification error on the CIFAR-10 test set.
<a id="S0174"></a> Source: p.7 S0174
All meth- Comparisons with State-of-the-art Methods.
<a id="S0175"></a> Source: p.7 S0175
In Table 4 ods are with data augmentation.
<a id="S0176"></a> Source: p.7 S0176
For ResNet-110, we run it 5 times we compare with the previous best single-model results. and show “best (mean±std)” as in [43].
<a id="S0177"></a> Source: p.7 S0177
Our baseline 34-layer ResNets have achieved very competitive accuracy.
<a id="S0178"></a> Source: p.7 S0178
Our 152-layer ResNet has a single-model so our residual models have exactly the same depth, width, top-5 validation error of 4.49%.
<a id="S0179"></a> Source: p.7 S0179
This single-model result and number of parameters as the plain counterparts. outperforms all previous ensemble results (Table 5).
<a id="S0180"></a> Source: p.7 S0180
We We use a weight decay of 0.0001 and momentum of 0.9, combine six models of different depth to form an ensemble and adopt the weight initialization in [13] and BN [16] but (only with two 152-layer ones at the time of submitting). with no dropout.
<a id="S0181"></a> Source: p.7 S0181
These models are trained with a mini- This leads to 3.57% top-5 error on the test set (Table 5). batch size of 128 on two GPUs.
<a id="S0182"></a> Source: p.7 S0182
We start with a learning This entry won the 1st place in ILSVRC 2015. rate of 0.1, divide it by 10 at 32k and 48k iterations, and 4.2.
<a id="S0183"></a> Source: p.7 S0183
CIFAR-10 and Analysis terminate training at 64k iterations, which is determined on a 45k/5k train/val split.
<a id="S0184"></a> Source: p.7 S0184
We follow the simple data augmen- We conducted more studies on the CIFAR-10 dataset tation in [24] for training: 4 pixels are padded on each side, [20], which consists of 50k training images and 10k testand a 32 32 crop is randomly sampled from the padded ing images in 10 classes.
<a id="S0185"></a> Source: p.7 S0185
We present experiments trained × image or its horizontal flip.
<a id="S0186"></a> Source: p.7 S0186
For testing, we only evaluate on the training set and evaluated on the test set.
<a id="S0187"></a> Source: p.7 S0187
Our focus the single view of the original 32 32 image. is on the behaviors of extremely deep networks, but not on × We compare n = 3, 5, 7, 9 , leading to 20, 32, 44, and pushing the state-of-the-art results, so we intentionally use { } 56-layer networks.
<a id="S0188"></a> Source: p.7 S0188
Fig. 6 (left) shows the behaviors of the simple architectures as follows. plain nets.
<a id="S0189"></a> Source: p.7 S0189
The deep plain nets suffer from increased depth, The plain/residual architectures follow the form in Fig. 3 and exhibit higher training error when going deeper.
<a id="S0190"></a> Source: p.7 S0190
The network inputs are 32 32 images, with phenomenon is similar to that on ImageNet (Fig. 4, left) and × the per-pixel mean subtracted.
<a id="S0191"></a> Source: p.7 S0191
The first layer is 3 3 convoon MNIST (see [42]), suggesting that such an optimization × lutions.
<a id="S0192"></a> Source: p.7 S0192
Then we use a stack of 6n layers with 3 3 convodifficulty is a fundamental problem. × lutions on the feature maps of sizes 32, 16, 8 respectively, Fig. 6 (middle) shows the behaviors of ResNets.
<a id="S0193"></a> Source: p.7 S0193
Also { } with 2n layers for each feature map size.
<a id="S0194"></a> Source: p.7 S0194
The numbers of similar to the ImageNet cases (Fig. 4, right), our ResNets filters are 16, 32, 64 respectively.
<a id="S0195"></a> Source: p.7 S0195
The subsampling is permanage to overcome the optimization difficulty and demon- { } formed by convolutions with a stride of 2.
<a id="S0196"></a> Source: p.7 S0196
The network ends strate accuracy gains when the depth increases. with a global average pooling, a 10-way fully-connected We further explore n = 18 that leads to a 110-layer layer, and softmax.
<a id="S0197"></a> Source: p.7 S0197
There are totally 6n+2 stacked weighted ResNet.
<a id="S0198"></a> Source: p.7 S0198
In this case, we find that the initial learning rate layers.
<a id="S0199"></a> Source: p.7 S0199
The following table summarizes the architecture: of 0.1 is slightly too large to start converging5.
<a id="S0200"></a> Source: p.7 S0200
So we use 0.01 to warm up the training until the training error is below output map size 32×32 16×16 8×8 80% (about 400 iterations), and then go back to 0.1 and con- # layers 1+2n 2n 2n tinue training.
<a id="S0201"></a> Source: p.7 S0201
The rest of the learning schedule is as done # filters 16 32 64 previously.
<a id="S0202"></a> Source: p.7 S0202
This 110-layer network converges well (Fig. 6, middle).
<a id="S0203"></a> Source: p.7 S0203
It has fewer parameters than other deep and thin When shortcut connections are used, they are connected to the pairs of 3 3 layers (totally 3n shortcuts).
<a id="S0204"></a> Source: p.7 S0204
On this 5With an initial learning rate of 0.1, it starts converging (<90% error) × dataset we use identity shortcuts in all cases (i.e., option A), after several epochs, but still reaches similar accuracy. 7
Page 8
<a id="S0205"></a> Source: p.8 S0205
20 10 5 00 1 2 3 4 5 6 iter. (1e4) )%( rorre 20 10 plain-20 5 plain-32 plain-44 plain-56 00 1 2 3 4 5 6 iter. (1e4) )%( rorre 20 ResNet-20 ResNet-32 ResNet-44 ResNet-56 56-layer ResNet-110 20-layer 20-layer 10 110-layer 5 1 0 4 5 6 iter. (1e4) )%( rorre residual-110 residual-1202 Figure 6.
<a id="S0206"></a> Source: p.8 S0206
Dashed lines denote training error, and bold lines denote testing error.
<a id="S0207"></a> Source: p.8 S0207
The error of plain-110 is higher than 60% and not displayed.
<a id="S0208"></a> Source: p.8 S0208
Right: ResNets with 110 and 1202 layers. 3 2 1 0 20 40 60 80 100 layer index (sorted by magnitude) dts 3 2 1 0 20 40 60 80 100 layer index (original) plain-20 plain-56 ResNet-20 ResNet-56 ResNet-110 dts plain-20 training data 07+12 07++12 plain-56 ResNet-20 test data VOC 07 test VOC 12 test ResNet-56 ResNet-110 VGG-16 73.2 70.4 ResNet-101 76.4 73.8 Table 7.
<a id="S0209"></a> Source: p.8 S0209
Object detection mAP (%) on the PASCAL VOC 2007/2012 test sets using baseline Faster R-CNN.
<a id="S0210"></a> Source: p.8 S0210
See also Table 10 and 11 for better results. metric mAP@.5 mAP@[.5, .95] VGG-16 41.5 21.2 ResNet-101 48.4 27.2 Figure 7.
<a id="S0211"></a> Source: p.8 S0211
Standard deviations (std) of layer responses on CIFAR- 10.
<a id="S0212"></a> Source: p.8 S0212
The responses are the outputs of each 3×3 layer, after BN and Table 8.
<a id="S0213"></a> Source: p.8 S0213
Object detection mAP (%) on the COCO validation set before nonlinearity.
<a id="S0214"></a> Source: p.8 S0214
Top: the layers are shown in their original using baseline Faster R-CNN.
<a id="S0215"></a> Source: p.8 S0215
See also Table 9 for better results. order.
<a id="S0216"></a> Source: p.8 S0216
Bottom: the responses are ranked in descending order. have similar training error.
<a id="S0217"></a> Source: p.8 S0217
We argue that this is because of networks such as FitNet [35] and Highway [42] (Table 6), overfitting.
<a id="S0218"></a> Source: p.8 S0218
The 1202-layer network may be unnecessarily yet is among the state-of-the-art results (6.43%, Table 6). large (19.4M) for this small dataset.
<a id="S0219"></a> Source: p.8 S0219
Strong regularization such as maxout [10] or dropout [14] is applied to obtain the Analysis of Layer Responses.
<a id="S0220"></a> Source: p.8 S0220
Fig. 7 shows the standard best results ([10, 25, 24, 35]) on this dataset.
<a id="S0221"></a> Source: p.8 S0221
In this paper, deviations (std) of the layer responses.
<a id="S0222"></a> Source: p.8 S0222
The responses are we use no maxout/dropout and just simply impose regularthe outputs of each 3 3 layer, after BN and before other ization via deep and thin architectures by design, without × nonlinearity (ReLU/addition).
<a id="S0223"></a> Source: p.8 S0223
For ResNets, this analydistracting from the focus on the difficulties of optimizasis reveals the response strength of the residual functions. tion.
<a id="S0224"></a> Source: p.8 S0224
But combining with stronger regularization may im- Fig. 7 shows that ResNets have generally smaller responses prove results, which we will study in the future. than their plain counterparts.
<a id="S0225"></a> Source: p.8 S0225
These results support our basic motivation (Sec.3.1) that the residual functions might 4.3.
<a id="S0226"></a> Source: p.8 S0226
Object Detection on PASCAL and MS COCO be generally closer to zero than the non-residual functions.
<a id="S0227"></a> Source: p.8 S0227
Our method has good generalization performance on We also notice that the deeper ResNet has smaller magniother recognition tasks.
<a id="S0228"></a> Source: p.8 S0228
Table 7 and 8 show the object detudes of responses, as evidenced by the comparisons among tection baseline results on PASCAL VOC 2007 and 2012 ResNet-20, 56, and 110 in Fig. 7.
<a id="S0229"></a> Source: p.8 S0229
We adopt Faster R-CNN [32] as the delayers, an individual layer of ResNets tends to modify the tection method.
<a id="S0230"></a> Source: p.8 S0230
Here we are interested in the improvements signal less. of replacing VGG-16 [41] with ResNet-101.
<a id="S0231"></a> Source: p.8 S0231
The detection Exploring Over 1000 layers.
<a id="S0232"></a> Source: p.8 S0232
We explore an aggressively implementation (see appendix) of using both models is the deep model of over 1000 layers.
<a id="S0233"></a> Source: p.8 S0233
We set n = 200 that same, so the gains can only be attributed to better networks. leads to a 1202-layer network, which is trained as described Most remarkably, on the challenging COCO dataset we obabove.
<a id="S0234"></a> Source: p.8 S0234
Our method shows no optimization difficulty, and tain a 6.0% increase in COCO’s standard metric (mAP@[.5, this 103-layer network is able to achieve training error .95]), which is a 28% relative improvement.
<a id="S0235"></a> Source: p.8 S0235
Its test error is still fairly good solely due to the learned representations. (7.93%, Table 6).
<a id="S0236"></a> Source: p.8 S0236
Based on deep residual nets, we won the 1st places in But there are still open problems on such aggressively several tracks in ILSVRC & COCO 2015 competitions: Imdeep models.
<a id="S0237"></a> Source: p.8 S0237
The testing result of this 1202-layer network ageNet detection, ImageNet localization, COCO detection, is worse than that of our 110-layer network, although both and COCO segmentation.
Page 9
<a id="S0238"></a> Source: p.9 S0238
On the number of linear regions of deep neural networks.
<a id="S0239"></a> Source: p.9 S0239
Rectified linear units improve restricted cies with gradient descent is difficult.
<a id="S0240"></a> Source: p.9 S0240
IEEE Transactions on Neural boltzmann machines.
<a id="S0241"></a> Source: p.9 S0241
Fisher kernels on visual vocabularies for [2] C. M.
<a id="S0242"></a> Source: p.9 S0242
Neural networks for pattern recognition.
<a id="S0243"></a> Source: p.9 S0243
In CVPR, 2007. university press, 1995. [31] T.
<a id="S0244"></a> Source: p.9 S0244
Siam, linear transformations in perceptrons.
<a id="S0245"></a> Source: p.9 S0245
The devil real-time object detection with region proposal networks.
<a id="S0246"></a> Source: p.9 S0246
In NIPS, is in the details: an evaluation of recent feature encoding methods. 2015.
<a id="S0247"></a> Source: p.9 S0247
Zisnetworks on convolutional feature maps. arXiv:1504.06066, 2015. serman.
<a id="S0248"></a> Source: p.9 S0248
The Pascal Visual Object Classes (VOC) Challenge.
<a id="S0249"></a> Source: p.9 S0249
Pattern recognition and neural networks.
<a id="S0250"></a> Source: p.9 S0250
Cambridge pages 303–338, 2010. university press, 1996. [6] S.
<a id="S0251"></a> Source: p.9 S0251
Object detection via a multi-region & [35] A.
<a id="S0252"></a> Source: p.9 S0252
Gatta, and semantic segmentation-aware cnn model.
<a id="S0253"></a> Source: p.9 S0253
Imagenet archies for accurate object detection and semantic segmentation.
<a id="S0254"></a> Source: p.9 S0254
In large scale visual recognition challenge. arXiv:1409.0575, 2014.
<a id="S0255"></a> Source: p.9 S0255
Understanding the difficulty of training the nonlinear dynamics of learning in deep linear neural networks. deep feedforward neural networks.
<a id="S0256"></a> Source: p.9 S0256
In AISTATS, 2010. arXiv:1312.6120, 2013. [10] I. J.
<a id="S0257"></a> Source: p.9 S0257
Accelerated gradient descent by factor-centering Y.
<a id="S0258"></a> Source: p.9 S0258
Maxout networks. arXiv:1302.4389, 2013. decomposition.
<a id="S0259"></a> Source: p.9 S0259
Convolutional neural networks at constrained time [39] N. N.
<a id="S0260"></a> Source: p.9 S0260
Centering neural network gradient factors.
<a id="S0261"></a> Source: p.9 S0261
Neural Networks: Tricks of the Trade, pages 207–226.
<a id="S0262"></a> Source: p.9 S0262
Spatial pyramid pooling in deep 1998. convolutional networks for visual recognition.
<a id="S0263"></a> Source: p.9 S0263
Overfeat: Integrated recognition, localization and detection Surpassing human-level performance on imagenet classification.
<a id="S0264"></a> Source: p.9 S0264
Very deep convolutional networks [14] G. E.
<a id="S0265"></a> Source: p.9 S0265
Sutskever, and for large-scale image recognition.
<a id="S0266"></a> Source: p.9 S0266
Improving neural networks by preventing co- [42] R. K.
<a id="S0267"></a> Source: p.9 S0267
Highway networks. adaptation of feature detectors. arXiv:1207.0580, 2012. arXiv:1505.00387, 2015. [15] S.
<a id="S0268"></a> Source: p.9 S0268
Training very deep computation, 9(8):1735–1780, 1997. networks. 1507.06228, 2015. [16] S.
<a id="S0269"></a> Source: p.9 S0269
Batch normalization: Accelerating deep [44] C.
<a id="S0270"></a> Source: p.9 S0270
Ernetwork training by reducing internal covariate shift.
<a id="S0271"></a> Source: p.9 S0271
Fast surface interpolation using hierarchical basis func- [18] H.
<a id="S0272"></a> Source: p.9 S0272
Aggregating local image descriptors into compact codes. [46] R.
<a id="S0273"></a> Source: p.9 S0273
Locally adapted hierarchical basis preconditioning.
<a id="S0274"></a> Source: p.9 S0274
Caffe: Convolutional architecture for tic gradient towards second-order methods–backpropagation learnfast feature embedding. arXiv:1408.5093, 2014. ing with transformations in nonlinearities.
<a id="S0275"></a> Source: p.9 S0275
Learning multiple layers of features from tiny im- Processing, 2013. ages.
<a id="S0276"></a> Source: p.9 S0276
VLFeat: An open and portable library [21] A.
<a id="S0277"></a> Source: p.9 S0277
Imagenet classification of computer vision algorithms, 2008. with deep convolutional neural networks.
<a id="S0278"></a> Source: p.9 S0278
Modern applied statistics with s-plus. [22] Y.
<a id="S0279"></a> Source: p.9 S0279
Backpropagation applied to hand- [50] M. D.
<a id="S0280"></a> Source: p.9 S0280
Visualizing and understanding convoluwritten zip code recognition.
<a id="S0281"></a> Source: p.9 S0281
Neural computation, 1989. tional neural networks.
<a id="S0282"></a> Source: p.9 S0282
In Neural Networks: Tricks of the Trade, pages 9–50.
<a id="S0283"></a> Source: p.9 S0283
Deeplysupervised nets. arXiv:1409.5185, 2014. [25] M.
<a id="S0284"></a> Source: p.9 S0284
Network in network. arXiv:1312.4400, 2013. [26] T.-Y.
<a id="S0285"></a> Source: p.9 S0285
Microsoft COCO: Common objects in context.
<a id="S0286"></a> Source: p.9 S0286
Fully convolutional networks for semantic segmentation.
Page 10
<a id="S0287"></a> Source: p.10 S0287
Object Detection Baselines 8 images (i.e., 1 per GPU) and the Fast R-CNN step has a mini-batch size of 16 images.
<a id="S0288"></a> Source: p.10 S0288
The RPN step and Fast R- In this section we introduce our detection method based CNN step are both trained for 240k iterations with a learnon the baseline Faster R-CNN [32] system.
<a id="S0289"></a> Source: p.10 S0289
The models are ing rate of 0.001 and then for 80k iterations with 0.0001. initialized by the ImageNet classification models, and then Table 8 shows the results on the MS COCO validation fine-tuned on the object detection data.
<a id="S0290"></a> Source: p.10 S0290
ResNet-101 has a 6% increase of mAP@[.5, .95] over mented with ResNet-50/101 at the time of the ILSVRC & VGG-16, which is a 28% relative improvement, solely con- COCO 2015 detection competitions. tributed by the features learned by the better network.
<a id="S0291"></a> Source: p.10 S0291
Re- Unlike VGG-16 used in [32], our ResNet has no hidden markably, the mAP@[.5, .95]’s absolute increase (6.0%) is fc layers.
<a id="S0292"></a> Source: p.10 S0292
We adopt the idea of “Networks on Conv feanearly as big as mAP@.5’s (6.9%).
<a id="S0293"></a> Source: p.10 S0293
This suggests that a ture maps” (NoC) [33] to address this issue.
<a id="S0294"></a> Source: p.10 S0294
We compute deeper network can improve both recognition and localizathe full-image shared conv feature maps using those laytion. ers whose strides on the image are no greater than 16 pixels (i.e., conv1, conv2 x, conv3 x, and conv4 x, totally 91 conv B.
<a id="S0295"></a> Source: p.10 S0295
Object Detection Improvements layers in ResNet-101; Table 1).
<a id="S0296"></a> Source: p.10 S0296
We consider these layers as analogous to the 13 conv layers in VGG-16, and by doing For completeness, we report the improvements made for so, both ResNet and VGG-16 have conv feature maps of the the competitions.
<a id="S0297"></a> Source: p.10 S0297
These improvements are based on deep same total stride (16 pixels).
<a id="S0298"></a> Source: p.10 S0298
These layers are shared by a features and thus should benefit from residual learning. region proposal network (RPN, generating 300 proposals) [32] and a Fast R-CNN detection network [7].
<a id="S0299"></a> Source: p.10 S0299
RoI pool- MS COCO ing [7] is performed before conv5 1.
<a id="S0300"></a> Source: p.10 S0300
Our box refinement partially follows the itfeature, all layers of conv5 x and up are adopted for each erative localization in [6].
<a id="S0301"></a> Source: p.10 S0301
In Faster R-CNN, the final output region, playing the roles of VGG-16’s fc layers.
<a id="S0302"></a> Source: p.10 S0302
The final is a regressed box that is different from its proposal box.
<a id="S0303"></a> Source: p.10 S0303
So classification layer is replaced by two sibling layers (classi- for inference, we pool a new feature from the regressed box fication and box regression [7]). and obtain a new classification score and a new regressed For the usage of BN layers, after pre-training, we com- box.
<a id="S0304"></a> Source: p.10 S0304
We combine these 300 new predictions with the origpute the BN statistics (means and variances) for each layer inal 300 predictions.
<a id="S0305"></a> Source: p.10 S0305
Non-maximum suppression (NMS) is on the ImageNet training set.
<a id="S0306"></a> Source: p.10 S0306
Then the BN layers are fixed applied on the union set of predicted boxes using an IoU during fine-tuning for object detection.
<a id="S0307"></a> Source: p.10 S0307
As such, the BN threshold of 0.3 [8], followed by box voting [6].
<a id="S0308"></a> Source: p.10 S0308
Box relayers become linear activations with constant offsets and finement improves mAP by about 2 points (Table 9). scales, and BN statistics are not updated by fine-tuning.
<a id="S0309"></a> Source: p.10 S0309
We combine global context in the Fast fix the BN layers mainly for reducing memory consumption R-CNN step.
<a id="S0310"></a> Source: p.10 S0310
Given the full-image conv feature map, we in Faster R-CNN training. pool a feature by global Spatial Pyramid Pooling [12] (with PASCAL VOC a “single-level” pyramid) which can be implemented as Following [7, 32], for the PASCAL VOC 2007 test set, “RoI” pooling using the entire image’s bounding box as the we use the 5k trainval images in VOC 2007 and 16k train- RoI.
<a id="S0311"></a> Source: p.10 S0311
This pooled feature is fed into the post-RoI layers to val images in VOC 2012 for training (“07+12”).
<a id="S0312"></a> Source: p.10 S0312
For the obtain a global context feature.
<a id="S0313"></a> Source: p.10 S0313
This global feature is con- PASCAL VOC 2012 test set, we use the 10k trainval+test catenated with the original per-region feature, followed by images in VOC 2007 and 16k trainval images in VOC 2012 the sibling classification and box regression layers.
<a id="S0314"></a> Source: p.10 S0314
The hyper-parameters for train- new structure is trained end-to-end.
<a id="S0315"></a> Source: p.10 S0315
Global context iming Faster R-CNN are the same as in [32].
<a id="S0316"></a> Source: p.10 S0316
Table 7 shows proves mAP@.5 by about 1 point (Table 9). the results.
<a id="S0317"></a> Source: p.10 S0317
ResNet-101 improves the mAP by >3% over Multi-scale testing.
<a id="S0318"></a> Source: p.10 S0318
In the above, all results are obtained by VGG-16.
<a id="S0319"></a> Source: p.10 S0319
This gain is solely because of the improved feasingle-scale training/testing as in [32], where the image’s tures learned by ResNet. shorter side is s = 600 pixels.
<a id="S0320"></a> Source: p.10 S0320
Multi-scale training/testing MS COCO has been developed in [12, 7] by selecting a scale from a The MS COCO dataset [26] involves 80 object cate- feature pyramid, and in [33] by using maxout layers.
<a id="S0321"></a> Source: p.10 S0321
We evaluate the PASCAL VOC metric (mAP @ our current implementation, we have performed multi-scale IoU = 0.5) and the standard COCO metric (mAP @ IoU = testing following [33]; we have not performed multi-scale .5:.05:.95).
<a id="S0322"></a> Source: p.10 S0322
We use the 80k images on the train set for train- training because of limited time.
<a id="S0323"></a> Source: p.10 S0323
In addition, we have pering and the 40k images on the val set for evaluation.
<a id="S0324"></a> Source: p.10 S0324
Our formed multi-scale testing only for the Fast R-CNN step detection system for COCO is similar to that for PASCAL (but not yet for the RPN step).
<a id="S0325"></a> Source: p.10 S0325
We train the COCO models with an 8-GPU imple- compute conv feature maps on an image pyramid, where the mentation, and thus the RPN step has a mini-batch size of image’s shorter sides are s 200, 400, 600, 800, 1000 . ∈ { } 10
Page 11
<a id="S0326"></a> Source: p.11 S0326
training data COCO train COCO trainval test data COCO val COCO test-dev mAP @.5 @[.5, .95] @.5 @[.5, .95] baseline Faster R-CNN (VGG-16) 41.5 21.2 baseline Faster R-CNN (ResNet-101) 48.4 27.2 +box refinement 49.9 29.9 +context 51.1 30.0 53.3 32.2 +multi-scale testing 53.8 32.5 55.7 34.9 ensemble 59.0 37.4 Table 9.
<a id="S0327"></a> Source: p.11 S0327
Object detection improvements on MS COCO using Faster R-CNN and ResNet-101. system net data mAP areo bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv baseline VGG-16 07+12 73.2 76.5 79.0 70.9 65.5 52.1 83.1 84.7 86.4 52.0 81.9 65.7 84.8 84.6 77.5 76.7 38.8 73.6 73.9 83.0 72.6 baseline ResNet-101 07+12 76.4 79.8 80.7 76.2 68.3 55.9 85.1 85.3 89.8 56.7 87.8 69.4 88.3 88.9 80.9 78.4 41.7 78.6 79.8 85.3 72.0 baseline+++ ResNet-101 COCO+07+12 85.6 90.0 89.6 87.8 80.8 76.1 89.9 89.9 89.6 75.5 90.0 80.7 89.6 90.3 89.1 88.7 65.4 88.1 85.6 89.0 86.8 Table 10.
<a id="S0328"></a> Source: p.11 S0328
Detection results on the PASCAL VOC 2007 test set.
<a id="S0329"></a> Source: p.11 S0329
The baseline is the Faster R-CNN system.
<a id="S0330"></a> Source: p.11 S0330
The system “baseline+++” include box refinement, context, and multi-scale testing in Table 9. system net data mAP areo bike bird boat bottle bus car cat chair cow table dog horse mbike person plant sheep sofa train tv baseline VGG-16 07++12 70.4 84.9 79.8 74.3 53.9 49.8 77.5 75.9 88.5 45.6 77.1 55.3 86.9 81.7 80.9 79.6 40.1 72.6 60.9 81.2 61.5 baseline ResNet-101 07++12 73.8 86.5 81.6 77.2 58.0 51.0 78.6 76.6 93.2 48.6 80.4 59.0 92.1 85.3 84.8 80.7 48.1 77.3 66.5 84.7 65.6 baseline+++ ResNet-101 COCO+07++12 83.8 92.1 88.4 84.8 75.9 71.4 86.3 87.8 94.2 66.8 89.4 69.2 93.9 91.9 90.9 89.6 67.9 88.2 76.8 90.3 80.0 Table 11.
<a id="S0331"></a> Source: p.11 S0331
Detection results on the PASCAL VOC 2012 test set (http://host.robots.ox.ac.uk:8080/leaderboard/ displaylb.php?challengeid=11&compid=4).
<a id="S0332"></a> Source: p.11 S0332
The baseline is the Faster R-CNN system.
<a id="S0333"></a> Source: p.11 S0333
The system “baseline+++” include box refinement, context, and multi-scale testing in Table 9.
<a id="S0334"></a> Source: p.11 S0334
We select two adjacent scales from the pyramid following val2 test [33].
<a id="S0335"></a> Source: p.11 S0335
RoI pooling and subsequent layers are performed on GoogLeNet [44] (ILSVRC’14) - 43.9 the feature maps of these two scales [33], which are merged our single model (ILSVRC’15) 60.5 58.8 by maxout as in [33].
<a id="S0336"></a> Source: p.11 S0336
Multi-scale testing improves the mAP our ensemble (ILSVRC’15) 63.6 62.1 by over 2 points (Table 9).
<a id="S0337"></a> Source: p.11 S0337
Our results (mAP, %) on the ImageNet detection dataset.
<a id="S0338"></a> Source: p.11 S0338
Next we use the 80k+40k trainval set Our detection system is Faster R-CNN [32] with the improvements for training and the 20k test-dev set for evaluation.
<a id="S0339"></a> Source: p.11 S0339
The test- in Table 9, using ResNet-101. dev set has no publicly available ground truth and the result is reported by the evaluation server.
<a id="S0340"></a> Source: p.11 S0340
Under this setting, the we achieve 85.6% mAP on PASCAL VOC 2007 (Table 10) results are an mAP@.5 of 55.7% and an mAP@[.5, .95] of and 83.8% on PASCAL VOC 2012 (Table 11)6.
<a id="S0341"></a> Source: p.11 S0341
This is our single-model result. on PASCAL VOC 2012 is 10 points higher than the previ- Ensemble.
<a id="S0342"></a> Source: p.11 S0342
In Faster R-CNN, the system is designed to learn ous state-of-the-art result [6]. region proposals and also object classifiers, so an ensemble ImageNet Detection can be used to boost both tasks.
<a id="S0343"></a> Source: p.11 S0343
We use an ensemble for The ImageNet Detection (DET) task involves 200 object proposing regions, and the union set of proposals are procategories.
<a id="S0344"></a> Source: p.11 S0344
Our cessed by an ensemble of per-region classifiers.
<a id="S0345"></a> Source: p.11 S0345
Table 9 object detection algorithm for ImageNet DET is the same shows our result based on an ensemble of 3 networks.
<a id="S0346"></a> Source: p.11 S0346
The networks are premAP is 59.0% and 37.4% on the test-dev set.
<a id="S0347"></a> Source: p.11 S0347
This result trained on the 1000-class ImageNet classification set, and won the 1st place in the detection task in COCO 2015. are fine-tuned on the DET data.
<a id="S0348"></a> Source: p.11 S0348
We split the validation set PASCAL VOC into two parts (val1/val2) following [8].
<a id="S0349"></a> Source: p.11 S0349
We fine-tune the We revisit the PASCAL VOC dataset based on the above detection models using the DET training set and the val1 model.
<a id="S0350"></a> Source: p.11 S0350
With the single model on the COCO dataset (55.7% set.
<a id="S0351"></a> Source: p.11 S0351
We do not use other mAP@.5 in Table 9), we fine-tune this model on the PAS- ILSVRC 2015 data.
<a id="S0352"></a> Source: p.11 S0352
Our single model with ResNet-101 has CAL VOC sets.
<a id="S0353"></a> Source: p.11 S0353
The improvements of box refinement, con- 6http://host.robots.ox.ac.uk:8080/anonymous/3OJ4OJ.html, text, and multi-scale testing are also adopted.
Page 12
<a id="S0354"></a> Source: p.12 S0354
LOC LOC LOC error classification top-5 LOC error top-5 localization err testing method method network on GT CLS network on predicted CLS val test VGG’s [41] VGG-16 1-crop 33.1 [41] OverFeat [40] (ILSVRC’13) 30.0 29.9 RPN ResNet-101 1-crop 13.3 RPN ResNet-101 dense 11.7 GoogLeNet [44] (ILSVRC’14) - 26.7 RPN ResNet-101 dense ResNet-101 14.4 VGG [41] (ILSVRC’14) 26.9 25.3 RPN+RCNN ResNet-101 dense ResNet-101 10.6 ours (ILSVRC’15) 8.9 9.0 RPN+RCNN ensemble dense ensemble 8.9 Table 14.
<a id="S0355"></a> Source: p.12 S0355
Comparisons of localization error (%) on the ImageNet Table 13.
<a id="S0356"></a> Source: p.12 S0356
Localization error (%) on the ImageNet validation.
<a id="S0357"></a> Source: p.12 S0357
In dataset with state-of-the-art methods. the column of “LOC error on GT class” ([41]), the ground truth class is used.
<a id="S0358"></a> Source: p.12 S0358
In the “testing” column, “1-crop” denotes testing on a center crop of 224×224 pixels, “dense” denotes dense (fully ports a center-crop error of 33.1% (Table 13) using ground convolutional) and multi-scale testing. truth classes.
<a id="S0359"></a> Source: p.12 S0359
Under the same setting, our RPN method using ResNet-101 net significantly reduces the center-crop error to 13.3%.
<a id="S0360"></a> Source: p.12 S0360
This comparison demonstrates the excellent 58.8% mAP and our ensemble of 3 models has 62.1% mAP performance of our framework.
<a id="S0361"></a> Source: p.12 S0361
With dense (fully convoluon the DET test set (Table 12).
<a id="S0362"></a> Source: p.12 S0362
This result won the 1st place tional) and multi-scale testing, our ResNet-101 has an error in the ImageNet detection task in ILSVRC 2015, surpassing of 11.7% using ground truth classes.
<a id="S0363"></a> Source: p.12 S0363
Using ResNet-101 for the second place by 8.5 points (absolute). predicting classes (4.6% top-5 classification error, Table 4), the top-5 localization error is 14.4%. C.
<a id="S0364"></a> Source: p.12 S0364
ImageNet Localization The above results are only based on the proposal network (RPN) in Faster R-CNN [32].
<a id="S0365"></a> Source: p.12 S0365
One may use the detection The ImageNet Localization (LOC) task [36] requires to network (Fast R-CNN [7]) in Faster R-CNN to improve the classify and localize the objects.
<a id="S0366"></a> Source: p.12 S0366
But we notice that on this dataset, one image usually assume that the image-level classifiers are first adopted for contains a single dominate object, and the proposal regions predicting the class labels of an image, and the localizahighly overlap with each other and thus have very similar tion algorithm only accounts for predicting bounding boxes RoI-pooled features.
<a id="S0367"></a> Source: p.12 S0367
As a result, the image-centric training based on the predicted classes.
<a id="S0368"></a> Source: p.12 S0368
We adopt the “per-class reof Fast R-CNN [7] generates samples of small variations, gression” (PCR) strategy [40, 41], learning a bounding box which may not be desired for stochastic training.
<a id="S0369"></a> Source: p.12 S0369
We pre-train the networks for Imby this, in our current experiment we use the original RageNet classification and then fine-tune them for localiza- CNN [8] that is RoI-centric, in place of Fast R-CNN. tion.
<a id="S0370"></a> Source: p.12 S0370
We train networks on the provided 1000-class Ima- Our R-CNN implementation is as follows.
<a id="S0371"></a> Source: p.12 S0371
We apply the geNet training set. per-class RPN trained as above on the training images to Our localization algorithm is based on the RPN framepredict bounding boxes for the ground truth class.
<a id="S0372"></a> Source: p.12 S0372
These work of [32] with a few modifications.
<a id="S0373"></a> Source: p.12 S0373
Unlike the way in predicted boxes play a role of class-dependent proposals. [32] that is category-agnostic, our RPN for localization is For each training image, the highest scored 200 proposals designed in a per-class form.
<a id="S0374"></a> Source: p.12 S0374
This RPN ends with two sibare extracted as training samples to train an R-CNN classiling 1 1 convolutional layers for binary classification (cls) × fier.
<a id="S0375"></a> Source: p.12 S0375
The image region is cropped from a proposal, warped and box regression (reg), as in [32].
<a id="S0376"></a> Source: p.12 S0376
The cls and reg layers to 224 224 pixels, and fed into the classification network are both in a per-class from, in contrast to [32].
<a id="S0377"></a> Source: p.12 S0377
The outputs of this network consist of two cally, the cls layer has a 1000-d output, and each dimension sibling fc layers for cls and reg, also in a per-class form. is binary logistic regression for predicting being or not be- This R-CNN network is fine-tuned on the training set using an object class; the reg layer has a 1000 4-d output × ing a mini-batch size of 256 in the RoI-centric fashion.
<a id="S0378"></a> Source: p.12 S0378
For consisting of box regressors for 1000 classes.
<a id="S0379"></a> Source: p.12 S0379
As in [32], testing, the RPN generates the highest scored 200 proposals our bounding box regression is with reference to multiple for each predicted class, and the R-CNN network is used to translation-invariant “anchor” boxes at each position. update these proposals’ scores and box positions.
<a id="S0380"></a> Source: p.12 S0380
As in our ImageNet classification training (Sec. 3.4), we This method reduces the top-5 localization error to randomly sample 224 224 crops for data augmentation. × 10.6% (Table 13).
<a id="S0381"></a> Source: p.12 S0381
This is our single-model result on the We use a mini-batch size of 256 images for fine-tuning.
<a id="S0382"></a> Source: p.12 S0382
Using an ensemble of networks for both clasavoid negative samples being dominate, 8 anchors are ransification and localization, we achieve a top-5 localization domly sampled for each image, where the sampled positive error of 9.0% on the test set.
<a id="S0383"></a> Source: p.12 S0383
This number significantly outand negative anchors have a ratio of 1:1 [32].
<a id="S0384"></a> Source: p.12 S0384
For testing, performs the ILSVRC 14 results (Table 14), showing a 64% the network is applied on the image fully-convolutionally. relative reduction of error.
<a id="S0385"></a> Source: p.12 S0385
This result won the 1st place in Table 13 compares the localization results.
<a id="S0386"></a> Source: p.12 S0386
Following the ImageNet localization task in ILSVRC 2015. [41], we first perform “oracle” testing using the ground truth class as the classification prediction.