Spaces:

tongxiaojun
/

alexnet

Running

App Files Files Community

tongxiaojun commited on May 30

Commit

16bcb11

verified ·

1 Parent(s): 83ae456

Add 2 files

Browse files

Files changed (2) hide show

index.html +113 -113
prompts.txt +2 -1

index.html CHANGED Viewed

@@ -1,9 +1,9 @@
 <!DOCTYPE html>
-<html lang="en">
 <head>
     <meta charset="UTF-8">
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
-    <title>ImageNet Classification with Deep Convolutional Neural Networks</title>
     <script src="https://cdn.tailwindcss.com"></script>
     <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.0/css/all.min.css">
     <style>
@@ -86,12 +86,12 @@
 </head>
 <body class="bg-gray-50 text-gray-900 font-sans">
     <div class="container mx-auto px-4 py-8 max-w-6xl">
-        <!-- Language Selector -->
         <div class="flex justify-end mb-6">
             <div class="relative">
                 <button id="languageBtn" class="flex items-center space-x-2 bg-white px-4 py-2 rounded-lg shadow-md hover:shadow-lg transition">
                     <i class="fas fa-globe"></i>
-                    <span id="currentLanguage">English</span>
                     <i class="fas fa-chevron-down"></i>
                 </button>
                 <div id="languageDropdown" class="hidden absolute right-0 mt-2 w-48 bg-white rounded-md shadow-lg z-10">
@@ -103,235 +103,235 @@
             </div>
         </div>
-        <!-- Header -->
         <header class="mb-12 text-center">
-            <h1 id="title" class="text-3xl md:text-4xl font-bold mb-4">ImageNet Classification with Deep Convolutional Neural Networks</h1>
             <div class="flex flex-wrap justify-center gap-4 mb-6">
-                <div class="author-highlight" data-affiliation="University of Toronto">
-                    <span id="author1" class="font-semibold">Alex Krizhevsky</span>
                     <span class="text-gray-600">[email protected]</span>
                 </div>
-                <div class="author-highlight" data-affiliation="University of Toronto">
-                    <span id="author2" class="font-semibold">Ilya Sutskever</span>
                     <span class="text-gray-600">[email protected]</span>
                 </div>
-                <div class="author-highlight" data-affiliation="University of Toronto">
-                    <span id="author3" class="font-semibold">Geoffrey E. Hinton</span>
                     <span class="text-gray-600">[email protected]</span>
                 </div>
             </div>
             <div class="flex justify-center space-x-4">
                 <button class="pdf-btn px-4 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700 transition flex items-center">
                     <i class="fas fa-file-pdf mr-2"></i>
-                    <span id="downloadPdf">Download PDF</span>
                 </button>
                 <button class="cite-btn px-4 py-2 bg-gray-200 rounded-lg hover:bg-gray-300 transition flex items-center">
                     <i class="fas fa-quote-right mr-2"></i>
-                    <span id="citePaper">Cite this paper</span>
                 </button>
             </div>
         </header>
-        <!-- Navigation -->
         <nav class="mb-12 sticky top-0 bg-white shadow-sm rounded-lg p-4 z-20">
             <ul class="flex flex-wrap justify-center gap-4">
-                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition active" data-section="abstract">Abstract</button></li>
-                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition" data-section="introduction">Introduction</button></li>
-                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition" data-section="dataset">Dataset</button></li>
-                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition" data-section="architecture">Architecture</button></li>
-                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition" data-section="overfitting">Overfitting</button></li>
-                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition" data-section="results">Results</button></li>
             </ul>
         </nav>
-        <!-- Main Content -->
         <main class="bg-white rounded-lg shadow-md p-6 md:p-8">
-            <!-- Abstract Section -->
             <section id="abstract" class="section mb-12">
-                <h2 class="text-2xl font-bold mb-4 border-b pb-2">Abstract</h2>
                 <p class="mb-4">
-                    We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art.
                 </p>
                 <p class="mb-4">
-                    The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective.
                 </p>
                 <p>
-                    We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.
                 </p>
             </section>
-            <!-- Introduction Section -->
             <section id="introduction" class="section mb-12">
-                <h2 class="text-2xl font-bold mb-4 border-b pb-2">1 Introduction</h2>
                 <p class="mb-4">
-                    Current approaches to object recognition make essential use of machine learning methods. To improve their performance, we can collect larger datasets, learn more powerful models, and use better techniques for preventing overfitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]).
                 </p>
                 <p class="mb-4">
-                    Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the current best error rate on the MNIST digit-recognition task (<0.3%) approaches human performance [4]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets.
                 </p>
                 <p class="mb-4">
-                    The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories.
                 </p>
                 <p class="mb-4">
-                    To learn about thousands of objects from millions of images, we need a model with a large learning capacity. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies).
                 </p>
                 <p class="mb-4">
-                    Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luckily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs.
                 </p>
                 <div class="citation p-4 rounded-lg mb-4">
-                    <p class="font-semibold">Key Contributions:</p>
                     <ul class="list-disc pl-5">
-                        <li>We trained one of the largest convolutional neural networks to date on ImageNet subsets</li>
-                        <li>We achieved by far the best results ever reported on these datasets</li>
-                        <li>We developed a highly-optimized GPU implementation of 2D convolution</li>
-                        <li>Our network contains several new and unusual features that improve performance</li>
                     </ul>
                 </div>
             </section>
-            <!-- Dataset Section -->
             <section id="dataset" class="section mb-12">
-                <h2 class="text-2xl font-bold mb-4 border-b pb-2">2 The Dataset</h2>
                 <p class="mb-4">
-                    ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Amazon's Mechanical Turk crowd-sourcing tool.
                 </p>
                 <p class="mb-4">
-                    Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly:
                 </p>
                 <div class="grid grid-cols-1 md:grid-cols-3 gap-4 mb-4">
                     <div class="bg-blue-50 p-4 rounded-lg text-center">
-                        <p class="text-3xl font-bold text-blue-700">1.2M</p>
-                        <p class="text-gray-700">Training images</p>
                     </div>
                     <div class="bg-blue-50 p-4 rounded-lg text-center">
-                        <p class="text-3xl font-bold text-blue-700">50K</p>
-                        <p class="text-gray-700">Validation images</p>
                     </div>
                     <div class="bg-blue-50 p-4 rounded-lg text-center">
-                        <p class="text-3xl font-bold text-blue-700">150K</p>
-                        <p class="text-gray-700">Testing images</p>
                     </div>
                 </div>
                 <p class="mb-4">
-                    On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model.
                 </p>
                 <p class="mb-4">
-                    ImageNet consists of variable-resolution images, while our system requires a constant input dimensionality. Therefore, we down-sampled the images to a fixed resolution of 256×256. Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central 256×256 patch from the resulting image.
                 </p>
             </section>
-            <!-- Architecture Section -->
             <section id="architecture" class="section mb-12">
-                <h2 class="text-2xl font-bold mb-4 border-b pb-2">3 The Architecture</h2>
                 <p class="mb-4">
-                    The architecture of our network contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network's architecture.
                 </p>
                 <div class="mb-8">
-                    <h3 class="text-xl font-semibold mb-3">3.1 ReLU Nonlinearity</h3>
                     <p class="mb-4">
-                        The standard way to model a neuron's output f as a function of its input x is with f(x) = tanh(x) or f(x) = (1 + e⁻ˣ)⁻¹. In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity f(x) = max(0, x). Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs).
                     </p>
                     <p class="mb-4">
-                        Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations required to reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional network.
                     </p>
                     <div class="figure p-4 rounded-lg mb-4 text-center">
-                        <img src="https://paperswithcode.com/media/thumbnails/paper/2012/image-net-classification-with-deep-convolutional-neural-networks-1.jpg" alt="ReLU vs tanh performance" class="mx-auto max-w-full h-auto rounded">
-                        <p class="text-sm text-gray-600 mt-2">Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a 25% training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line).</p>
                     </div>
                 </div>
                 <div class="mb-8">
-                    <h3 class="text-xl font-semibold mb-3">3.2 Training on Multiple GPUs</h3>
                     <p class="mb-4">
-                        A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs.
                     </p>
                     <p class="mb-4">
-                        The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU.
                     </p>
                     <p class="mb-4">
-                        The two-GPU net takes slightly less time to train than the one-GPU net, and reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively.
                     </p>
                 </div>
                 <div class="mb-8">
-                    <h3 class="text-xl font-semibold mb-3">3.5 Overall Architecture</h3>
                     <p class="mb-4">
-                        The net contains eight layers with weights; the first five are convolutional and the remaining three are fully-connected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels.
                     </p>
                     <div class="figure p-4 rounded-lg mb-4 text-center">
-                        <img src="https://miro.medium.com/v2/resize:fit:1400/1*1TkRB2T0qm5Fv5vApTkpeA.png" alt="Network architecture" class="mx-auto max-w-full h-auto rounded">
-                        <p class="text-sm text-gray-600 mt-2">Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities between the two GPUs.</p>
                     </div>
                     <p class="mb-4">
-                        The first convolutional layer filters the 224×224×3 input image with 96 kernels of size 11×11×3 with a stride of 4 pixels. The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size 5×5×48. The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers.
                     </p>
                     <p>
-                        The fully-connected layers have 4096 neurons each. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer.
                     </p>
                 </div>
             </section>
-            <!-- Overfitting Section -->
             <section id="overfitting" class="section mb-12">
-                <h2 class="text-2xl font-bold mb-4 border-b pb-2">4 Reducing Overfitting</h2>
                 <p class="mb-4">
-                    Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting. Below, we describe the two primary ways in which we combat overfitting.
                 </p>
                 <div class="mb-8">
-                    <h3 class="text-xl font-semibold mb-3">4.1 Data Augmentation</h3>
                     <p class="mb-4">
-                        The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations. We employ two distinct forms of data augmentation:
                     </p>
                     <div class="grid grid-cols-1 md:grid-cols-2 gap-4 mb-4">
                         <div class="bg-gray-50 p-4 rounded-lg">
-                            <h4 class="font-semibold mb-2">1. Image Translations and Reflections</h4>
-                            <p>We extract random 224×224 patches (and their horizontal reflections) from the 256×256 images. This increases the size of our training set by a factor of 2048.</p>
                         </div>
                         <div class="bg-gray-50 p-4 rounded-lg">
-                            <h4 class="font-semibold mb-2">2. Altering RGB Intensities</h4>
-                            <p>We perform PCA on the set of RGB pixel values and add multiples of the principal components with magnitudes proportional to the corresponding eigenvalues.</p>
                         </div>
                     </div>
                     <p>
-                        This scheme reduces the top-1 error rate by over 1%. At test time, the network makes a prediction by extracting five 224×224 patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions.
                     </p>
                 </div>
                 <div class="mb-8">
                     <h3 class="text-xl font-semibold mb-3">4.2 Dropout</h3>
                     <p class="mb-4">
-                        The recently-introduced technique, called "dropout" [10], consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are "dropped out" in this way do not contribute to the forward pass and do not participate in backpropagation.
                     </p>
                     <p class="mb-4">
-                        This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons.
                     </p>
                     <p>
-                        We use dropout in the first two fully-connected layers. Without dropout, our network exhibits substantial overfitting. Dropout roughly doubles the number of iterations required to converge.
                     </p>
                 </div>
             </section>
-            <!-- Results Section -->
             <section id="results" class="section">
-                <h2 class="text-2xl font-bold mb-4 border-b pb-2">6 Results</h2>
                 <p class="mb-4">
-                    Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5 test set error rates of 37.5% and 17.0%. The best performance achieved during the ILSVRC-2010 competition was 47.1% and 28.2% with an approach that averages the predictions produced from six sparse-coding models trained on different features [2].
                 </p>
                 <div class="table-container mb-6">
                     <table class="w-full mb-4">
                         <thead>
                             <tr>
-                                <th class="px-4 py-2">Model</th>
                                 <th class="px-4 py-2">Top-1</th>
                                 <th class="px-4 py-2">Top-5</th>
                             </tr>
                         </thead>
                         <tbody>
                             <tr>
-                                <td class="px-4 py-2">Sparse coding[2]</td>
                                 <td class="px-4 py-2">47.1%</td>
                                 <td class="px-4 py-2">28.2%</td>
                             </tr>
@@ -341,50 +341,50 @@
                                 <td class="px-4 py-2">25.7%</td>
                             </tr>
                             <tr class="bg-blue-50 font-semibold">
-                                <td class="px-4 py-2">CNN (Our model)</td>
                                 <td class="px-4 py-2">37.5%</td>
                                 <td class="px-4 py-2">17.0%</td>
                             </tr>
                         </tbody>
                     </table>
-                    <p class="text-sm text-gray-600 text-center">Table 1: Comparison of results on ILSVRC-2010 test set.</p>
                 </div>
                 <p class="mb-4">
-                    We also entered our model in the ILSVRC-2012 competition. The CNN described in this paper achieves a top-5 error rate of 18.2%. Averaging the predictions of five similar CNNs gives an error rate of 16.4%. The second-best contest entry achieved an error rate of 26.2% with an approach that averages the predictions of several classifiers trained on FVs computed from different types of densely-sampled features [7].
                 </p>
                 <div class="grid grid-cols-1 md:grid-cols-2 gap-6 mt-8">
                     <div class="figure p-4 rounded-lg">
-                        <img src="https://production-media.paperswithcode.com/datasets/4ef7862e315766034c533cfa2473998dbd0f3ef1.jpg" alt="Network filters" class="w-full h-auto rounded">
-                        <p class="text-sm text-gray-600 mt-2">Figure 3: 96 convolutional kernels of size 11×11×3 learned by the first convolutional layer.</p>
                     </div>
                     <div class="figure p-4 rounded-lg">
-                        <img src="https://production-media.paperswithcode.com/datasets/66491abbb6c923fad2c100a539ff8b47812a62b4.jpg" alt="Classification examples" class="w-full h-auto rounded">
-                        <p class="text-sm text-gray-600 mt-2">Figure 4: (Left) Test images and top-5 predictions. (Right) Test images with similar training images based on feature activations.</p>
                     </div>
                 </div>
             </section>
         </main>
-        <!-- Footer -->
         <footer class="mt-12 text-center text-gray-600 text-sm">
-            <p id="footerText">© 2012 Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton. All rights reserved.</p>
             <div class="flex justify-center space-x-4 mt-4">
-                <a href="#" class="hover:text-blue-600"><i class="fab fa-github"></i> Code</a>
-                <a href="#" class="hover:text-blue-600"><i class="fas fa-book"></i> Cite</a>
-                <a href="#" class="hover:text-blue-600"><i class="fas fa-share-alt"></i> Share</a>
             </div>
         </footer>
-        <!-- Back to top button -->
         <button id="backToTop" class="back-to-top bg-blue-600 text-white p-3 rounded-full shadow-lg hover:bg-blue-700 transition">
             <i class="fas fa-arrow-up"></i>
         </button>
     </div>
     <script>
-        // Language translations
         const translations = {
             en: {
                 title: "ImageNet Classification with Deep Convolutional Neural Networks",
@@ -424,7 +424,7 @@
             }
         };
-        // Language selector functionality
         const languageBtn = document.getElementById('languageBtn');
         const languageDropdown = document.getElementById('languageDropdown');
         const currentLanguage = document.getElementById('currentLanguage');
@@ -439,7 +439,7 @@
                 currentLanguage.textContent = button.textContent.trim();
                 languageDropdown.classList.add('hidden');
-                // Update all translatable elements
                 document.getElementById('title').textContent = translations[lang].title;
                 document.getElementById('author1').textContent = translations[lang].author1;
                 document.getElementById('author2').textContent = translations[lang].author2;
@@ -450,7 +450,7 @@
             });
         });
-        // Navigation functionality
         const sections = document.querySelectorAll('.section');
         const navButtons = document.querySelectorAll('.nav-btn');
@@ -480,13 +480,13 @@
                 const sectionId = button.getAttribute('data-section');
                 const section = document.getElementById(sectionId);
-                // Update active button
                 navButtons.forEach(btn => {
                     btn.classList.remove('bg-blue-100', 'text-blue-700', 'active');
                 });
                 button.classList.add('bg-blue-100', 'text-blue-700', 'active');
-                // Scroll to section
                 window.scrollTo({
                     top: section.offsetTop - 100,
                     behavior: 'smooth'
@@ -494,24 +494,24 @@
             });
         });
-        // Set first nav button as active initially
         document.querySelector('.nav-btn').classList.add('bg-blue-100', 'text-blue-700', 'active');
-        // Citation modal
         const citeBtn = document.querySelector('.cite-btn');
         citeBtn.addEventListener('click', () => {
             alert('Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 1097-1105.');
         });
-        // PDF download
         const pdfBtn = document.querySelector('.pdf-btn');
         pdfBtn.addEventListener('click', () => {
-            alert('Redirecting to PDF download...');
-            // In a real implementation, this would download the PDF
             // window.location.href = 'path/to/paper.pdf';
         });
-        // Back to top button
         const backToTopBtn = document.getElementById('backToTop');
         window.addEventListener('scroll', () => {
@@ -531,7 +531,7 @@
             });
         });
-        // Initialize active nav on load
         window.addEventListener('load', updateActiveNav);
     </script>
 <p style="border-radius: 8px; text-align: center; font-size: 12px; color: #fff; margin-top: 16px;position: fixed; left: 8px; bottom: 8px; z-index: 10; background: rgba(0, 0, 0, 0.8); padding: 4px 8px;">Made with <img src="https://enzostvs-deepsite.hf.space/logo.svg" alt="DeepSite Logo" style="width: 16px; height: 16px; vertical-align: middle;display:inline-block;margin-right:3px;filter:brightness(0) invert(1);"><a href="https://enzostvs-deepsite.hf.space" style="color: #fff;text-decoration: underline;" target="_blank" >DeepSite</a> - 🧬 <a href="https://enzostvs-deepsite.hf.space?remix=tongxiaojun/alexnet" style="color: #fff;text-decoration: underline;" target="_blank" >Remix</a></p></body>

 <!DOCTYPE html>
+<html lang="zh-CN">
 <head>
     <meta charset="UTF-8">
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>使用深度卷积神经网络进行ImageNet分类</title>
     <script src="https://cdn.tailwindcss.com"></script>
     <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.0/css/all.min.css">
     <style>
 </head>
 <body class="bg-gray-50 text-gray-900 font-sans">
     <div class="container mx-auto px-4 py-8 max-w-6xl">
+        <!-- 语言选择器 -->
         <div class="flex justify-end mb-6">
             <div class="relative">
                 <button id="languageBtn" class="flex items-center space-x-2 bg-white px-4 py-2 rounded-lg shadow-md hover:shadow-lg transition">
                     <i class="fas fa-globe"></i>
+                    <span id="currentLanguage">中文</span>
                     <i class="fas fa-chevron-down"></i>
                 </button>
                 <div id="languageDropdown" class="hidden absolute right-0 mt-2 w-48 bg-white rounded-md shadow-lg z-10">
             </div>
         </div>
+        <!-- 页眉 -->
         <header class="mb-12 text-center">
+            <h1 id="title" class="text-3xl md:text-4xl font-bold mb-4">使用深度卷积神经网络进行ImageNet分类</h1>
             <div class="flex flex-wrap justify-center gap-4 mb-6">
+                <div class="author-highlight" data-affiliation="多伦多大学">
+                    <span id="author1" class="font-semibold">亚历克斯·克里热夫斯基</span>
                     <span class="text-gray-600">[email protected]</span>
                 </div>
+                <div class="author-highlight" data-affiliation="多伦多大学">
+                    <span id="author2" class="font-semibold">伊利亚·苏茨克弗</span>
                     <span class="text-gray-600">[email protected]</span>
                 </div>
+                <div class="author-highlight" data-affiliation="多伦多大学">
+                    <span id="author3" class="font-semibold">杰弗里·辛顿</span>
                     <span class="text-gray-600">[email protected]</span>
                 </div>
             </div>
             <div class="flex justify-center space-x-4">
                 <button class="pdf-btn px-4 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700 transition flex items-center">
                     <i class="fas fa-file-pdf mr-2"></i>
+                    <span id="downloadPdf">下载PDF</span>
                 </button>
                 <button class="cite-btn px-4 py-2 bg-gray-200 rounded-lg hover:bg-gray-300 transition flex items-center">
                     <i class="fas fa-quote-right mr-2"></i>
+                    <span id="citePaper">引用本文</span>
                 </button>
             </div>
         </header>
+        <!-- 导航 -->
         <nav class="mb-12 sticky top-0 bg-white shadow-sm rounded-lg p-4 z-20">
             <ul class="flex flex-wrap justify-center gap-4">
+                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition active" data-section="abstract">摘要</button></li>
+                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition" data-section="introduction">引言</button></li>
+                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition" data-section="dataset">数据集</button></li>
+                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition" data-section="architecture">架构</button></li>
+                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition" data-section="overfitting">过拟合</button></li>
+                <li><button class="nav-btn nav-indicator px-4 py-2 rounded-lg hover:bg-blue-50 transition" data-section="results">结果</button></li>
             </ul>
         </nav>
+        <!-- 主要内容 -->
         <main class="bg-white rounded-lg shadow-md p-6 md:p-8">
+            <!-- 摘要部分 -->
             <section id="abstract" class="section mb-12">
+                <h2 class="text-2xl font-bold mb-4 border-b pb-2">摘要</h2>
                 <p class="mb-4">
+                    我们训练了一个大型深度卷积神经网络，将ImageNet LSVRC-2010竞赛中的120万张高分辨率图像分类为1000个不同的类别。在测试数据上，我们实现了37.5%和17.0%的top-1和top-5错误率，这比之前的最先进技术要好得多。
                 </p>
                 <p class="mb-4">
+                    该神经网络有6000万个参数和65万个神经元，由五个卷积层(其中一些后面跟着最大池化层)和三个全连接层组成，最后是一个1000路的softmax。为了加快训练速度，我们使用了非饱和神经元和卷积操作的非常高效的GPU实现。为了减少全连接层中的过拟合，我们采用了一种最近开发的称为"dropout"的正则化方法，该方法被证明非常有效。
                 </p>
                 <p>
+                    我们还在ILSVRC-2012竞赛中提交了这个模型的一个变体，并获得了15.3%的top-5测试错误率，而第二名的成绩是26.2%。
                 </p>
             </section>
+            <!-- 引言部分 -->
             <section id="introduction" class="section mb-12">
+                <h2 class="text-2xl font-bold mb-4 border-b pb-2">1 引言</h2>
                 <p class="mb-4">
+                    当前的对象识别方法主要使用机器学习方法。为了提高它们的性能，我们可以收集更大的数据集，学习更强大的模型，并使用更好的技术来防止过拟合。直到最近，标记图像的数据集相对较小——大约数万张图像(例如NORB[16]、Caltech-101/256[8,9]和CIFAR-10/100[12])。
                 </p>
                 <p class="mb-4">
+                    简单的识别任务可以用这种大小的数据集很好地解决，特别是如果用保留标签的变换进行增强。例如，MNIST数字识别任务上的当前最佳错误率(<0.3%)接近人类表现[4]。但是现实环境中的对象表现出相当大的可变性，所以要学会识别它们，必须使用更大的训练集。
                 </p>
                 <p class="mb-4">
+                    新的更大的数据集包括LabelMe[23]，它由数十万张完全分割的图像组成，以及ImageNet[6]，它由超过1500万张标记的高分辨率图像组成，分为22000多个类别。
                 </p>
                 <p class="mb-4">
+                    要从数百万张图像中学习数千个对象，我们需要一个具有大学习能力的模型。卷积神经网络(CNN)构成了这样一类模型[16,11,13,18,15,22,26]。它们的容量可以通过改变其深度和广度来控制，并且它们还对图像的性质做出了强大且基本正确的假设(即统计的平稳性和像素依赖的局部性)。
                 </p>
                 <p class="mb-4">
+                    尽管CNN具有吸引人的特性，并且尽管它们的局部架构相对高效，但将它们大规模应用于高分辨率图像仍然过于昂贵。幸运的是，当前的GPU与高度优化的2D卷积实现相结合，已经足够强大，可以促进有趣的大型CNN的训练。
                 </p>
                 <div class="citation p-4 rounded-lg mb-4">
+                    <p class="font-semibold">主要贡献:</p>
                     <ul class="list-disc pl-5">
+                        <li>我们在ImageNet子集上训练了迄今为止最大的卷积神经网络之一</li>
+                        <li>我们在这些数据集上取得了迄今为止最好的结果</li>
+                        <li>我们开发了高度优化的2D卷积GPU实现</li>
+                        <li>我们的网络包含几个新的和不寻常的特征，可以提高性能</li>
                     </ul>
                 </div>
             </section>
+            <!-- 数据集部分 -->
             <section id="dataset" class="section mb-12">
+                <h2 class="text-2xl font-bold mb-4 border-b pb-2">2 数据集</h2>
                 <p class="mb-4">
+                    ImageNet是一个包含超过1500万张标记高分辨率图像的数据集，属于大约22000个类别。这些图像是从网络上收集的，并由人类标注者使用亚马逊的Mechanical Turk众包工具进行标记。
                 </p>
                 <p class="mb-4">
+                    从2010年开始，作为Pascal视觉对象挑战赛的一部分，每年都会举办一场名为ImageNet大规模视觉识别挑战赛(ILSVRC)的比赛。ILSVRC使用ImageNet的一个子集，每个1000个类别中大约有1000张图像。总共有大约:
                 </p>
                 <div class="grid grid-cols-1 md:grid-cols-3 gap-4 mb-4">
                     <div class="bg-blue-50 p-4 rounded-lg text-center">
+                        <p class="text-3xl font-bold text-blue-700">120万</p>
+                        <p class="text-gray-700">训练图像</p>
                     </div>
                     <div class="bg-blue-50 p-4 rounded-lg text-center">
+                        <p class="text-3xl font-bold text-blue-700">5万</p>
+                        <p class="text-gray-700">验证图像</p>
                     </div>
                     <div class="bg-blue-50 p-4 rounded-lg text-center">
+                        <p class="text-3xl font-bold text-blue-700">15万</p>
+                        <p class="text-gray-700">测试图像</p>
                     </div>
                 </div>
                 <p class="mb-4">
+                    在ImageNet上，通常报告两个错误率：top-1和top-5，其中top-5错误率是测试图像中正确标签不在模型认为最可能的五个标签中的比例。
                 </p>
                 <p class="mb-4">
+                    ImageNet由可变分辨率的图像组成，而我们的系统需要恒定的输入维度。因此，我们将图像下采样到固定的256×256分辨率。给定一个矩形图像，我们首先重新缩放图像，使短边长度为256，然后从结果图像中裁剪出中心的256×256块。
                 </p>
             </section>
+            <!-- 架构部分 -->
             <section id="architecture" class="section mb-12">
+                <h2 class="text-2xl font-bold mb-4 border-b pb-2">3 架构</h2>
                 <p class="mb-4">
+                    我们的网络架构包含八个学习层——五个卷积层和三个全连接层。下面，我们描述了我们网络架构中一些新颖或不寻常的特征。
                 </p>
                 <div class="mb-8">
+                    <h3 class="text-xl font-semibold mb-3">3.1 ReLU非线性</h3>
                     <p class="mb-4">
+                        模拟神经元输出f作为其输入x的函数的标准方法是使用f(x) = tanh(x)或f(x) = (1 + e⁻ˣ)⁻¹。就梯度下降的训练时间而言，这些饱和非线性比非饱和非线性f(x) = max(0, x)慢得多。遵循Nair和Hinton[20]，我们将具有这种非线性的神经元称为整流线性单元(ReLU)。
                     </p>
                     <p class="mb-4">
+                        具有ReLU的深度卷积神经网络的训练速度比具有tanh单元的等效网络快几倍。这在图1中得到了证明，该图显示了在CIFAR-10数据集上���一个特定的四层卷积网络达到25%训练错误率所需的迭代次数。
                     </p>
                     <div class="figure p-4 rounded-lg mb-4 text-center">
+                        <img src="https://paperswithcode.com/media/thumbnails/paper/2012/image-net-classification-with-deep-convolutional-neural-networks-1.jpg" alt="ReLU与tanh性能对比" class="mx-auto max-w-full h-auto rounded">
+                        <p class="text-sm text-gray-600 mt-2">图1: 在CIFAR-10上，具有ReLU(实线)的四层卷积神经网络达到25%训练错误率的速度比具有tanh神经元(虚线)的等效网络快六倍。</p>
                     </div>
                 </div>
                 <div class="mb-8">
+                    <h3 class="text-xl font-semibold mb-3">3.2 在多GPU上训练</h3>
                     <p class="mb-4">
+                        单个GTX 580 GPU只有3GB内存，这限制了可以在其上训练的网络的最大尺寸。事实证明，120万个训练样本足以训练太大而无法在一个GPU上容纳的网络。因此，我们将网络分布在两个GPU上。
                     </p>
                     <p class="mb-4">
+                        我们采用的并行化方案基本上将一半的内核(或神经元)放在每个GPU上，还有一个额外的技巧：GPU只在某些层中通信。这意味着，例如，第3层的内核从第2层的所有内核映射中获取输入。然而，第4层的内核仅从位于同一GPU上的第3层的内核映射中获取输入。
                     </p>
                     <p class="mb-4">
+                        双GPU网络的训练时间略少于单GPU网络，并将我们的top-1和top-5错误率分别降低了1.7%和1.2%。
                     </p>
                 </div>
                 <div class="mb-8">
+                    <h3 class="text-xl font-semibold mb-3">3.5 整体架构</h3>
                     <p class="mb-4">
+                        该网络包含八个带权重的层；前五个是卷积层，其余三个是全连接层。最后一个全连接层的输出被馈送到一个1000路的softmax，它产生1000个类别标签上的分布。
                     </p>
                     <div class="figure p-4 rounded-lg mb-4 text-center">
+                        <img src="https://miro.medium.com/v2/resize:fit:1400/1*1TkRB2T0qm5Fv5vApTkpeA.png" alt="网络架构" class="mx-auto max-w-full h-auto rounded">
+                        <p class="text-sm text-gray-600 mt-2">图2: 我们的CNN架构示意图，明确显示了两个GPU之间的职责划分。</p>
                     </div>
                     <p class="mb-4">
+                        第一个卷积层用96个大小为11×11×3的内核以4像素的步长过滤224×224×3的输入图像。第二个卷积层将第一个卷积层的(响应归一化和池化的)输出作为输入，并用256个大小为5×5×48的内核进行过滤。第三、第四和第五卷积层相互连接，中间没有任何池化或归一化层。
                     </p>
                     <p>
+                        全连接层各有4096个神经元。ReLU非线性应用于每个卷积层和全连接层的输出。
                     </p>
                 </div>
             </section>
+            <!-- 过拟合部分 -->
             <section id="overfitting" class="section mb-12">
+                <h2 class="text-2xl font-bold mb-4 border-b pb-2">4 减少过拟合</h2>
                 <p class="mb-4">
+                    我们的神经网络架构有6000万个参数。尽管ILSVRC的1000个类别使每个训练样本对从图像到标签的映射施加了10位的约束，但这不足以在没有相当大的过拟合的情况下学习这么多参数。下面，我们描述了我们对抗过拟合的两种主要方式。
                 </p>
                 <div class="mb-8">
+                    <h3 class="text-xl font-semibold mb-3">4.1 数据增强</h3>
                     <p class="mb-4">
+                        减少图像数据过拟合的最简单和最常见的方法是使用保留标签的变换人为地扩大数据集。我们采用两种不同形式的数据增强：
                     </p>
                     <div class="grid grid-cols-1 md:grid-cols-2 gap-4 mb-4">
                         <div class="bg-gray-50 p-4 rounded-lg">
+                            <h4 class="font-semibold mb-2">1. 图像平移和反射</h4>
+                            <p>我们从256×256图像中提取随机224×224块(及其水平反射)。这使我们的训练集大小增加了2048倍。</p>
                         </div>
                         <div class="bg-gray-50 p-4 rounded-lg">
+                            <h4 class="font-semibold mb-2">2. 改变RGB强度</h4>
+                            <p>我们对RGB像素值集执行PCA，并添加主成分的倍数，其大小与相应的特征值成比例。</p>
                         </div>
                     </div>
                     <p>
+                        该方案将top-1错误率降低了1%以上。在测试时，网络通过提取五个224×224块(四个角块和中心块)及其水平反射(总共十个块)并平均预测结果来进行预测。
                     </p>
                 </div>
                 <div class="mb-8">
                     <h3 class="text-xl font-semibold mb-3">4.2 Dropout</h3>
                     <p class="mb-4">
+                        最近引入的技术"dropout"[10]包括以0.5的概率将每个隐藏神经元的输出设置为零。以这种方式"丢弃"的神经元不参与前向传递，也不参与反向传播。
                     </p>
                     <p class="mb-4">
+                        这种技术减少了神经元的复杂共适应，因为神经元不能依赖于特定其他神经元的存在。因此，它被迫学习与许多其他神经元的随机子集一起使用的更鲁棒的特征。
                     </p>
                     <p>
+                        我们在前两个全连接层中使用dropout。没有dropout，我们的网络表现出严重的过拟合。Dropout大约使收敛所需的迭代次数翻倍。
                     </p>
                 </div>
             </section>
+            <!-- 结果部分 -->
             <section id="results" class="section">
+                <h2 class="text-2xl font-bold mb-4 border-b pb-2">6 结果</h2>
                 <p class="mb-4">
+                    我们在ILSVRC-2010上的结果总结在表1中。我们的网络实现了37.5%和17.0%的top-1和top-5测试集错误率。ILSVRC-2010竞赛期间实现的最佳性能是47.1%和28.2%，采用的方法是对在不同特征上训练的六个稀疏编码模型产生的预测进行平均[2]。
                 </p>
                 <div class="table-container mb-6">
                     <table class="w-full mb-4">
                         <thead>
                             <tr>
+                                <th class="px-4 py-2">模型</th>
                                 <th class="px-4 py-2">Top-1</th>
                                 <th class="px-4 py-2">Top-5</th>
                             </tr>
                         </thead>
                         <tbody>
                             <tr>
+                                <td class="px-4 py-2">稀疏编码[2]</td>
                                 <td class="px-4 py-2">47.1%</td>
                                 <td class="px-4 py-2">28.2%</td>
                             </tr>
                                 <td class="px-4 py-2">25.7%</td>
                             </tr>
                             <tr class="bg-blue-50 font-semibold">
+                                <td class="px-4 py-2">CNN(我们的模型)</td>
                                 <td class="px-4 py-2">37.5%</td>
                                 <td class="px-4 py-2">17.0%</td>
                             </tr>
                         </tbody>
                     </table>
+                    <p class="text-sm text-gray-600 text-center">表1: ILSVRC-2010测试集上的结果比较。</p>
                 </div>
                 <p class="mb-4">
+                    我们还在ILSVRC-2012竞赛中提交了这个模型的一个变体。本文描述的CNN实现了18.2%的top-5错误率。对五个类似CNN的预测进行平均得到了16.4%的错误率。第二名的参赛作品实现了26.2%的错误率，采用的方法是对从不同类型密集采样特征计算的FVs训练的多个分类器的预测进行平均[7]。
                 </p>
                 <div class="grid grid-cols-1 md:grid-cols-2 gap-6 mt-8">
                     <div class="figure p-4 rounded-lg">
+                        <img src="https://production-media.paperswithcode.com/datasets/4ef7862e315766034c533cfa2473998dbd0f3ef1.jpg" alt="网络滤波器" class="w-full h-auto rounded">
+                        <p class="text-sm text-gray-600 mt-2">图3: 第一个卷积层学习的96个大小为11×11×3的卷积核。</p>
                     </div>
                     <div class="figure p-4 rounded-lg">
+                        <img src="https://production-media.paperswithcode.com/datasets/66491abbb6c923fad2c100a539ff8b47812a62b4.jpg" alt="分类示例" class="w-full h-auto rounded">
+                        <p class="text-sm text-gray-600 mt-2">图4: (左)测试图像和top-5预测。(右)基于特征激活的测试图像与类似训练图像。</p>
                     </div>
                 </div>
             </section>
         </main>
+        <!-- 页脚 -->
         <footer class="mt-12 text-center text-gray-600 text-sm">
+            <p id="footerText">© 2012 亚历克斯·克里热夫斯基, 伊利亚·苏茨克弗, 杰弗里·辛顿。保留所有权利。</p>
             <div class="flex justify-center space-x-4 mt-4">
+                <a href="#" class="hover:text-blue-600"><i class="fab fa-github"></i> 代码</a>
+                <a href="#" class="hover:text-blue-600"><i class="fas fa-book"></i> 引用</a>
+                <a href="#" class="hover:text-blue-600"><i class="fas fa-share-alt"></i> 分享</a>
             </div>
         </footer>
+        <!-- 返回顶部按钮 -->
         <button id="backToTop" class="back-to-top bg-blue-600 text-white p-3 rounded-full shadow-lg hover:bg-blue-700 transition">
             <i class="fas fa-arrow-up"></i>
         </button>
     </div>
     <script>
+        // 语言翻译
         const translations = {
             en: {
                 title: "ImageNet Classification with Deep Convolutional Neural Networks",
             }
         };
+        // 语言选择器功能
         const languageBtn = document.getElementById('languageBtn');
         const languageDropdown = document.getElementById('languageDropdown');
         const currentLanguage = document.getElementById('currentLanguage');
                 currentLanguage.textContent = button.textContent.trim();
                 languageDropdown.classList.add('hidden');
+                // 更新所有可翻译元素
                 document.getElementById('title').textContent = translations[lang].title;
                 document.getElementById('author1').textContent = translations[lang].author1;
                 document.getElementById('author2').textContent = translations[lang].author2;
             });
         });
+        // 导航功能
         const sections = document.querySelectorAll('.section');
         const navButtons = document.querySelectorAll('.nav-btn');
                 const sectionId = button.getAttribute('data-section');
                 const section = document.getElementById(sectionId);
+                // 更新活动按钮
                 navButtons.forEach(btn => {
                     btn.classList.remove('bg-blue-100', 'text-blue-700', 'active');
                 });
                 button.classList.add('bg-blue-100', 'text-blue-700', 'active');
+                // 滚动到部分
                 window.scrollTo({
                     top: section.offsetTop - 100,
                     behavior: 'smooth'
             });
         });
+        // 初始设置第一个导航按钮为活动状态
         document.querySelector('.nav-btn').classList.add('bg-blue-100', 'text-blue-700', 'active');
+        // 引用模态框
         const citeBtn = document.querySelector('.cite-btn');
         citeBtn.addEventListener('click', () => {
             alert('Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 1097-1105.');
         });
+        // PDF下载
         const pdfBtn = document.querySelector('.pdf-btn');
         pdfBtn.addEventListener('click', () => {
+            alert('正在跳转到PDF下载...');
+            // 在实际实现中，这将下载PDF
             // window.location.href = 'path/to/paper.pdf';
         });
+        // 返回顶部按钮
         const backToTopBtn = document.getElementById('backToTop');
         window.addEventListener('scroll', () => {
             });
         });
+        // 加载时初始化活动导航
         window.addEventListener('load', updateActiveNav);
     </script>
 <p style="border-radius: 8px; text-align: center; font-size: 12px; color: #fff; margin-top: 16px;position: fixed; left: 8px; bottom: 8px; z-index: 10; background: rgba(0, 0, 0, 0.8); padding: 4px 8px;">Made with <img src="https://enzostvs-deepsite.hf.space/logo.svg" alt="DeepSite Logo" style="width: 16px; height: 16px; vertical-align: middle;display:inline-block;margin-right:3px;filter:brightness(0) invert(1);"><a href="https://enzostvs-deepsite.hf.space" style="color: #fff;text-decoration: underline;" target="_blank" >DeepSite</a> - 🧬 <a href="https://enzostvs-deepsite.hf.space?remix=tongxiaojun/alexnet" style="color: #fff;text-decoration: underline;" target="_blank" >Remix</a></p></body>

prompts.txt CHANGED Viewed

@@ -1,2 +1,3 @@
 设计一个支持 i18n 的网站，内容 # ImageNet Classification with Deep Convolutional Neural Networks    Alex Krizhevsky University of Toronto [email protected]    Ilya Sutskever University of Toronto [email protected]    Geoffrey E. Hinton University of Toronto [email protected]    # Abstract    We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of $3 7 . 5 \%$ and $1 7 . 0 \%$ which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of $1 5 . 3 \%$ , compared to $2 6 . 2 \%$ achieved by the second-best entry.    # 1 Introduction    Current approaches to object recognition make essential use of machine learning methods. To improve their performance, we can collect larger datasets, learn more powerful models, and use better techniques for preventing overfitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the currentbest error rate on the MNIST digit-recognition task $( < 0 . 3 \% )$ approaches human performance [4]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to collect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories.    To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this problem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.    Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luckily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting.    The specific contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly1. Our network contains a number of new and unusual features which improve its performance and reduce its training time, which are detailed in Section 3. The size of our network made overfitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing overfitting, which are described in Section 4. Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than $1 \%$ of the model’s parameters) resulted in inferior performance.    In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.    # 2 The Dataset    ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Amazon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images.    ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model.    ImageNet consists of variable-resolution images, while our system requires a constant input dimensionality. Therefore, we down-sampled the images to a fixed resolution of $2 5 6 \times 2 5 6$ . Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central $2 5 6 \times 2 5 6$ patch from the resulting image. We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels.    # 3 The Architecture    The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first.    # 3.1 ReLU Nonlinearity    The standard way to model a neuron’s output $f$ as a function of its input $x$ is with $f ( x ) \ = \ \operatorname { t a n h } ( x )$ or $f ( x ) = ( 1 + e ^ { - x } ) ^ { - 1 }$ . In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity $f ( x ) = \operatorname* { m a x } ( 0 , x )$ . Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs). Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations required to reach $2 5 \%$ training error on the CIFAR-10 dataset for a particular four-layer convolutional network. This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron models.    We are not the first to consider alternatives to traditional neuron models in CNNs. For example, Jarrett et al. [11] claim that the nonlinearity $f ( x ) \stackrel { \textstyle - } { = } \left| \operatorname { t a n h } ( x ) \right|$ works particularly well with their type of contrast normalization followed by local average pooling on the Caltech-101 dataset. However, on this dataset the primary concern is preventing overfitting, so the effect they are observing is different from the accelerated ability to fit the training set which we report when using ReLUs. Faster learning has a great influence on the performance of large models trained on large datasets.    ![](images/4ef7862e315766034c533cfa2473998dbd0f3ef1c4bf94d4602c6bacef7b5bda.jpg)   Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a $2 5 \%$ training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line). The learning rates for each network were chosen independently to make training as fast as possible. No regularization of any kind was employed. The magnitude of the effect demonstrated here varies with network architecture, but networks with ReLUs consistently learn several times faster than equivalents with saturating neurons.    # 3.2 Training on Multiple GPUs    A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another’s memory directly, without going through host machine memory. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation.    The resultant architecture is somewhat similar to that of the “columnar” CNN employed by Cire¸san et al. [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1 and top-5 error rates by $1 . 7 \%$ and $1 . 2 \%$ , respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net2.    # 3.3 Local Response Normalization    ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by $a _ { x , y } ^ { i }$ the activity of a neuron computed by applying kernel $i$ at position $( x , y )$ and then applying the ReLU nonlinearity, the response-normalized activity $b _ { x , y } ^ { i }$ is given by the expression    $$ b _ { x , y } ^ { i } = a _ { x , y } ^ { i } / \left( k + \alpha \sum _ { j = \operatorname* { m a x } ( 0 , i - n / 2 ) } ^ { \operatorname* { m i n } ( N - 1 , i + n / 2 ) } ( a _ { x , y } ^ { j } ) ^ { 2 } \right) ^ { \beta } $$    where the sum runs over $n$ “adjacent” kernel maps at the same spatial position, and $N$ is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants $k , n , \alpha$ , and $\beta$ are hyper-parameters whose values are determined using a validation set; we used $k = 2$ , $n = 5$ , $\alpha = 1 0 ^ { - 4 }$ , and $\beta = 0 . 7 5$ . We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5).    This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by $1 . 4 \%$ and $1 . 2 \%$ , respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a $13 \%$ test error rate without normalization and $11 \%$ with normalization3.    # 3.4 Overlapping Pooling    Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced $s$ pixels apart, each summarizing a neighborhood of size $z \times z$ centered at the location of the pooling unit. If we set $s = z$ , we obtain traditional local pooling as commonly employed in CNNs. If we set $s < z$ , we obtain overlapping pooling. This is what we use throughout our network, with $s = 2$ and $z = 3$ . This scheme reduces the top-1 and top-5 error rates by $0 . 4 \%$ and $0 . 3 \%$ , respectively, as compared with the non-overlapping scheme $s = 2 , z = 2$ , which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.    # 3.5 Overall Architecture    Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fullyconnected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution.    The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fullyconnected layers are connected to all neurons in the previous layer. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer.    The first convolutional layer filters the $2 2 4 \times 2 2 4 \times 3$ input image with 96 kernels of size $1 1 \times 1 1 \times 3$ with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size $5 \times 5 \times 4 8$ . The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size $3 \times 3 \times$ 256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size $3 \times 3 \times 1 9 2$ , and the fifth convolutional layer has 256 kernels of size $3 \times 3 \times 1 9 2$ . The fully-connected layers have 4096 neurons each.    ![](images/c2c1e038483a692fb59e8b21254b08a0ad17b5535e38d56450018d8321cf78e0.jpg)   Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities between the two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264– 4096–4096–1000.    # 4 Reducing Overfitting    Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting. Below, we describe the two primary ways in which we combat overfitting.    # 4.1 Data Augmentation    The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk. In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free.    The first form of data augmentation consists of generating image translations and horizontal reflections. We do this by extracting random $2 2 4 \times 2 2 4$ patches (and their horizontal reflections) from the $2 5 6 \times 2 5 6$ images and training our network on these extracted patches4. This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly interdependent. Without this scheme, our network suffers from substantial overfitting, which would have forced us to use much smaller networks. At test time, the network makes a prediction by extracting five $2 2 4 \times 2 2 4$ patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches.    The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components, with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel $I _ { x y } =$ $[ I _ { x y } ^ { R } , I _ { x y } ^ { G } , I _ { x y } ^ { B } ] ^ { T }$ we add the following quantity:    $$ \big [ { \bf p } _ { 1 } , { \bf p } _ { 2 } , { \bf p } _ { 3 } \big ] \big [ \alpha _ { 1 } \lambda _ { 1 } , \alpha _ { 2 } \lambda _ { 2 } , \alpha _ { 3 } \lambda _ { 3 } \big ] ^ { T } $$    where $\mathbf { p } _ { i }$ and $\lambda _ { i }$ are $i$ th eigenvector and eigenvalue of the $3 \times 3$ covariance matrix of RGB pixel values, respectively, and $\alpha _ { i }$ is the aforementioned random variable. Each $\alpha _ { i }$ is drawn only once for all the pixels of a particular training image until that image is used for training again, at which point it is re-drawn. This scheme approximately captures an important property of natural images, namely, that object identity is invariant to changes in the intensity and color of the illumination. This scheme reduces the top-1 error rate by over $1 \%$ .    # 4.2 Dropout    Combining the predictions of many different models is a very successful way to reduce test errors [1, 3], but it appears to be too expensive for big neural networks that already take several days to train. There is, however, a very efficient version of model combination that only costs about a factor of two during training. The recently-introduced technique, called “dropout” [10], consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in backpropagation. So every time an input is presented, the neural network samples a different architecture, but all these architectures share weights. This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric mean of the predictive distributions produced by the exponentially-many dropout networks.    We use dropout in the first two fully-connected layers of Figure 2. Without dropout, our network exhibits substantial overfitting. Dropout roughly doubles the number of iterations required to converge.    # 5 Details of learning    We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. We found that this small amount of weight decay was important for the model to learn. In other words, weight decay here is not merely a regularizer: it reduces the model’s training error. The update rule for weight $w$ was    $$ \begin{array} { r c l } { { v _ { i + 1 } } } & { { : = } } & { { 0 . 9 \cdot v _ { i } - 0 . 0 0 0 5 \cdot \epsilon \cdot w _ { i } - \epsilon \cdot \left. \frac { \partial L } { \partial w } \big | _ { w _ { i } } \right. _ { D _ { i } } } } \\ { { w _ { i + 1 } } } & { { : = } } & { { w _ { i } + v _ { i + 1 } } } \end{array} $$    ![](images/aaadf74b496eeacac269d7bde735834da5d714894530fe01fa38484d6d587d03.jpg)   Figure 3: 96 convolutional kernels of size $1 1 \times 1 1 \times 3$ learned by the first convolutional layer on the $2 2 4 \times 2 2 4 \times 3$ input images. The top 48 kernels were learned on GPU 1 while the bottom 48 kernels were learned on GPU 2. See Section 6.1 for details.    where $i$ is the iteration index, $v$ is the momentum variable, $\epsilon$ is the learning rate, and $\begin{array} { r l r } {  {  \frac { \partial L } { \partial w } \Big | _ { w _ { i } }  _ { D _ { i } } } } \end{array}$ is the average over the $i$ th batch $D _ { i }$ of the derivative of the objective with respect to $w$ , evaluated at $w _ { i }$ .    We initialized the weights in each layer from a zero-mean Gaussian distribution with standard deviation 0.01. We initialized the neuron biases in the second, fourth, and fifth convolutional layers, as well as in the fully-connected hidden layers, with the constant 1. This initialization accelerates the early stages of learning by providing the ReLUs with positive inputs. We initialized the neuron biases in the remaining layers with the constant 0.    We used an equal learning rate for all layers, which we adjusted manually throughout training. The heuristic which we followed was to divide the learning rate by 10 when the validation error rate stopped improving with the current learning rate. The learning rate was initialized at 0.01 and reduced three times prior to termination. We trained the network for roughly 90 cycles through the training set of 1.2 million images, which took five to six days on two NVIDIA GTX 580 3GB GPUs.    # 6 Results    Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5 test set error rates of $3 7 . 5 \%$ and $1 7 . 0 \% ^ { 5 }$ . The best performance achieved during the ILSVRC2010 competition was $4 7 . 1 \%$ and $2 8 . 2 \%$ with an approach that averages the predictions produced from six sparse-coding models trained on different features [2], and since then the best published results are $4 5 . 7 \%$ and $2 5 . 7 \%$ with an approach that averages the predictions of two classifiers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features [24].    We also entered our model in the ILSVRC-2012 competition and report our results in Table 2. Since the ILSVRC-2012 test set labels are not publicly available, we cannot report test error rates for all the models that we tried. In the remainder of this paragraph, we use validation and test error rates interchangeably because in our experience they do not differ by more than $0 . 1 \%$ (see Table 2). The CNN described in this paper achieves a top-5 error rate of $1 8 . 2 \%$ . Averaging the predictions    <html><body><table><tr><td>Model</td><td>Top-1</td><td>Top-5</td></tr><tr><td>Sparse coding[2]</td><td>47.1%</td><td>28.2%</td></tr><tr><td>SIFT+FVs[24]</td><td>45.7%</td><td>25.7%</td></tr><tr><td>CNN</td><td>37.5%</td><td>17.0%</td></tr></table></body></html>    Table 1: Comparison of results on ILSVRC2010 test set. In italics are best results achieved by others.    of five similar CNNs gives an error rate of $1 6 . 4 \%$ . Training one CNN, with an extra sixth convolutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release (15M images, 22K categories), and then “fine-tuning” it on ILSVRC-2012 gives an error rate of $1 6 . 6 \%$ . Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 release with the aforementioned five CNNs gives an error rate of $1 5 . 3 \%$ . The second-best contest entry achieved an error rate of $2 6 . 2 \%$ with an approach that averages the predictions of several classifiers trained on FVs computed from different types of densely-sampled features [7].    Finally, we also report our error rates on the Fall 2009 version of ImageNet with 10,184 categories and 8.9 million images. On this dataset we follow the convention in the literature of using half of the images for training and half for testing. Since there is no established test set, our split necessarily differs from the splits used by previous authors, but this does not affect the results appreciably. Our top-1 and top-5 error rates on this dataset are $67 . 4 \%$ and    Table 2: Comparison of error rates on ILSVRC-2012 validation and test sets. In italics are best results achieved by others. Models with an asterisk\* were “pre-trained” to classify the entire ImageNet 2011 Fall release. See Section 6 for details.      <html><body><table><tr><td>Model</td><td>Top-1 (val)</td><td>Top-5 (val)</td><td>Top-5 (test)</td></tr><tr><td>SIFT+FVs[7]</td><td>二</td><td>一</td><td>26.2%</td></tr><tr><td>1 CNN</td><td>40.7%</td><td>18.2%</td><td>二</td></tr><tr><td>5 CNNs</td><td>38.1%</td><td>16.4%</td><td>16.4%</td></tr><tr><td>1 CNN*</td><td>39.0%</td><td>16.6%</td><td>二</td></tr><tr><td>7 CNNs*</td><td>36.7%</td><td>15.4%</td><td>15.3%</td></tr></table></body></html>    $4 0 . 9 \%$ , attained by the net described above but with an additional, sixth convolutional layer over the last pooling layer. The best published results on this dataset are $78 . 1 \%$ and $6 0 . 9 \%$ [19].    # 6.1 Qualitative Evaluations    Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers. The network has learned a variety of frequency- and orientation-selective kernels, as well as various colored blobs. Notice the specialization exhibited by the two GPUs, a result of the restricted connectivity described in Section 3.5. The kernels on GPU 1 are largely color-agnostic, while the kernels on on GPU 2 are largely color-specific. This kind of specialization occurs during every run and is independent of any particular random weight initialization (modulo a renumbering of the GPUs).    ![](images/66491abbb6c923fad2c100a539ff8b47812a62b4315065acf330ec9877558f6d.jpg)   Figure 4: (Left) Eight ILSVRC-2010 test images and the five labels considered most probable by our model. The correct label is written under each image, and the probability assigned to the correct label is also shown with a red bar (if it happens to be in the top 5). (Right) Five ILSVRC-2010 test images in the first column. The remaining columns show the six training images that produce feature vectors in the last hidden layer with the smallest Euclidean distance from the feature vector for the test image.    In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its top-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example, only other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry) there is genuine ambiguity about the intended focus of the photograph.    Another way to probe the network’s visual knowledge is to consider the feature activations induced by an image at the last, 4096-dimensional hidden layer. If two images produce feature activation vectors with a small Euclidean separation, we can say that the higher levels of the neural network consider them to be similar. Figure 4 shows five images from the test set and the six images from the training set that are most similar to each of them according to this measure. Notice that at the pixel level, the retrieved training images are generally not close in L2 to the query images in the first column. For example, the retrieved dogs and elephants appear in a variety of poses. We present the results for many more test images in the supplementary material.    Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vectors is inefficient, but it could be made efficient by training an auto-encoder to compress these vectors to short binary codes. This should produce a much better image retrieval method than applying autoencoders to the raw pixels [14], which does not make use of image labels and hence has a tendency to retrieve images with similar patterns of edges, whether or not they are semantically similar.    # 7 Discussion    Our results show that a large, deep convolutional neural network is capable of achieving recordbreaking results on a highly challenging dataset using purely supervised learning. It is notable that our network’s performance degrades if a single convolutional layer is removed. For example, removing any of the middle layers results in a loss of about $2 \%$ for the top-1 performance of the network. So the depth really is important for achieving our results.    To simplify our experiments, we did not use any unsupervised pre-training even though we expect that it will help, especially if we obtain enough computational power to significantly increase the size of the network without obtaining a corresponding increase in the amount of labeled data. Thus far, our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infero-temporal pathway of the human visual system. Ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing or far less obvious in static images.    # References    9(2):75–79, 2007.    [2] A. Berg, J. Deng, and L. Fei-Fei. Large scale visual recognition challenge 2010. www.imagenet.org/challenges. 2010.    [3] L. Breiman. Random forests. Machine learning, 45(1):5–32, 2001.    [4] D. Cires¸an, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. Arxiv preprint arXiv:1202.2745, 2012.    [5] D.C. Cires¸an, U. Meier, J. Masci, L.M. Gambardella, and J. Schmidhuber. High-performance neural networks for visual object classification. Arxiv preprint arXiv:1102.0183, 2011.    [6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.    [7] J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei. ILSVRC-2012, 2012. URL http://www.image-net.org/challenges/LSVRC/2012/.    [8] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Image Understanding, 106(1):59–70, 2007.    [9] G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. Technical Report 7694, California Institute of Technology, 2007. URL http://authors.library.caltech.edu/7694.    [10] G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R.R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.    [11] K. Jarrett, K. Kavukcuoglu, M. A. Ranzato, and Y. LeCun. What is the best multi-stage architecture for object recognition? In International Conference on Computer Vision, pages 2146–2153. IEEE, 2009.    [12] A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto, 2009.    [13] A. Krizhevsky. Convolutional deep belief networks on cifar-10. Unpublished manuscript, 2010.    [14] A. Krizhevsky and G.E. Hinton. Using very deep autoencoders for content-based image retrieval. In ESANN, 2011.    [15] Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, et al. Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems, 1990.    [16] Y. LeCun, F.J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II–97. IEEE, 2004.    [17] Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in vision. In Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on, pages 253–256. IEEE, 2010.    [18] H. Lee, R. Grosse, R. Ranganath, and A.Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 609–616. ACM, 2009.    [19] T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Metric Learning for Large Scale Image Classification: Generalizing to New Classes at Near-Zero Cost. In ECCV - European Conference on Computer Vision, Florence, Italy, October 2012.    [20] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proc. 27th International Conference on Machine Learning, 2010.    [21] N. Pinto, D.D. Cox, and J.J. DiCarlo. Why is real-world visual object recognition hard? PLoS computational biology, 4(1):e27, 2008.    [22] N. Pinto, D. Doukhan, J.J. DiCarlo, and D.D. Cox. A high-throughput screening approach to discovering good forms of biologically inspired visual representation. PLoS computational biology, 5(11):e1000579, 2009.    [23] B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman. Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77(1):157–173, 2008.    [24] J. Sánchez and F. Perronnin. High-dimensional signature compression for large-scale image classification. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1665–1672. IEEE, 2011.    [25] P.Y. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In Proceedings of the Seventh International Conference on Document Analysis and Recognition, volume 2, pages 958–962, 2003.    [26] S.C. Turaga, J.F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman, W. Denk, and H.S. Seung. Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Computation, 22(2):511–538, 2010.
-设计成一个完整的页面，不要翻页。

 设计一个支持 i18n 的网站，内容 # ImageNet Classification with Deep Convolutional Neural Networks    Alex Krizhevsky University of Toronto [email protected]    Ilya Sutskever University of Toronto [email protected]    Geoffrey E. Hinton University of Toronto [email protected]    # Abstract    We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of $3 7 . 5 \%$ and $1 7 . 0 \%$ which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of $1 5 . 3 \%$ , compared to $2 6 . 2 \%$ achieved by the second-best entry.    # 1 Introduction    Current approaches to object recognition make essential use of machine learning methods. To improve their performance, we can collect larger datasets, learn more powerful models, and use better techniques for preventing overfitting. Until recently, datasets of labeled images were relatively small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9], and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of this size, especially if they are augmented with label-preserving transformations. For example, the currentbest error rate on the MNIST digit-recognition task $( < 0 . 3 \% )$ approaches human performance [4]. But objects in realistic settings exhibit considerable variability, so to learn to recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of small image datasets have been widely recognized (e.g., Pinto et al. [21]), but it has only recently become possible to collect labeled datasets with millions of images. The new larger datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in over 22,000 categories.    To learn about thousands of objects from millions of images, we need a model with a large learning capacity. However, the immense complexity of the object recognition task means that this problem cannot be specified even by a dataset as large as ImageNet, so our model should also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.    Despite the attractive qualities of CNNs, and despite the relative efficiency of their local architecture, they have still been prohibitively expensive to apply in large scale to high-resolution images. Luckily, current GPUs, paired with a highly-optimized implementation of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs, and recent datasets such as ImageNet contain enough labeled examples to train such models without severe overfitting.    The specific contributions of this paper are as follows: we trained one of the largest convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010 and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the other operations inherent in training convolutional neural networks, which we make available publicly1. Our network contains a number of new and unusual features which improve its performance and reduce its training time, which are detailed in Section 3. The size of our network made overfitting a significant problem, even with 1.2 million labeled training examples, so we used several effective techniques for preventing overfitting, which are described in Section 4. Our final network contains five convolutional and three fully-connected layers, and this depth seems to be important: we found that removing any convolutional layer (each of which contains no more than $1 \%$ of the model’s parameters) resulted in inferior performance.    In the end, the network’s size is limited mainly by the amount of memory available on current GPUs and by the amount of training time that we are willing to tolerate. Our network takes between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest that our results can be improved simply by waiting for faster GPUs and bigger datasets to become available.    # 2 The Dataset    ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly 22,000 categories. The images were collected from the web and labeled by human labelers using Amazon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training images, 50,000 validation images, and 150,000 testing images.    ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this is the version on which we performed most of our experiments. Since we also entered our model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the model.    ImageNet consists of variable-resolution images, while our system requires a constant input dimensionality. Therefore, we down-sampled the images to a fixed resolution of $2 5 6 \times 2 5 6$ . Given a rectangular image, we first rescaled the image such that the shorter side was of length 256, and then cropped out the central $2 5 6 \times 2 5 6$ patch from the resulting image. We did not pre-process the images in any other way, except for subtracting the mean activity over the training set from each pixel. So we trained our network on the (centered) raw RGB values of the pixels.    # 3 The Architecture    The architecture of our network is summarized in Figure 2. It contains eight learned layers — five convolutional and three fully-connected. Below, we describe some of the novel or unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our estimation of their importance, with the most important first.    # 3.1 ReLU Nonlinearity    The standard way to model a neuron’s output $f$ as a function of its input $x$ is with $f ( x ) \ = \ \operatorname { t a n h } ( x )$ or $f ( x ) = ( 1 + e ^ { - x } ) ^ { - 1 }$ . In terms of training time with gradient descent, these saturating nonlinearities are much slower than the non-saturating nonlinearity $f ( x ) = \operatorname* { m a x } ( 0 , x )$ . Following Nair and Hinton [20], we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs). Deep convolutional neural networks with ReLUs train several times faster than their equivalents with tanh units. This is demonstrated in Figure 1, which shows the number of iterations required to reach $2 5 \%$ training error on the CIFAR-10 dataset for a particular four-layer convolutional network. This plot shows that we would not have been able to experiment with such large neural networks for this work if we had used traditional saturating neuron models.    We are not the first to consider alternatives to traditional neuron models in CNNs. For example, Jarrett et al. [11] claim that the nonlinearity $f ( x ) \stackrel { \textstyle - } { = } \left| \operatorname { t a n h } ( x ) \right|$ works particularly well with their type of contrast normalization followed by local average pooling on the Caltech-101 dataset. However, on this dataset the primary concern is preventing overfitting, so the effect they are observing is different from the accelerated ability to fit the training set which we report when using ReLUs. Faster learning has a great influence on the performance of large models trained on large datasets.    ![](images/4ef7862e315766034c533cfa2473998dbd0f3ef1c4bf94d4602c6bacef7b5bda.jpg)   Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a $2 5 \%$ training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons (dashed line). The learning rates for each network were chosen independently to make training as fast as possible. No regularization of any kind was employed. The magnitude of the effect demonstrated here varies with network architecture, but networks with ReLUs consistently learn several times faster than equivalents with saturating neurons.    # 3.2 Training on Multiple GPUs    A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it. It turns out that 1.2 million training examples are enough to train networks which are too big to fit on one GPU. Therefore we spread the net across two GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able to read from and write to one another’s memory directly, without going through host machine memory. The parallelization scheme that we employ essentially puts half of the kernels (or neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers. This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2. However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this allows us to precisely tune the amount of communication until it is an acceptable fraction of the amount of computation.    The resultant architecture is somewhat similar to that of the “columnar” CNN employed by Cire¸san et al. [5], except that our columns are not independent (see Figure 2). This scheme reduces our top-1 and top-5 error rates by $1 . 7 \%$ and $1 . 2 \%$ , respectively, as compared with a net with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net takes slightly less time to train than the one-GPU net2.    # 3.3 Local Response Normalization    ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by $a _ { x , y } ^ { i }$ the activity of a neuron computed by applying kernel $i$ at position $( x , y )$ and then applying the ReLU nonlinearity, the response-normalized activity $b _ { x , y } ^ { i }$ is given by the expression    $$ b _ { x , y } ^ { i } = a _ { x , y } ^ { i } / \left( k + \alpha \sum _ { j = \operatorname* { m a x } ( 0 , i - n / 2 ) } ^ { \operatorname* { m i n } ( N - 1 , i + n / 2 ) } ( a _ { x , y } ^ { j } ) ^ { 2 } \right) ^ { \beta } $$    where the sum runs over $n$ “adjacent” kernel maps at the same spatial position, and $N$ is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants $k , n , \alpha$ , and $\beta$ are hyper-parameters whose values are determined using a validation set; we used $k = 2$ , $n = 5$ , $\alpha = 1 0 ^ { - 4 }$ , and $\beta = 0 . 7 5$ . We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5).    This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by $1 . 4 \%$ and $1 . 2 \%$ , respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a $13 \%$ test error rate without normalization and $11 \%$ with normalization3.    # 3.4 Overlapping Pooling    Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of a grid of pooling units spaced $s$ pixels apart, each summarizing a neighborhood of size $z \times z$ centered at the location of the pooling unit. If we set $s = z$ , we obtain traditional local pooling as commonly employed in CNNs. If we set $s < z$ , we obtain overlapping pooling. This is what we use throughout our network, with $s = 2$ and $z = 3$ . This scheme reduces the top-1 and top-5 error rates by $0 . 4 \%$ and $0 . 3 \%$ , respectively, as compared with the non-overlapping scheme $s = 2 , z = 2$ , which produces output of equivalent dimensions. We generally observe during training that models with overlapping pooling find it slightly more difficult to overfit.    # 3.5 Overall Architecture    Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the net contains eight layers with weights; the first five are convolutional and the remaining three are fullyconnected. The output of the last fully-connected layer is fed to a 1000-way softmax which produces a distribution over the 1000 class labels. Our network maximizes the multinomial logistic regression objective, which is equivalent to maximizing the average across training cases of the log-probability of the correct label under the prediction distribution.    The kernels of the second, fourth, and fifth convolutional layers are connected only to those kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of the third convolutional layer are connected to all kernel maps in the second layer. The neurons in the fullyconnected layers are connected to all neurons in the previous layer. Response-normalization layers follow the first and second convolutional layers. Max-pooling layers, of the kind described in Section 3.4, follow both response-normalization layers as well as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every convolutional and fully-connected layer.    The first convolutional layer filters the $2 2 4 \times 2 2 4 \times 3$ input image with 96 kernels of size $1 1 \times 1 1 \times 3$ with a stride of 4 pixels (this is the distance between the receptive field centers of neighboring neurons in a kernel map). The second convolutional layer takes as input the (response-normalized and pooled) output of the first convolutional layer and filters it with 256 kernels of size $5 \times 5 \times 4 8$ . The third, fourth, and fifth convolutional layers are connected to one another without any intervening pooling or normalization layers. The third convolutional layer has 384 kernels of size $3 \times 3 \times$ 256 connected to the (normalized, pooled) outputs of the second convolutional layer. The fourth convolutional layer has 384 kernels of size $3 \times 3 \times 1 9 2$ , and the fifth convolutional layer has 256 kernels of size $3 \times 3 \times 1 9 2$ . The fully-connected layers have 4096 neurons each.    ![](images/c2c1e038483a692fb59e8b21254b08a0ad17b5535e38d56450018d8321cf78e0.jpg)   Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of responsibilities between the two GPUs. One GPU runs the layer-parts at the top of the figure while the other runs the layer-parts at the bottom. The GPUs communicate only at certain layers. The network’s input is 150,528-dimensional, and the number of neurons in the network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264– 4096–4096–1000.    # 4 Reducing Overfitting    Our neural network architecture has 60 million parameters. Although the 1000 classes of ILSVRC make each training example impose 10 bits of constraint on the mapping from image to label, this turns out to be insufficient to learn so many parameters without considerable overfitting. Below, we describe the two primary ways in which we combat overfitting.    # 4.1 Data Augmentation    The easiest and most common method to reduce overfitting on image data is to artificially enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two distinct forms of data augmentation, both of which allow transformed images to be produced from the original images with very little computation, so the transformed images do not need to be stored on disk. In our implementation, the transformed images are generated in Python code on the CPU while the GPU is training on the previous batch of images. So these data augmentation schemes are, in effect, computationally free.    The first form of data augmentation consists of generating image translations and horizontal reflections. We do this by extracting random $2 2 4 \times 2 2 4$ patches (and their horizontal reflections) from the $2 5 6 \times 2 5 6$ images and training our network on these extracted patches4. This increases the size of our training set by a factor of 2048, though the resulting training examples are, of course, highly interdependent. Without this scheme, our network suffers from substantial overfitting, which would have forced us to use much smaller networks. At test time, the network makes a prediction by extracting five $2 2 4 \times 2 2 4$ patches (the four corner patches and the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging the predictions made by the network’s softmax layer on the ten patches.    The second form of data augmentation consists of altering the intensities of the RGB channels in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the ImageNet training set. To each training image, we add multiples of the found principal components, with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each RGB image pixel $I _ { x y } =$ $[ I _ { x y } ^ { R } , I _ { x y } ^ { G } , I _ { x y } ^ { B } ] ^ { T }$ we add the following quantity:    $$ \big [ { \bf p } _ { 1 } , { \bf p } _ { 2 } , { \bf p } _ { 3 } \big ] \big [ \alpha _ { 1 } \lambda _ { 1 } , \alpha _ { 2 } \lambda _ { 2 } , \alpha _ { 3 } \lambda _ { 3 } \big ] ^ { T } $$    where $\mathbf { p } _ { i }$ and $\lambda _ { i }$ are $i$ th eigenvector and eigenvalue of the $3 \times 3$ covariance matrix of RGB pixel values, respectively, and $\alpha _ { i }$ is the aforementioned random variable. Each $\alpha _ { i }$ is drawn only once for all the pixels of a particular training image until that image is used for training again, at which point it is re-drawn. This scheme approximately captures an important property of natural images, namely, that object identity is invariant to changes in the intensity and color of the illumination. This scheme reduces the top-1 error rate by over $1 \%$ .    # 4.2 Dropout    Combining the predictions of many different models is a very successful way to reduce test errors [1, 3], but it appears to be too expensive for big neural networks that already take several days to train. There is, however, a very efficient version of model combination that only costs about a factor of two during training. The recently-introduced technique, called “dropout” [10], consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons which are “dropped out” in this way do not contribute to the forward pass and do not participate in backpropagation. So every time an input is presented, the neural network samples a different architecture, but all these architectures share weights. This technique reduces complex co-adaptations of neurons, since a neuron cannot rely on the presence of particular other neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with many different random subsets of the other neurons. At test time, we use all the neurons but multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric mean of the predictive distributions produced by the exponentially-many dropout networks.    We use dropout in the first two fully-connected layers of Figure 2. Without dropout, our network exhibits substantial overfitting. Dropout roughly doubles the number of iterations required to converge.    # 5 Details of learning    We trained our models using stochastic gradient descent with a batch size of 128 examples, momentum of 0.9, and weight decay of 0.0005. We found that this small amount of weight decay was important for the model to learn. In other words, weight decay here is not merely a regularizer: it reduces the model’s training error. The update rule for weight $w$ was    $$ \begin{array} { r c l } { { v _ { i + 1 } } } & { { : = } } & { { 0 . 9 \cdot v _ { i } - 0 . 0 0 0 5 \cdot \epsilon \cdot w _ { i } - \epsilon \cdot \left. \frac { \partial L } { \partial w } \big | _ { w _ { i } } \right. _ { D _ { i } } } } \\ { { w _ { i + 1 } } } & { { : = } } & { { w _ { i } + v _ { i + 1 } } } \end{array} $$    ![](images/aaadf74b496eeacac269d7bde735834da5d714894530fe01fa38484d6d587d03.jpg)   Figure 3: 96 convolutional kernels of size $1 1 \times 1 1 \times 3$ learned by the first convolutional layer on the $2 2 4 \times 2 2 4 \times 3$ input images. The top 48 kernels were learned on GPU 1 while the bottom 48 kernels were learned on GPU 2. See Section 6.1 for details.    where $i$ is the iteration index, $v$ is the momentum variable, $\epsilon$ is the learning rate, and $\begin{array} { r l r } {  {  \frac { \partial L } { \partial w } \Big | _ { w _ { i } }  _ { D _ { i } } } } \end{array}$ is the average over the $i$ th batch $D _ { i }$ of the derivative of the objective with respect to $w$ , evaluated at $w _ { i }$ .    We initialized the weights in each layer from a zero-mean Gaussian distribution with standard deviation 0.01. We initialized the neuron biases in the second, fourth, and fifth convolutional layers, as well as in the fully-connected hidden layers, with the constant 1. This initialization accelerates the early stages of learning by providing the ReLUs with positive inputs. We initialized the neuron biases in the remaining layers with the constant 0.    We used an equal learning rate for all layers, which we adjusted manually throughout training. The heuristic which we followed was to divide the learning rate by 10 when the validation error rate stopped improving with the current learning rate. The learning rate was initialized at 0.01 and reduced three times prior to termination. We trained the network for roughly 90 cycles through the training set of 1.2 million images, which took five to six days on two NVIDIA GTX 580 3GB GPUs.    # 6 Results    Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5 test set error rates of $3 7 . 5 \%$ and $1 7 . 0 \% ^ { 5 }$ . The best performance achieved during the ILSVRC2010 competition was $4 7 . 1 \%$ and $2 8 . 2 \%$ with an approach that averages the predictions produced from six sparse-coding models trained on different features [2], and since then the best published results are $4 5 . 7 \%$ and $2 5 . 7 \%$ with an approach that averages the predictions of two classifiers trained on Fisher Vectors (FVs) computed from two types of densely-sampled features [24].    We also entered our model in the ILSVRC-2012 competition and report our results in Table 2. Since the ILSVRC-2012 test set labels are not publicly available, we cannot report test error rates for all the models that we tried. In the remainder of this paragraph, we use validation and test error rates interchangeably because in our experience they do not differ by more than $0 . 1 \%$ (see Table 2). The CNN described in this paper achieves a top-5 error rate of $1 8 . 2 \%$ . Averaging the predictions    <html><body><table><tr><td>Model</td><td>Top-1</td><td>Top-5</td></tr><tr><td>Sparse coding[2]</td><td>47.1%</td><td>28.2%</td></tr><tr><td>SIFT+FVs[24]</td><td>45.7%</td><td>25.7%</td></tr><tr><td>CNN</td><td>37.5%</td><td>17.0%</td></tr></table></body></html>    Table 1: Comparison of results on ILSVRC2010 test set. In italics are best results achieved by others.    of five similar CNNs gives an error rate of $1 6 . 4 \%$ . Training one CNN, with an extra sixth convolutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011 release (15M images, 22K categories), and then “fine-tuning” it on ILSVRC-2012 gives an error rate of $1 6 . 6 \%$ . Averaging the predictions of two CNNs that were pre-trained on the entire Fall 2011 release with the aforementioned five CNNs gives an error rate of $1 5 . 3 \%$ . The second-best contest entry achieved an error rate of $2 6 . 2 \%$ with an approach that averages the predictions of several classifiers trained on FVs computed from different types of densely-sampled features [7].    Finally, we also report our error rates on the Fall 2009 version of ImageNet with 10,184 categories and 8.9 million images. On this dataset we follow the convention in the literature of using half of the images for training and half for testing. Since there is no established test set, our split necessarily differs from the splits used by previous authors, but this does not affect the results appreciably. Our top-1 and top-5 error rates on this dataset are $67 . 4 \%$ and    Table 2: Comparison of error rates on ILSVRC-2012 validation and test sets. In italics are best results achieved by others. Models with an asterisk\* were “pre-trained” to classify the entire ImageNet 2011 Fall release. See Section 6 for details.      <html><body><table><tr><td>Model</td><td>Top-1 (val)</td><td>Top-5 (val)</td><td>Top-5 (test)</td></tr><tr><td>SIFT+FVs[7]</td><td>二</td><td>一</td><td>26.2%</td></tr><tr><td>1 CNN</td><td>40.7%</td><td>18.2%</td><td>二</td></tr><tr><td>5 CNNs</td><td>38.1%</td><td>16.4%</td><td>16.4%</td></tr><tr><td>1 CNN*</td><td>39.0%</td><td>16.6%</td><td>二</td></tr><tr><td>7 CNNs*</td><td>36.7%</td><td>15.4%</td><td>15.3%</td></tr></table></body></html>    $4 0 . 9 \%$ , attained by the net described above but with an additional, sixth convolutional layer over the last pooling layer. The best published results on this dataset are $78 . 1 \%$ and $6 0 . 9 \%$ [19].    # 6.1 Qualitative Evaluations    Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers. The network has learned a variety of frequency- and orientation-selective kernels, as well as various colored blobs. Notice the specialization exhibited by the two GPUs, a result of the restricted connectivity described in Section 3.5. The kernels on GPU 1 are largely color-agnostic, while the kernels on on GPU 2 are largely color-specific. This kind of specialization occurs during every run and is independent of any particular random weight initialization (modulo a renumbering of the GPUs).    ![](images/66491abbb6c923fad2c100a539ff8b47812a62b4315065acf330ec9877558f6d.jpg)   Figure 4: (Left) Eight ILSVRC-2010 test images and the five labels considered most probable by our model. The correct label is written under each image, and the probability assigned to the correct label is also shown with a red bar (if it happens to be in the top 5). (Right) Five ILSVRC-2010 test images in the first column. The remaining columns show the six training images that produce feature vectors in the last hidden layer with the smallest Euclidean distance from the feature vector for the test image.    In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its top-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example, only other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry) there is genuine ambiguity about the intended focus of the photograph.    Another way to probe the network’s visual knowledge is to consider the feature activations induced by an image at the last, 4096-dimensional hidden layer. If two images produce feature activation vectors with a small Euclidean separation, we can say that the higher levels of the neural network consider them to be similar. Figure 4 shows five images from the test set and the six images from the training set that are most similar to each of them according to this measure. Notice that at the pixel level, the retrieved training images are generally not close in L2 to the query images in the first column. For example, the retrieved dogs and elephants appear in a variety of poses. We present the results for many more test images in the supplementary material.    Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vectors is inefficient, but it could be made efficient by training an auto-encoder to compress these vectors to short binary codes. This should produce a much better image retrieval method than applying autoencoders to the raw pixels [14], which does not make use of image labels and hence has a tendency to retrieve images with similar patterns of edges, whether or not they are semantically similar.    # 7 Discussion    Our results show that a large, deep convolutional neural network is capable of achieving recordbreaking results on a highly challenging dataset using purely supervised learning. It is notable that our network’s performance degrades if a single convolutional layer is removed. For example, removing any of the middle layers results in a loss of about $2 \%$ for the top-1 performance of the network. So the depth really is important for achieving our results.    To simplify our experiments, we did not use any unsupervised pre-training even though we expect that it will help, especially if we obtain enough computational power to significantly increase the size of the network without obtaining a corresponding increase in the amount of labeled data. Thus far, our results have improved as we have made our network larger and trained it longer but we still have many orders of magnitude to go in order to match the infero-temporal pathway of the human visual system. Ultimately we would like to use very large and deep convolutional nets on video sequences where the temporal structure provides very helpful information that is missing or far less obvious in static images.    # References    9(2):75–79, 2007.    [2] A. Berg, J. Deng, and L. Fei-Fei. Large scale visual recognition challenge 2010. www.imagenet.org/challenges. 2010.    [3] L. Breiman. Random forests. Machine learning, 45(1):5–32, 2001.    [4] D. Cires¸an, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. Arxiv preprint arXiv:1202.2745, 2012.    [5] D.C. Cires¸an, U. Meier, J. Masci, L.M. Gambardella, and J. Schmidhuber. High-performance neural networks for visual object classification. Arxiv preprint arXiv:1102.0183, 2011.    [6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.    [7] J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei. ILSVRC-2012, 2012. URL http://www.image-net.org/challenges/LSVRC/2012/.    [8] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Image Understanding, 106(1):59–70, 2007.    [9] G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. Technical Report 7694, California Institute of Technology, 2007. URL http://authors.library.caltech.edu/7694.    [10] G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R.R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.    [11] K. Jarrett, K. Kavukcuoglu, M. A. Ranzato, and Y. LeCun. What is the best multi-stage architecture for object recognition? In International Conference on Computer Vision, pages 2146–2153. IEEE, 2009.    [12] A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, Department of Computer Science, University of Toronto, 2009.    [13] A. Krizhevsky. Convolutional deep belief networks on cifar-10. Unpublished manuscript, 2010.    [14] A. Krizhevsky and G.E. Hinton. Using very deep autoencoders for content-based image retrieval. In ESANN, 2011.    [15] Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel, et al. Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems, 1990.    [16] Y. LeCun, F.J. Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II–97. IEEE, 2004.    [17] Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in vision. In Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on, pages 253–256. IEEE, 2010.    [18] H. Lee, R. Grosse, R. Ranganath, and A.Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference on Machine Learning, pages 609–616. ACM, 2009.    [19] T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Metric Learning for Large Scale Image Classification: Generalizing to New Classes at Near-Zero Cost. In ECCV - European Conference on Computer Vision, Florence, Italy, October 2012.    [20] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proc. 27th International Conference on Machine Learning, 2010.    [21] N. Pinto, D.D. Cox, and J.J. DiCarlo. Why is real-world visual object recognition hard? PLoS computational biology, 4(1):e27, 2008.    [22] N. Pinto, D. Doukhan, J.J. DiCarlo, and D.D. Cox. A high-throughput screening approach to discovering good forms of biologically inspired visual representation. PLoS computational biology, 5(11):e1000579, 2009.    [23] B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman. Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77(1):157–173, 2008.    [24] J. Sánchez and F. Perronnin. High-dimensional signature compression for large-scale image classification. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1665–1672. IEEE, 2011.    [25] P.Y. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In Proceedings of the Seventh International Conference on Document Analysis and Recognition, volume 2, pages 958–962, 2003.    [26] S.C. Turaga, J.F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman, W. Denk, and H.S. Seung. Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Computation, 22(2):511–538, 2010.
+设计成一个完整的页面，不要翻页。
+翻译全部网页而不是只是标题。