Deep Visual-Semantic Alignments for Generating Image Descriptions
https://cs.stanford.edu/people/karpathy/cvpr2015.pdf
Andrej Karpathy Li Fei-Fei Department of Computer Science, Stanford University {karpathy,feifeili}@cs.stanford.edu
Abstract We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.
ICLR2016
Generating Images from Captions with Attention
http://arxiv.org/abs/1511.02793
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov
(Submitted on 9 Nov 2015 (v1), last revised 29 Feb 2016 (this version, v2))
Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov
(Submitted on 9 Nov 2015 (v1), last revised 29 Feb 2016 (this version, v2))
Motivated by the recent progress in generative models, we introduce a model that generates images from natural language descriptions. The proposed model iteratively draws patches on a canvas, while attending to the relevant words in the description. After training on Microsoft COCO, we compare our model with several baseline generative models on image generation and retrieval tasks. We demonstrate that our model produces higher quality samples than other approaches and generates images with novel scene compositions corresponding to previously unseen captions in the dataset.
3 MODEL Our proposed model defines a generative process of images conditioned on captions. In particular, captions are represented as a sequence of consecutive words and images are represented as a sequence of patches drawn on a canvas ct over time t = 1, ..., T. The model can be viewed as a part of the sequence-to-sequence framework (Sutskever et al., 2014; Cho et al., 2014; Srivastava et al., 2015).
3.1 LANGUAGE MODEL: THE BIDIRECTIONAL ATTENTION RNN
Show and Tell: A Neural Image Caption Generator
http://arxiv.org/pdf/1411.4555v2.pdf
Oriol Vinyals Google vinyals@google.com Alexander Toshev Google toshev@google.com Samy Bengio Google bengio@google.com Dumitru Erhan Google dumitru@google.com
Abstract
Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art
AlexNet
http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky University of Toronto kriz@cs.utoronto.ca Ilya Sutskever University of Toronto ilya@cs.utoronto.ca Geoffrey E. Hinton University of Toronto hinton@cs.utoronto.ca
Abstract
We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.
AlexNet - GitHub
SegNet
Alex Kendall, Vijay Badrinarayanan, Roberto Cipollahttp://mi.eng.cam.ac.uk/projects/segnet/
Caffe
http://caffe.berkeleyvision.org/
Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by the Berkeley Vision and Learning Center (BVLC) and by community contributors.Yangqing Jia created the project during his PhD at UC Berkeley. Caffe is released under the BSD 2-Clause license.
Check out our web image classification demo!
Why Caffe?
Expressive architecture encourages application and innovation. Models and optimization are defined by configuration without hard-coding. Switch between CPU and GPU by setting a single flag to train on a GPU machine then deploy to commodity clusters or mobile devices.
Extensible code fosters active development. In Caffe’s first year, it has been forked by over 1,000 developers and had many significant changes contributed back. Thanks to these contributors the framework tracks the state-of-the-art in both code and models.
Speed makes Caffe perfect for research experiments and industry deployment. Caffe can processover 60M images per day with a single NVIDIA K40 GPU*. That’s 1 ms/image for inference and 4 ms/image for learning. We believe that Caffe is the fastest convnet implementation available.
Community: Caffe already powers academic research projects, startup prototypes, and even large-scale industrial applications in vision, speech, and multimedia. Join our community of brewers on the caffe-users group and Github.
* With the ILSVRC2012-winning SuperVision model and caching IO. Consult performance details.
Computer Vision Open Source Code
OpenCV
http://docs.opencv.org/
Java API
http://docs.opencv.org/java/
http://opencv.org/opencv-java-api.html
API
http://docs.opencv.org/modules/refman.html
Downloads
http://opencv.org/downloads.html
Installation
http://docs.opencv.org/doc/tutorials/introduction/table_of_content_introduction/table_of_content_introduction.html
Tutorials
http://docs.opencv.org/doc/tutorials/tutorials.html
http://docs.opencv.org/2.4.4-beta/doc/tutorials/introduction/desktop_java/java_dev_intro.html
JavaCV
https://code.google.com/p/javacv/
Silicon Valley Computer Vision Meetup
http://www.meetup.com/Silicon-Valley-Computer-Vision/events/176686442/
John Brewer on github
https://github.com/jeradesign
https://github.com/jeradesign/spot-it-challenge
Signal & Image Processing : An International Journal (SIPIJ) Vol.3, No.5, October 2012
DOI : 10.5121/sipij.2012.3503 29
AN AUTOMATIC ALGORITHM FOR OBJECT
RECOGNITION AND DETECTION BASED ON ASIFT
KEYPOINTS
Reza Oji
Department of Computer Engineering and IT, Shiraz University
Shiraz, Iran
oji.reza@gmail.com
http://arxiv.org/ftp/arxiv/papers/1211/1211.5829.pdf
http://arxiv.org/abs/1510.00149
Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
Song Han, Huizi Mao, William J. Dally
(Submitted on 1 Oct 2015 (v1), last revised 15 Feb 2016 (this version, v5))
Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources. To address this limitation, we introduce "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy. Our method first prunes the network by learning only the important connections. Next, we quantize the weights to enforce weight sharing, finally, we apply Huffman coding. After the first two steps we retrain the network to fine tune the remaining connections and the quantized centroids. Pruning, reduces the number of connections by 9x to 13x; Quantization then reduces the number of bits that represent each connection from 32 to 5. On the ImageNet dataset, our method reduced the storage required by AlexNet by 35x, from 240MB to 6.9MB, without loss of accuracy. Our method reduced the size of VGG-16 by 49x from 552MB to 11.3MB, again with no loss of accuracy. This allows fitting the model into on-chip SRAM cache rather than off-chip DRAM memory. Our compression method also facilitates the use of complex neural networks in mobile applications where application size and download bandwidth are constrained. Benchmarked on CPU, GPU and mobile GPU, compressed network has 3x to 4x layerwise speedup and 3x to 7x better energy efficiency.
FractalNet: Ultra-Deep Neural Networks without Residuals
3 MODEL Our proposed model defines a generative process of images conditioned on captions. In particular, captions are represented as a sequence of consecutive words and images are represented as a sequence of patches drawn on a canvas ct over time t = 1, ..., T. The model can be viewed as a part of the sequence-to-sequence framework (Sutskever et al., 2014; Cho et al., 2014; Srivastava et al., 2015).
3.1 LANGUAGE MODEL: THE BIDIRECTIONAL ATTENTION RNN
Show and Tell: A Neural Image Caption Generator
http://arxiv.org/pdf/1411.4555v2.pdf
Oriol Vinyals Google vinyals@google.com Alexander Toshev Google toshev@google.com Samy Bengio Google bengio@google.com Dumitru Erhan Google dumitru@google.com
Abstract
Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art
AlexNet
http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky University of Toronto kriz@cs.utoronto.ca Ilya Sutskever University of Toronto ilya@cs.utoronto.ca Geoffrey E. Hinton University of Toronto hinton@cs.utoronto.ca
Abstract
We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called “dropout” that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry.
AlexNet - GitHub
This model is a replication of the model described in the AlexNet publication.
SegNet
Alex Kendall, Vijay Badrinarayanan, Roberto Cipollahttp://mi.eng.cam.ac.uk/projects/segnet/
Caffe
http://caffe.berkeleyvision.org/
Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by the Berkeley Vision and Learning Center (BVLC) and by community contributors.Yangqing Jia created the project during his PhD at UC Berkeley. Caffe is released under the BSD 2-Clause license.
Check out our web image classification demo!
Why Caffe?
Expressive architecture encourages application and innovation. Models and optimization are defined by configuration without hard-coding. Switch between CPU and GPU by setting a single flag to train on a GPU machine then deploy to commodity clusters or mobile devices.
Extensible code fosters active development. In Caffe’s first year, it has been forked by over 1,000 developers and had many significant changes contributed back. Thanks to these contributors the framework tracks the state-of-the-art in both code and models.
Speed makes Caffe perfect for research experiments and industry deployment. Caffe can processover 60M images per day with a single NVIDIA K40 GPU*. That’s 1 ms/image for inference and 4 ms/image for learning. We believe that Caffe is the fastest convnet implementation available.
Community: Caffe already powers academic research projects, startup prototypes, and even large-scale industrial applications in vision, speech, and multimedia. Join our community of brewers on the caffe-users group and Github.
* With the ILSVRC2012-winning SuperVision model and caching IO. Consult performance details.
Computer Vision Open Source Code
OpenCV
http://docs.opencv.org/
Java API
http://docs.opencv.org/java/
http://opencv.org/opencv-java-api.html
API
http://docs.opencv.org/modules/refman.html
Downloads
http://opencv.org/downloads.html
Installation
http://docs.opencv.org/doc/tutorials/introduction/table_of_content_introduction/table_of_content_introduction.html
Tutorials
http://docs.opencv.org/doc/tutorials/tutorials.html
http://docs.opencv.org/2.4.4-beta/doc/tutorials/introduction/desktop_java/java_dev_intro.html
JavaCV
https://code.google.com/p/javacv/
Silicon Valley Computer Vision Meetup
http://www.meetup.com/Silicon-Valley-Computer-Vision/events/176686442/
John Brewer on github
https://github.com/jeradesign
https://github.com/jeradesign/spot-it-challenge
Signal & Image Processing : An International Journal (SIPIJ) Vol.3, No.5, October 2012
DOI : 10.5121/sipij.2012.3503 29
AN AUTOMATIC ALGORITHM FOR OBJECT
RECOGNITION AND DETECTION BASED ON ASIFT
KEYPOINTS
Reza Oji
Department of Computer Engineering and IT, Shiraz University
Shiraz, Iran
oji.reza@gmail.com
http://arxiv.org/abs/1510.00149
Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
Song Han, Huizi Mao, William J. Dally
(Submitted on 1 Oct 2015 (v1), last revised 15 Feb 2016 (this version, v5))
Neural networks are both computationally intensive and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources. To address this limitation, we introduce "deep compression", a three stage pipeline: pruning, trained quantization and Huffman coding, that work together to reduce the storage requirement of neural networks by 35x to 49x without affecting their accuracy. Our method first prunes the network by learning only the important connections. Next, we quantize the weights to enforce weight sharing, finally, we apply Huffman coding. After the first two steps we retrain the network to fine tune the remaining connections and the quantized centroids. Pruning, reduces the number of connections by 9x to 13x; Quantization then reduces the number of bits that represent each connection from 32 to 5. On the ImageNet dataset, our method reduced the storage required by AlexNet by 35x, from 240MB to 6.9MB, without loss of accuracy. Our method reduced the size of VGG-16 by 49x from 552MB to 11.3MB, again with no loss of accuracy. This allows fitting the model into on-chip SRAM cache rather than off-chip DRAM memory. Our compression method also facilitates the use of complex neural networks in mobile applications where application size and download bandwidth are constrained. Benchmarked on CPU, GPU and mobile GPU, compressed network has 3x to 4x layerwise speedup and 3x to 7x better energy efficiency.
FractalNet: Ultra-Deep Neural Networks without Residuals
http://arxiv.org/abs/1605.07648Gustav Larsson, Michael Maire, Gregory Shakhnarovich
(Submitted on 24 May 2016)
We introduce a design strategy for neural network macro-architecture based on self-similarity. Repeated application of a single expansion rule generates an extremely deep network whose structural layout is precisely a truncated fractal. Such a network contains interacting subpaths of different lengths, but does not include any pass-through connections: every internal signal is transformed by a filter and nonlinearity before being seen by subsequent layers. This property stands in stark contrast to the current approach of explicitly structuring very deep networks so that training is a residual learning problem. Our experiments demonstrate that residual representation is not fundamental to the success of extremely deep convolutional neural networks. A fractal design achieves an error rate of 22.85% on CIFAR-100, matching the state-of-the-art held by residual networks.
Fractal networks exhibit intriguing properties beyond their high performance. They can be regarded as a computationally efficient implicit union of subnetworks of every depth. We explore consequences for training, touching upon connection with student-teacher behavior, and, most importantly, demonstrating the ability to extract high-performance fixed-depth subnetworks. To facilitate this latter task, we develop drop-path, a natural extension of dropout, to regularize co-adaptation of subpaths in fractal architectures. With such regularization, fractal networks exhibit an anytime property: shallow subnetworks provide a quick answer, while deeper subnetworks, with higher latency, provide a more accurate answer.
faception
https://docs.com/flavio-bernardotti/9946/faception
"Our personality is determined by our DNA and reflected in our face. It's kind of a signal."
Social and Life Science Research personalities modify or stay the same, according to the genes. Thus, experts believe that people's faces are reflections of their DNA's.
As of the moment, Faception has revealed 15 personalities, as reported by Rt.com. These personalities include extrovert, genius, academic researcher, professional poker players, bingo player, brand promoter, white collar offender, paedophile and terrorist.
Faception has allegedly been able to identify with success the nine terrorists who were the culprits of November terror incidents in Paris, as reported by The Daily Mail UK.
(Submitted on 24 May 2016)
We introduce a design strategy for neural network macro-architecture based on self-similarity. Repeated application of a single expansion rule generates an extremely deep network whose structural layout is precisely a truncated fractal. Such a network contains interacting subpaths of different lengths, but does not include any pass-through connections: every internal signal is transformed by a filter and nonlinearity before being seen by subsequent layers. This property stands in stark contrast to the current approach of explicitly structuring very deep networks so that training is a residual learning problem. Our experiments demonstrate that residual representation is not fundamental to the success of extremely deep convolutional neural networks. A fractal design achieves an error rate of 22.85% on CIFAR-100, matching the state-of-the-art held by residual networks.
Fractal networks exhibit intriguing properties beyond their high performance. They can be regarded as a computationally efficient implicit union of subnetworks of every depth. We explore consequences for training, touching upon connection with student-teacher behavior, and, most importantly, demonstrating the ability to extract high-performance fixed-depth subnetworks. To facilitate this latter task, we develop drop-path, a natural extension of dropout, to regularize co-adaptation of subpaths in fractal architectures. With such regularization, fractal networks exhibit an anytime property: shallow subnetworks provide a quick answer, while deeper subnetworks, with higher latency, provide a more accurate answer.
faception
https://docs.com/flavio-bernardotti/9946/faception
"Our personality is determined by our DNA and reflected in our face. It's kind of a signal."
Social and Life Science Research personalities modify or stay the same, according to the genes. Thus, experts believe that people's faces are reflections of their DNA's.
As of the moment, Faception has revealed 15 personalities, as reported by Rt.com. These personalities include extrovert, genius, academic researcher, professional poker players, bingo player, brand promoter, white collar offender, paedophile and terrorist.
Faception has allegedly been able to identify with success the nine terrorists who were the culprits of November terror incidents in Paris, as reported by The Daily Mail UK.
No comments:
Post a Comment