Embedding for NLP with Deep Learning

Word Embedding was introduced by Bengio in early 2000s, and interest in it really accelerated when Google presented Word2Vec in 2013.

This blog post has recent papers related to embedding for Natural Language Processing with Deep Learning. Example application areas embedding is used for in the papers include finance (stock market prediction), biomedical text analysis, part-of-speech tagging, sentiment analysis, pharmacology (drug adverse effects).

I recommend you to start with the paper: In Defense of Word Embedding for Generic Text Representation

Best regards,

Amund Tveit

  1. An approach to the use of word embeddings in an opinion classification task
    – Authors: F Enríquez, Ja Troyano, T López (2016)
  2. Learning Document Embeddings by Predicting N-grams for Sentiment Classification of Long Movie Reviews
    – Authors: B Li, T Liu, X Du, D Zhang, Z Zhao (2016)
  3. A Distributed Chinese Naive Bayes Classifier Based on Word Embedding
    – Authors: M Feng, G Wu (2016)
  4. An Empirical Study on Sentiment Classification of Chinese Review using Word Embedding
    – Authors: Y Lin, H Lei, J Wu, X Li (2015)
  5. Learning Bilingual Embedding Model for Cross-Language Sentiment Classification
    – Authors: X Tang, X Wan (2014)
  6. Training word embeddings for deep learning in biomedical text mining tasks
    – Authors: Z Jiang, L Li, D Huang, L Jin (2016)
  7. Learning Dense Convolutional Embeddings For Semantic Segmentation
    – Authors: Aw Harley, Kg Derpanis, I Kokkinos (2016)
  8. Creating Causal Embeddings for Question Answering with Minimal Supervision
    – Authors: R Sharp, M Surdeanu, P Jansen, P Clark, M Hammond (2016)
  9. Discriminative Phrase Embedding for Paraphrase Identification
    – Authors: W Yin, H Schütze (2016)
  10. Word embedding based retrieval model for similar cases recommendation
    – Authors: Y Zhao, J Wang, F Wang (2016)
  11. Learning Embeddings of API Tokens to Facilitate Deep Learning Based Program Processing
    – Authors: Y Lu, G Li, R Miao, Z Jin (2016)
  12. Locally Embedding Autoencoders: A Semi-Supervised Manifold Learning Approach of Document Representation
    – Authors: C Wei, S Luo, X Ma, H Ren, J Zhang, L Pan (2016)
  13. Deep Learning Architecture for Part-of-Speech Tagging with Word and Suffix Embeddings
    – Authors: A Popov (2016)
  14. Robobarista: Learning to Manipulate Novel Objects via Deep Multimodal Embedding
    – Authors: J Sung, Sh Jin, I Lenz, A Saxena (2016)
  15. Pruning subsequence search with attention-based embedding
    – Authors: C Raffel, Dpw Ellis (2016)
  16. Sentence Embedding Evaluation Using Pyramid Annotation
    – Authors: T Baumel, R Cohen, M Elhadad (2016)
  17. Deep Sentence Embedding Using the Long Short Term Memory Network: Analysis and Application to Information Retrieval
    – Authors: H Palangi, L Deng, Y Shen, J Gao, X He, J Chen… (2015)
  18. Feedback recurrent neural network-based embedded vector and its application in topic model
    – Authors: L Li, S Gan, X Yin (2016)
  19. Enhancing Sentence Relation Modeling with Auxiliary Character-level Embedding
    – Authors: P Li, H Huang (2016)
  20. Learning Word Meta-Embeddings by Using Ensembles of Embedding Sets
    – Authors: W Yin, H Schütze (2015)
  21. Syntax-Aware Multi-Sense Word Embeddings for Deep Compositional Models of Meaning
    – Authors: J Cheng, D Kartsaklis (2015)
  22. A Deep Embedding Model for Co-occurrence Learning
    – Authors: Y Shen, R Jin, J Chen, X He, J Gao, L Deng (2015)
  23. Jointly Modeling Embedding and Translation to Bridge Video and Language
    – Authors: Y Pan, T Mei, T Yao, H Li, Y Rui (2015)
  24. Learning semantic word embeddings based on ordinal knowledge constraints
    – Authors: Q Liu, H Jiang, S Wei, Zh Ling, Y Hu (2015)
  25. Boosting Named Entity Recognition with Neural Character Embeddings
    – Authors: C Dos Santos, V Guimaraes, Rj Niterói, R De Janeiro (2015)
  26. Evaluating word embeddings and a revised corpus for part-of-speech tagging in Portuguese
    – Authors: Er Fonseca, Jlg Rosa, Sm Aluísio (2015)
  27. Temporal Embedding in Convolutional Neural Networks for Robust Learning of Abstract Snippets
    – Authors: J Liu, K Zhao, B Kusy, J Wen, R Jurdak (2015)
  28. Learning Feature Hierarchies: A Layer-wise Tag-embedded Approach
    – Authors: Z Yuan, C Xu, J Sang, S Yan, M Hossain (2015)
  29. Multi-Source Bayesian Embeddings for Learning Social Knowledge Graphs
    – Authors: Z Yang, J Tang (2015)
  30. PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks
    – Authors: J Tang, M Qu, Q Mei (2015)
  31. Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features
    – Authors: A Nikfarjam, A Sarker, K O’Connor, R Ginn, G Gonzalez (2015)
  32. Projective Label Propagation by Label Embedding
    – Authors: Z Zhang, W Jiang, F Li, L Zhang, M Zhao, L Jia (2015)
  33. An Investigation of Neural Embeddings for Coreference Resolution
    – Authors: V Godbole, W Liu, R Togneri (2015)
  34. Deep Multimodal Embedding: Manipulating Novel Objects with Point-clouds, Language and Trajectories
    – Authors: J Sung, I Lenz, A Saxena (2015)
  35. AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes
    – Authors: S Rothe, H Schütze (2015)
  36. Deep Multilingual Correlation for Improved Word Embeddings
    – Authors: A Lu, W Wang, M Bansal, K Gimpel, K Livescu (2015)
  37. Leverage Financial News to Predict Stock Price Movements Using Word Embeddings and Deep Neural Networks
    – Authors: Y Peng, H Jiang (2015)
  38. The Impact of Structured Event Embeddings on Scalable Stock Forecasting Models
    – Authors: Jb Nascimento, M Cristo (2015)
  39. Representing Text for Joint Embedding of Text and Knowledge Bases
    – Authors: K Toutanova, D Chen, P Pantel, H Poon, P Choudhury… (2015)
  40. In Defense of Word Embedding for Generic Text Representation
    – Authors: G Lev, B Klein, L Wolf (2015)
  41. Learning Multi-Relational Semantics Using Neural-Embedding Models
    – Authors: B Yang, W Yih, X He, J Gao, L Deng (2014)
  42. Improving relation descriptor extraction with word embeddings and cluster features
    – Authors: T Liu, M Li (2014)
Continue Reading

Deep Learning for Natural Language Processing – ICLR 2017 Discoveries

Update: 2017-Feb-03 – launched new service – ai.amundtveit.com (navigation and search in papers). Try e.g. out its Natural Language Processing Page.

The 5th International Conference on Learning Representation (ICLR 2017) is coming to Toulon, France (April 24-26 2017), and there is large amount of Deep Learning papers submitted to the conference, looks like it will be a great event (see word cloud below for most frequent words used in submitted paper titles).


This blog post gives an overview of Natural Language Processing related papers submitted to ICLR 2017, see underneath for the list of papers. If you want to learn about Deep Learning with NLP check out Stanford’s CS224d: Deep Learning for Natural Language Processing

Best regards,

Amund Tveit


Character/Word/Sentence Representation

  1. Character-aware Attention Residual Network for Sentence Representation – Authors: Xin Zheng, Zhenzhou Wu
  2. Program Synthesis for Character Level Language Modeling – Authors: Pavol Bielik, Veselin Raychev, Martin Vechev
  3. Words or Characters? Fine-grained Gating for Reading Comprehension – Authors: Zhilin Yang, Bhuwan Dhingra, Ye Yuan, Junjie Hu, William W. Cohen, Ruslan Salakhutdinov
  4. Deep Character-Level Neural Machine Translation By Learning Morphology – Authors: Shenjian Zhao, Zhihua Zhang
  5. Opening the vocabulary of neural language models with character-level word representations – Authors: Matthieu Labeau, Alexandre Allauzen
  6. Unsupervised sentence representation learning with adversarial auto-encoder – Authors: Shuai Tang, Hailin Jin, Chen Fang, Zhaowen Wang
  7. Offline Bilingual Word Vectors Without a Dictionary – Authors: Samuel L. Smith, David H. P. Turban, Nils Y. Hammerla, Steven Hamblin
  8. Learning Word-Like Units from Joint Audio-Visual Analylsis – Authors: David Harwath, James R. Glass
  9. Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling – Authors: Hakan Inan, Khashayar Khosravi, Richard Socher
  10. Sentence Ordering using Recurrent Neural Networks – Authors: Lajanugen Logeswaran, Honglak Lee, Dragomir Radev

Search/Question-Answer/Recommender Systems

  1. Learning to Query, Reason, and Answer Questions On Ambiguous Texts – Authors: Xiaoxiao Guo, Tim Klinger, Clemens Rosenbaum, Joseph P. Bigus, Murray Campbell, Ban Kawas, Kartik Talamadupula, Gerry Tesauro, Satinder Singh
  2. Group Sparse CNNs for Question Sentence Classification with Answer Sets – Authors: Mingbo Ma, Liang Huang, Bing Xiang, Bowen Zhou
  3. CONTENT2VEC: Specializing Joint Representations of Product Images and Text for the task of Product Recommendation – Authors: Thomas Nedelec, Elena Smirnova, Flavian Vasile
  4. Is a picture worth a thousand words? A Deep Multi-Modal Fusion Architecture for Product Classification in e-commerce – Authors: Tom Zahavy, Alessandro Magnani, Abhinandan Krishnan, Shie Mannor

Word/Sentence Embedding

  1. A Simple but Tough-to-Beat Baseline for Sentence Embeddings – Authors: Sanjeev Arora, Yingyu Liang, Tengyu Ma
  2. Investigating Different Context Types and Representations for Learning Word Embeddings – Authors: Bofang Li, Tao Liu, Zhe Zhao, Xiaoyong Du
  3. Multi-view Recurrent Neural Acoustic Word Embeddings – Authors: Wanjia He, Weiran Wang, Karen Livescu
  4. A Self-Attentive Sentence Embedding – Authors: Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, Yoshua Bengio
  5. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks – Authors: Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, Yoav Goldberg


  1. Neural Machine Translation with Latent Semantic of Image and Text – Authors: Joji Toyama, Masanori Misono, Masahiro Suzuki, Kotaro Nakayama, Yutaka Matsuo
  2. Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context – Authors: Shyam Upadhyay, Kai-Wei Chang, James Zhou, Matt Taddy, Adam Kalai
  3. Learning to Understand: Incorporating Local Contexts with Global Attention for Sentiment Classification – Authors: Zhigang Yuan, Yuting Hu, Yongfeng Huang
  4. Adaptive Feature Abstraction for Translating Video to Language – Authors: Yunchen Pu, Martin Renqiang Min, Zhe Gan, Lawrence Carin
  5. A Convolutional Encoder Model for Neural Machine Translation – Authors: Jonas Gehring, Michael Auli, David Grangier, Yann N. Dauphin
  6. Fuzzy paraphrases in learning word representations with a corpus and a lexicon – Authors: Yuanzhi Ke, Masafumi Hagiwara
  7. Iterative Refinement for Machine Translation – Authors: Roman Novak, Michael Auli, David Grangier
  8. Vocabulary Selection Strategies for Neural Machine Translation – Authors: Gurvan L’Hostis, David Grangier, Michael Auli

Language Models/Text Comprehension/Matching/Compression/Classification/++

  1. A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks – Authors: Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, Richard Socher
  2. Gated-Attention Readers for Text Comprehension – Authors: Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William W. Cohen, Ruslan Salakhutdinov
  3. A Compare-Aggregate Model for Matching Text Sequences – Authors: Shuohang Wang, Jing Jiang
  4. A Context-aware Attention Network for Interactive Question Answering – Authors: Huayu Li, Martin Renqiang Min, Yong Ge, Asim Kadav
  5. FastText.zip: Compressing text classification models – Authors: Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jegou, Tomas Mikolov
  6. Multi-Agent Cooperation and the Emergence of (Natural) Language – Authors: Angeliki Lazaridou, Alexander Peysakhovich, Marco Baroni
  7. Learning a Natural Language Interface with Neural Programmer – Authors: Arvind Neelakantan, Quoc V. Le, Martin Abadi, Andrew McCallum, Dario Amodei
  8. Learning similarity preserving representations with neural similarity and context encoders – Authors: Franziska Horn, Klaus-Robert Müller
  9. Adversarial Training Methods for Semi-Supervised Text Classification – Authors: Takeru Miyato, Andrew M. Dai, Ian Goodfellow
  10. Multi-Label Learning using Tensor Decomposition for Large Text Corpora – Authors: Sayantan Dasgupta


Sign up for Deep Learning newsletter!

Continue Reading