Skip to main content
SpringerLink
Log in
Book cover

International Conference on Neural Information Processing

ICONIP 2016: Neural Information Processing pp 269–279Cite as

Acceleration of Word2vec Using GPUs

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 9948))

Abstract

Word2vec is a widely used word embedding toolkit which generates word vectors by training input corpus. Since word vector can represent an exponential number of word cluster and enables reasoning of words with simple algebraic operations, it has become a widely used representation for the subsequent NLP tasks. In this paper, we present an efficient parallelization of word2vec using GPUs that preserves the accuracy. With two K20 GPUs, the proposed acceleration technique achieves 1.7M words/sec, which corresponds to about 20× of speedup compared to a single-threaded CPU execution.

This work was supported by ICT R&D program of MSIP/IITP. [R0101-15-0054, WiseKB: Big data based self-evolving knowledge base and reasoning platform].

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. NIPS (2013)

    Google Scholar 

  2. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: ICLR Workshop (2013)

    Google Scholar 

  3. Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3, 1137–1155 (2003)

    MATH  Google Scholar 

  4. word2vec_cbow. https://github.com/ChenglongChen/word2vec_cbow

  5. word2vec-keras-in-gensim. https://github.com/niitsuma/word2vec-keras-in-gensim

  6. Meng, X., Bradley, J., Yavuz, B., Sparks, E., Venkataraman, S., Liu, D., Freeman, J., Tsai, D., Amde, M., Owen, S., et al.: MLlib: machine learning in apache spark. arXiv preprint arXiv:1505.06807 (2015)

  7. Huang, E., Socher, R., Manning, C., Ng, A.: Improving word representations via global context and multiple word prototypes. In: Association for Computational Linguistics, pp. 873–882 (2012)

    Google Scholar 

  8. Collobert, R., Weston, J.: A unified architecture for natural language processing: deep neural networks with multitask learning. In: International Conference on Machine Learning (2008)

    Google Scholar 

  9. Mnih, A., Hinton, G.: Three new graphical models for statistical language modelling. In: International Conference on Machine Learning (2007)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electrical and Computer Engineering, University of Seoul, Seoul, Republic of Korea

    Seulki Bae & Youngmin Yi

Authors
  1. Seulki Bae
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Youngmin Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Youngmin Yi .

Editor information

Editors and Affiliations

  1. The University of Tokyo, Tokyo, Japan

    Akira Hirose

  2. Kobe University, Kobe, Japan

    Seiichi Ozawa

  3. Okinawa Institute of Science and Technology Graduate University, Onna, Japan

    Kenji Doya

  4. Nara Institute of Science and Technology, Ikoma, Japan

    Kazushi Ikeda

  5. Kyungpook National University, Daegu, Korea (Republic of)

    Minho Lee

  6. Chinese Academy of Sciences, Beijing, China

    Derong Liu

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Bae, S., Yi, Y. (2016). Acceleration of Word2vec Using GPUs. In: Hirose, A., Ozawa, S., Doya, K., Ikeda, K., Lee, M., Liu, D. (eds) Neural Information Processing. ICONIP 2016. Lecture Notes in Computer Science(), vol 9948. Springer, Cham. https://doi.org/10.1007/978-3-319-46672-9_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-46672-9_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-46671-2

  • Online ISBN: 978-3-319-46672-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation