ISSN 2071-8594

Russian academy of sciences


Gennady Osipov

E. I. Gribkov, Yu. P. Yekhlakov Neural transition-based model for extraction and sentiment analysis of user opinions


Extraction and analysis of user opinions towards products and services is important task in research and applications of natural language processing methods. We frame this task as structured prediction task where each data instance represented by group of interdependent labels. To solve this task, we describe transition-based model that decomposes it to the prediction of transition sequence which incrementally build final structure. The proposed model uses deep neural networks as feature extractor for classifier that predicts next transition based on previous transitions and the parts of predicted structure. To evaluate quality of the proposed model, we conducted a series of experiments on user reviews texts from two sources: English reviews from Amazon and Russian reviews from AliExpress. The experiments show that our model performs equally or better than an alternative and suffer less accuracy drop from distributional shift.


machine learning, sentiment analysis, deep learning, user opinions.

PP. 99-110.

DOI 10.14357/20718594200209


1. Turney P.D. 2002. Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. 40th Annual Meeting on Association for Computational Lin-guistics Proceedings. Philadelphia. PA. 417–424.
2. Socher R., Perelygin A., Wu. J.Y., Chuang J., Manning C.D., Ng A. and Potts C. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP) Proceedings. Seattle, WA. 1631–1642.
3. Indurkhya N. and Damerau F.J. 2010. Handbook of Natural Language Processing. N.Y.: Chapman and Hall/CRC. 704 p.
4. Schouten K. and Frasincar F. 2016. Survey on Aspect-Level Sentiment Analysis. IEEE Transactions on Knowledge and Data Engineering. 28(3):813–830.
5. Klinger R. and Cimiano P. 2013. Joint and Pipeline Probabilistic Models for Fine-Grained Sentiment Analysis: Ex-tracting Aspects, Subjective Phrases and their Relations. IEEE 13th International Conference on Data Mining Workshops Proceedings. Dallas, TX. 937–944.
6. Klinger, R. and P. Cimiano. 2014. The USAGE review corpus for fine-grained, multi-lingual opinion analysis. 9th International Conference on Language Resources and Evaluation (LREC-2014) Proceedings. Reykjavik. 2211–2218.
7. Jebbara S. and Cimiano P. 2016. Aspect-Based Relational Sentiment Analysis Using a Stacked Neural Network Architecture. 22nd European Conference on Artificial Intelligence Proceedings. Hague. 1123–1131.
8. Hu M. and Liu B. 2004. Mining Opinion Features in Cus-tomer Reviews. 19th National Conference on Artificial Intelligence (AAAI 2004) Proceedings. San Jose. 755–760.
9. Kobayashi N., Iida R., Inui K. and Matsumoto Y. 2006. Opinion mining on the web by extracting subject-aspect-evaluation relations. AAAI Spring Symposium 2006: Computational Approaches to Analyzing Weblogs (AAAI-SS 2006) Proceedings. Palo Alto, CA. 86–91.
10. Laddha A. and Mukherjee A. 2016. Extracting Aspect Specific Opinion Expressions. 2016 Conference on Empirical Methods in Natural Language Processing Proceedings. Austin, TX. 627–637.
11. Alimova I., Tutubalina E. and Alferova J. 2017. A Machine Learning Approach to Classification of Drug Reviews in Russian. 2017 Ivannikov ISPRAS Open Conference (ISPRAS) Proceedings. Moscow. 64–69.
12. Miftahutdinov Z. Sh, Tutubalina E.V. and Tropsha A.E. 2017. Identifying disease-related expressions in reviews using conditional random fields. International Conference Dialogue-2017. Moscow.
13. Mikolov T., Sutskever I., Chen K., Corrado G. and Dean J. 2013. Distributed representations of words and phrases and their compositionality. 26th International Conference on Neural Information Processing System Proceedings. Lake Tahoe, NV. 2:3111–3119.
14. Grave E., Bojanowski P., Gupta P., Joulin A. and Mikolov T. 2018. Learning Word Vectors for 157 Languages. 11th International Conference on Language Resources and Evaluation (LREC 2018) Proceedings. Miyazaki. 3483-3487.
15. Yang Z., Yang D., Dyer C., He X., Smola A. and Hovy E. 2016. Hierarchical Attention Networks for Document Classification. 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Proceedings. San Diego, CA. 1480–1489.
16. Graves A. 2012. Sequence transduction with recurrent neural networks. arXiv:1211.3711 [cs.NE]. Available at: (accessed May 29, 2019).
17. Irsoy O. and Cardie C. 2014. Opinion mining with deep recurrent neural networks. 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) Proceedings. Doha. 720–728.
18. Liu P., Joty S. and Meng H. 2015. Fine-grained opinion mining with recurrent neural networks and word embeddings. 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP) Proceedings. Lis-bon. 1433–1443.
19. Dyer C. Kuncoro A., Ballesteros M. Recurrent Neural Network Grammars. 2016. Proceedings of the 15th Annu-al Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San Diego, CA, USA.199–209.
20. Kiperwasser E. 2016. Goldberg Y. Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations. Transactions of the Association for Computational Linguistics. 2016. Vol. 4. 313–327.
21. Lample G., Ballesteros M., Subramanian S. 2016. Neural Architectures for Named Entity Recognition. Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San Diego, CA, USA. 260–270.
22. Cho K., Merrienboer B. van, Bahdanau D. and Bengio Y. 2014. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. 8th Workshop on Syntax, Semantics and Structure in Statistical Translation Proceedings. Doha. 103–111.
23. Hochreiter S. and Schmidhuber J. 1997. Long short-term memory. Neural Computation. 9(8): 1735–1780.
24. Johansson R. and Moschitti A. 2010. Syntactic and semantic structure for opinion expression detection. 14th Conference on Computational Natural Language Learning Proceedings. Uppsala. 67–76.
25. Bethard S., Yu H., Thornton A., Hatzivassiloglou V. and Jurafsky D. 2006 Extracting opinion propositions and opinion holders using syntactic and lexical cues. In: Shanahan, J. G., Y. Qu, and J. Wiebe (eds.). Computing Attitude and Affect in Text: Theory and Applications. Dordrecht: Springer. 125-141.
26. Neubig, G., C. Dyer, Y. Goldberg [et al.]. 2017. DyNet: The Dynamic Neural Network Toolkit. arXiv:1701.03980 [stat.ML]. Available at: (accessed May 29, 2019).
27. Kingma D.P. and Ba J.L. 2014. Adam: A method for sto-chastic optimization. arXiv:1412.6980v9 [cs.LG]. Availa-ble at: (accessed May 29, 2019).
28. Srivastava N., Hinton G., Krizhevsky A., Sutskever I. and Salakhutdinov R. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research. 15:1929–1958.
29. Gal Y. and Ghahramani Z. 2016. A theoretically grounded application of dropout in recurrent neural networks. 30th International Conference on Neural Information Processing System Proceedings. Barcelona. 1027–1035.
30. Blitzer J., Dredze M. and Pereira F. 2007. Biographies Bollywood Boom-Boxes and Blenders: Domain Adapta-tion for Sentiment Classification. 45th Annual Meeting of the Association of Computational Linguistics Proceedings. Prague. 432–439.
31. Dozat T. and Manning C. D. 2017. Deep biaffine attention for neural dependency parsing. 5th International Conference on Learning Representations (ICLR) Proceedings. Toulon.
32. Kovriguina L., Shilin I., Shipilo A. and Putintseva A. 2017. Russian Tagging and Dependency Parsing Models for Stanford CoreNLP Natural Language Toolkit. In: Różewski, P. and C. Lange (eds.). Knowledge Engineering and Semantic Web. Communications in Computer and Information Science. Cham: Springer. 786:101-111.