Skip to main content
SpringerLink
Log in

Bootstrapping polarity classifiers with rule-based classification

  • Original Paper
  • Published:
Language Resources and Evaluation Aims and scope Submit manuscript

Abstract

In this article, we examine the effectiveness of bootstrapping supervised machine-learning polarity classifiers with the help of a domain-independent rule-based classifier that relies on a lexical resource, i.e., a polarity lexicon and a set of linguistic rules. The benefit of this method is that though no labeled training data are required, it allows a classifier to capture in-domain knowledge by training a supervised classifier with in-domain features, such as bag of words, on instances labeled by a rule-based classifier. Thus, this approach can be considered as a simple and effective method for domain adaptation. Among the list of components of this approach, we investigate how important the quality of the rule-based classifier is and what features are useful for the supervised classifier. In particular, the former addresses the issue in how far linguistic modeling is relevant for this task. We not only examine how this method performs under more difficult settings in which classes are not balanced and mixed reviews are included in the data set but also compare how this linguistically-driven method relates to state-of-the-art statistical domain adaptation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. http://www.rateitall.com.

  2. Stemming may also negatively affect polar expressions (i.e., words containing a prior polarity, such as great and awful) by conflating expressions with different polarity types to the same stem, such as hopeful and hopeless to hope$. To estimate the impact of that problem, we stemmed the entries of the polarity lexicon we use in this work (i.e., a list of polar expressions along their respective polarity type) and counted the cases of those erroneously conflated expressions. Less than 1 % of the entries were affected; most critical suffixes, such as -less, were preserved by our stemmer (Porter 1980). On average, we measured only some slight improvement by using stemming (<1 % point).

  3. http://svn.ask.it.usyd.edu.au/trac/candc.

  4. By polarity shifters, we refer to all entries marked as genshifter, shiftneg, or shiftpos from that lexicon.

  5. Those rules are encoded by entries marked as notshifter.

  6. This classification of course requires a correct identification of the scope of the negation.

  7. We measure this by the average number of sentences within a document.

  8. http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words.

  9. http://svmlight.joachims.org.

  10. Even NormByPol reflects the length of the document as the longer a document is the more polar expressions it will (potentially) contain.

  11. Since those features will occur much more frequently than plain words throughout the documents, the inverted document frequency will always be very low which would consequently heavily downweight those features.

  12. We mean the class labels that are predicted by RB Weight and henceforth treated as actual class labels by the supervised learner.

  13. We also ensure that both classifiers predict the same default polarity if the rule-based classifier predicts a tie.

  14. Titov (2011) made his experiments on the data set available at: http://www.cs.jhu.edu/~mdredze/datasets/sentiment/processed_acl.tar.gz.

  15. Available at: http://www.cs.jhu.edu/~mdredze/datasets/sentiment/unprocessed.tar.gz.

  16. We even replicated the distribution of positive and negative instances in the unlabeled training data (note that the crawl does not contain any mixed reviews), even though those distributions were always close to uniform class distribution.

  17. Note that unigrams alone did not produce better results for either in-domain or out-of-domain classification.

  18. In Titov (2011) this model is referred to as Reg+.

  19. Of course, we only consider those source domains which are different to the target domain on which is tested.

  20. This is due to the fact that many items in Kitchen are electric devices whose reviews cover aspects that are similar to the ones discussed in the reviews from the Electronics domain, such as usability or malfunctioning components.

  21. Significance is based on a chi-square test using p < 0.05.

  22. Unfortunately, we cannot carry out any statistical significance tests on the results of this comparison, as there is no commonly established significance test to compare an averaged result (i.e., Average Domain Adaptation) with an individual result (i.e., SelfTr).

  23. Similar to Sect. 6, we refrain from doing statistical significance tests in this section since Ratio-20, Ratio-50, and Ratio-100 are averaged results over 10 samples whereas the remaining classifiers are single results and there is no commonly accepted way of comparing those different types of data (i.e., averaged results vs. single results).

  24. Example: if the actual class ratio is 80:20 and the estimated ratio is 90:10, then the deviation will be 10.

References

  • Akkaya, C., Wiebe, J., & Mihalcea, R. (2009). Subjectivity word sense disambiguation. In Proceedings of the conference on empirical methods in natural language processing (EMNLP) (pp. 190–199). Singapore.

  • Akkaya, C., Wiebe, J., Conrad, A., & Mihalcea, R. (2011). Improving the impact of subjectivity word sense disambiguation on contextual opinion analysis. In Proceedings of the conference on computational natural language learning (pp. 87–96). Portland, OR, USA.

  • Andreevskaia, A., & Bergler, S. (2008). When specialists and generalists work together: Overcoming domain dependence in sentiment tagging. In Proceedings of the annual meeting of the association for computational linguistics: Human language technologies (ACL/HLT) (pp. 290–298), Columbus, OH, USA.

  • Aue, A., & Gamon, M. (2005). Customizing sentiment classifiers to new domains: A case study. In Proceedings of recent advances in natural language processing (RANLP). Borovets, Bulgaria.

  • Baccianella, S., Esuli, A., & Sebastiani, F. (2010). SentiWordNet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In Proceedings of the conference on language resources and evaluation (LREC) (pp. 2200–2204). Valletta.

  • Balamurali, A., Joshi, A., & Bhattacharyya, P. (2011). Harnessing wordnet senses for supervised sentiment classification. In Proceedings of the conference on empirical methods in natural language processing (EMNLP) (pp. 1081–1091). Edinburgh, Scotland, UK.

  • Becker, I., & Aharonson, V. (2010). Last but definitely not least: On the role of the last sentence in automatic polarity-classification. In Proceedings of the annual meeting of the association for computational linguistics (ACL) (pp. 331–335). Uppsala, Sweden.

  • Beineke, P., Hastie, T., & Vaithyanathan, S. (2004). The sentimental factor: Improving review classification via human-provided information. In Proceedings of the annual meeting of the association for computational linguistics (ACL) (pp. 263–270). Barcelona, Spain.

  • Blitzer, J., McDonald, R., & Pereira, F. (2006). Domain adaptation with structural correspondence learning. In Proceedings of the conference on empirical methods in natural language processing (EMNLP) (pp. 120–128). Sydney, Australia.

  • Blitzer, J., Dredze, M., & Pereira, F. (2007). Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the annual meeting of the association for computational linguistics (ACL) (pp. 440–447). Prague, Czech Republic.

  • Choi, Y., & Cardie, C. (2008). Learning with compositional semantics as structural inference for subsentential sentiment analysis. In Proceedings of the conference on empirical methods in natural language processing (EMNLP) (pp. 793–801). Waikiki, HI, USA.

  • Council, I., McDonald, R., & Velikovich, L. (2010). What’s great and what’s not: Learning to classify the scope of negation for improved sentiment analysis. In Proceedings of the workshop on negation and speculation in natural language processing (pp. 51–59). Uppsala, Sweden.

  • Dasgupta, S., & Ng, V. (2009). Mine the easy, classify the hard: A semi-supervised approach to automatic sentiment classification. In Proceedings of the joint conference of the annual meeting of the association for computational linguistics and the international joint conference on natural language processing of the Asian federation of natural language processing (ACL/IJCNLP) (pp. 701–709). Suntec, Singapore.

  • Esuli, A., & Sebastiani, F. (2006). Derminining term subjectivity and term orientation for opinion mining. In Proceedings of the conference on European chapter of the association for computational linguistics (EACL) (pp. 193–200). Trento, Italy.

  • Esuli, A., & Sebastiani, F. (2007). PageRanking wordnet synsets: An application to opinion mining. In Proceedings of the annual meeting of the association for computational linguistics (ACL) (pp. 424–431). Prague, Czech Republic.

  • Hatzivassiloglou, V., & McKeown, K. R. (1997). Predicting the semantic orientation of adjectives. In Proceedings of the conference on european chapter of the association for computational linguistics (EACL) (pp. 174–181). Madrid, Spain.

  • Hatzivassiloglou, V., & Wiebe, J. (2000). Effects of adjective orientation and gradability on sentence subjectivity. In Proceedings of the international conference on computational linguistics (COLING), Saarbrücken (pp. 299–305). Germany.

  • Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. In Proceedings of the ACM SIGKDD international conference on knowledge discovery & data mining (KDD) (pp. 168–177). Seattle, WA, USA.

  • Huang, Y., & Lowe, H. J. (2007). A novel hybrid approach to automated negation detection in clinical radiology reports. Journal of the American Medical Infomatics Association (JAMIA), 14, 304–311.

    Article  Google Scholar 

  • Jason, G. (1988). Hedging as a fallacy of language. Informal Logic, 10(3), 169–175.

    Google Scholar 

  • Jia, L., Yu, C., & Meng, W. (2009). The effect of negation on sentiment analysis and retrieval effectiveness. In Proceedings of the conference on information and knowledge management (CIKM) (pp. 1827–1830). Singapore.

  • Joachims, T. (1999). Making large-scale SVM learning practical. In: Schölkopf, B., Burges, C., Smola, A. (Eds.) Advances in Kernel methods—Support vector learning. Cambridge MA: MIT Press.

    Google Scholar 

  • Kennedy, A., & Inkpen, D. (2006). Sentiment classification of movie reviews using contextual valence shifters. Computational Intelligence, 22(2), 110–125.

    Article  Google Scholar 

  • Klenner, M. (2009). Süsse Beklommenheit und schmerzvolle Ekstase. Automatische Sentimentanalyse in den Werken von Eduard von Keyserling. In Proceedings of the biennial GSCL-conference (pp. 91–97). Potsdam, Germany.

  • Klenner, M., Fahrni, A., & Petrakis, S. (2009a). PolArt: A robust tool for sentiment analysis. In Proceedings of the nordic conference on computational linguistics (NoDaLiDa) (pp. 235–238). Odense, Denmark.

  • Klenner, M., Petrakis, S., & Fahrni, A. (2009b). Robust Compositional Polarity Classification. In Proceedings of Recent Advances in Natural Language Processing (RANLP) (pp. 180–184). Borovets

  • Melville, P., Gryc, W., & Lawrence, R. D. (2009). Text analysis of blogs by combining lexical knowledge with text classification. In Proceedings of the ACM SIGKDD conference on knowledge discovery and data mining (KDD) (pp. 1275–1283). Paris, France.

  • Min, H. J., & Park, J. C. (2011). Detecting and blocking false sentiment propagation. In Proceedings of the international joint conference on natural language processing (IJCNLP) (pp. 354–362). Chian Mai, Thailand.

  • Moilanen, K., & Pulman, S. (2007). Sentiment construction. In Proceedings of recent advances in natural language processing (RANLP). Borovets, Bulgaria.

  • Morante, R. (2010). Descriptive analysis of negation cues in biomedical texts. In Proceedings of the conference on language resources and evaluation (LREC) (pp. 1429–1436). Valletta, Malta.

  • Neviarouskaya, A., Prendinger, H., & Ishizuka, M. (2009). Semantically distinct verb classes involved in sentiment analysis. In Proceedings of the IADIS international conference on applied computing (pp. 27–34). Rome, Italy.

  • Ng, V., Dasgupta, S., & Arifin, S. M. N. (2006). Examining the role of linguistic knowledge sources in the automatic identification and classification of reviews. In Proceedings of the international conference on computational linguistics and annual meeting of the association for computational linguistics (COLING/ACL) (pp. 611–618). Sydney, Australia.

  • Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up? Sentiment classification using machine learning techniques. In Proceedings of the conference on empirical methods in natural language processing (EMNLP) (pp. 79–86). Philadelphia, PA, USA.

  • Polanyi, L., & Zaenen, A. (2006). Contextual valence shifters. In Computing attitude and affect in text: Theory and applications (pp. 1–10). Springer.

  • Porter, M. F. (1980). An algorithm for suffix stripping. Program, 14(3), 130–137.

    Article  Google Scholar 

  • Prabowo, R., & Thelwall, M. (2009). Sentiment analysis: A combined approach. Journal of Informetrics, 3(1), 143–157.

    Article  Google Scholar 

  • Qiu, L., Zhang, W., Hu, C., & Zhao, K. (2009). SELC: A self-supervised model for sentiment classification. In Proceedings of the conference on information and knowledge management (CIKM) (pp. 929–936). Hong Kong, China.

  • Rao, D., & Ravichandran, D. (2009). Semi-supervised polarity lexicon induction. In Proceedings of the conference on European chapter of the association for computational linguistics (EACL) (pp. 675–682). Athens, Greece.

  • Salvetti, F., Reichenbach, C., & Lewis, S. (2006). Opinion Polarity Identification of Movie Reviews. In Shanahan, J. G., Qu, Y., & Wiebe, J. (Eds.) Computing Attitude and Affect in Text: Theory and Applications (pp. 303–316). Berlin: Springer.

    Chapter  Google Scholar 

  • Schmid, H. (1994). Probabilistic part-of-speech tagging using decision trees. In Proceedings of the international conference on new methods in language processing (pp. 44–49). Manchester, UK.

  • Shaikh, M. A. M., Prendinger, H., & Ishizuka, M. (2007). Assessing sentiment of text by semantic dependency and contextual valence analysis. In Proceedings of the international conference of affective computing and intelligent interface (ACII)(pp. 191–202). Lisbon, Portugal: Springer.

  • Taboada, M., Brooke, J., Tofiloski, M., Voll, K., & Stede, M. (2011). Lexicon-based methods for sentiment analysis. Computational Linguistics, 37(2), 267–307.

    Article  Google Scholar 

  • Tan, S., Wang, & Y., Cheng, X. (2008). Combining learn-based and lexicon-based techniques for sentiment detection without using labeled examples. In Proceedings of the ACM special interest group on information retrieval (SIGIR) (pp. 744–745). Singapore.

  • Tan, S., Cheng, X., Wang, Y., & Xu, H. (2009). Adapting naive bayes to domain adaptation for sentiment analysis. In Proceedings of the European conference in information retrieval (ECIR) (pp. 337–349). Toulouse, France.

  • Titov, I. (2011). Domain adaptation by constraining inter-domain variability of latent feature representation. In Proceedings of the annual meeting of the association for computational linguistics (ACL) (pp. 62–71). Portland, OR, USA.

  • Turney, P., & Littman, M. (2003). Measuring praise and criticism: Inference of semantic orientation from association. In Proceedings of ACM transactions on information systems (TOIS) (pp. 315–346).

  • Velikovich, L., Blair-Goldensohn, S., Hannan, K., & McDonald, R. (2010). The viability of web-derived polarity lexicons. In Proceedings of the human language technology conference of the North American chapter of the ACL (HLT/NAACL) (pp. 777–785). Los Angeles, CA, USA.

  • Wiebe, J., & Mihalcea, R. (2006). Word sense and subjectivity. In Proceedings of the international conference on computational linguistics and annual meeting of the association for computational linguistics (COLING/ACL) (pp. 1065–1072). Syndney, Australia.

  • Wiebe, J., & Riloff, E. (2005). Creating subjective and objective sentence classifiers from unannotated texts. In Proceedings of the international conference on intelligent text processing and computational linguistics (CICLing) (pp. 486–497). Mexico City, Mexico.

  • Wiegand, M., & Klakow, D. (2009a). Predictive features in semi-supervised learning for polarity classification and the role of adjectives. In Proceedings of the Nordic conference on computational linguistics (NoDaLiDa) (pp. 198–205). Odense, Denmark.

  • Wiegand, M., & Klakow, D. (2009b). The role of knowledge-based features in polarity classification at sentence level. In Proceedings of the international FLAIRS conference (FLAIRS) (pp. 296–301). Sanibel Island, FL, USA.

  • Wiegand, M., & Klakow, D. (2010). Bootstrapping supervised machine-learning polarity classifiers with rule-based classification. In Proceedings of the workshop on computational approaches to subjectivity and sentiment analysis (WASSA) (pp. 59–66). Lisbon, Portugal.

  • Wiegand, M., Balahur, A., Roth, B., Klakow, D., & Montoyo, A. (2010). A survey on the role of negation in sentiment analysis. In Proceedings of the workshop on negation and speculation in natural language processing (pp. 60–68). Uppsala, Sweden.

  • Wilson, T., Wiebe, J., & Hoffmann, P. (2009). Recognizing contextual polarity: An exploration for phrase-level analysis. Computational Linguistics, 35, 399–433.

    Article  Google Scholar 

Download references

Acknowledgments

This article is an extension of (Wiegand and Klakow 2010). We refer the reader to the section on related work for the full list of extensions. This work was funded by the German Federal Ministry of Education and Research (Software-Cluster) under grant no. ”01IC10S01“ and the Cluster of Excellence for Multimodal Computing and Interaction. The authors would like to thank Ivan Titov for running his statistical domain adaptation method on the data used in this article and Sabrina Wilske for insightful comments.

Author information

Authors and Affiliations

  1. Spoken Language Systems, Saarland University, Building C7.1, 66123, Saarbrücken, Germany

    Michael Wiegand & Dietrich Klakow

  2. Institute of Computational Linguistics, Zürich University, Binzmühlestrasse 14, 8050, Zürich, Switzerland

    Manfred Klenner

Authors
  1. Michael Wiegand
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manfred Klenner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dietrich Klakow
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Wiegand.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wiegand, M., Klenner, M. & Klakow, D. Bootstrapping polarity classifiers with rule-based classification. Lang Resources & Evaluation 47, 1049–1088 (2013). https://doi.org/10.1007/s10579-013-9218-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10579-013-9218-3

Keywords

Search

Navigation