What are developers talking about information security? A large-scale study using semantic analysis of Q&A posts

RELATED WORK

Information security existed prior to the development of the computer. According to Rusell & Gangemi (1991), information security has been around as long as information (Rusell & Gangemi, 1991). Information has required security since it was first sent, stored, and processed. This dates back to when people first started to write. Denning (1999) transports us back to the first century, when Julius Caesar created a code of secrecy to avoid the interception of secret communications transmitted to his friends (Denning, 1999). From the earliest days of communication, diplomats and military commanders understood that it was necessary to provide some mechanism to protect the confidentiality of correspondence and to have some tools to detect security breaches, and they took initiatives to ensure information security. Later, procedural transaction control processes and practices were mostly used to ensure information protection. With the spread of postal services, governments created official bodies to seize, decipher, read, and reseal letters (for example, the United Kingdom Secret Office, established in 1653) (Johnson, 1998). At the end of the 20th century and in the first years of the 21st century, telecommunications, computer hardware and software, and data encryption all changed quickly and in new ways. Innovations and new ideas in the 21st century have made it clear how important IT infrastructure is for information security (Yang et al., 2016). Companies have realized that their ability to stay competitive in highly volatile and changing markets depends on how well they protect their information assets and IT infrastructures from security risks (Soomro, Shah & Ahmed, 2016; Ključnikov, Mura & Sklenár, 2019). The fact that all of these new technological advancements are susceptible to hazards and negative effects, just like any other new digital inventions, is now widely acknowledged by the IT communities (McCormac et al., 2017).

In line with the developments since its existence, information security has become a common concern and work area not only of the IT communities but also of most industries and academies (Silic & Back, 2014; Glaspie & Karwowski, 2018). Numerous workshops and conferences were held to form joint working groups for information security, discuss information security issues, and improve cooperation (Dlamini, Eloff & Eloff, 2009; Silic & Back, 2014; Soomro, Shah & Ahmed, 2016). With these interactions and activities, the continuous participation of the ever-expanding communities in the field has increased the popularity of information security day by day. While the information security paradigm has developed rapidly, the number of studies in the information security literature has increased rapidly (Whitman & Mattord, 2021).

Numerous literature reviews have been conducted so far to understand the information security paradigm (Dlamini, Eloff & Eloff, 2009; Silic & Back, 2014; Soomro, Shah & Ahmed, 2016; Glaspie & Karwowski, 2018). Domain-specific studies have generally addressed information security architectures, techniques, and tools (Whitman & Mattord, 2021), risk analysis and management (Soomro, Shah & Ahmed, 2016), threats to information security and solutions (Lopez et al., 2018), applications in different fields (Silic & Back, 2014), information security awareness (McCormac et al., 2017; Albayrak & Bağcı, 2022), the development of secure protocols (Dlamini, Eloff & Eloff, 2009), and the evolution and future of information security (de Leeuw & Bergstra, 2007; Stewart, 2013). These reviews offered insights into the themes and trends of information security in both academia and industry (Furnell, Fischer & Finch, 2017; Glaspie & Karwowski, 2018).

In addition, numerous studies have focused on online Q&A forums similar to Stack Overflow (Stack Exchange, Quora, Kaggle, and Reddit), which is a crucial venue for developers from diverse backgrounds in the IT industry to share knowledge and experiences (Vasilescu, 2014; Ahmad et al., 2018; Croft et al., 2022). In particular, several studies have been carried out based on the examination of posts from Q&A forums to investigate the issues and difficulties faced by developers and security specialists in various cybersecurity scenarios (Yang et al., 2016; Lopez et al., 2018, 2019; Croft et al., 2022). Yang et al. (2016) analyzed the security-related posts of developers on the Stack Overflow Q&A website with a topic modeling approach. As a result of this analysis, they classified safety-related topics into five main categories (Yang et al., 2016). Lopez et al. (2019) created an anatomy of security conversations using a manual categorization on Stack Overflow. From a similar perspective, Lopez et al. (2018) analyzed a series of questions in the security channel of Stack Overflow to draw inferences about how developers use Q&A resources to solve security problems. Croft et al. (2022) conducted a large-scale study of the security challenges of 15 programming languages by analyzing developers’ discussions on Stack Overflow and GitHub using a topic modeling approach.

The scope of our study encompasses novel methodologies and a more detailed dataset, which builds upon and expands earlier research. Our study analyzes all posts shared on ISSE using a semi-automated methodology that is not influenced by author prejudice. We employ a topic modeling approach based on unsupervised machine learning to address this gap in the existing literature. Topic modeling generates the word clusters that best define the semantic map of the contents as different themes (Blei, 2012; Gurcan & Cagiltay, 2022). Owing to the methodological advantages it offers over semantic content analysis, topic modeling is growing in popularity for unstructured textual data (Gurcan et al., 2022b; Ozyurt et al., 2022). Therefore, numerous studies have been conducted using topic modeling procedures in sub-contexts of various disciplines (Chen, Thomas & Hassan, 2016; Silva, Galster & Gilson, 2021). In conclusion, topic modeling has proven to be an effective and appropriate way for semantic analysis and interpretation of unstructured texts, which has further motivated us to use this methodology to investigate information security challenges.

Data query and retrieval

To make available a systematic and unbiased methodological approach, we investigated an archive of all postings generated on the Information Security Stack Exchange (ISSE), an information security-focused Q&A community. (Stack Exchange, 2023). The XML data dump of the datasets generated and examined for the present study is accessible to the general public through the Internet Archive repository (Internet Archive, 2023). Initially, the up-to-date XML data dump (updated October 8, 2022) was downloaded and parsed into a relational data file before being processed. There are two different types of posts in the data dump: question and answer posts. In total, the experimental corpus includes 180,937 posts published from November 2010 to October 2022, of which 66,077 are questions and 114,860 are answer posts. In addition, it has been determined that approximately 5,082 questions and 8,835 answers are shared on the ISSE platform on average every year. Each question post has a number of metadata components, including title, tags, body, code, answers, comments, and further information. (Yang et al., 2016). Answer posts, unlike question posts, do not have title or tag elements.

Data preprocessing

We perform a five-step data preprocessing to transform the unstructured textual raw data in the experimental corpus into noise-free and structured data. Since posts on Stack Overflow often contain code snippets, we removed them in the first step (Uddin et al., 2021). Since code snippets in posts on Stack Overflow often contain syntax and code specific to programming languages, they have no semantic contribution to topic modeling and can degrade the performance of the analysis (Barua, Thomas & Hassan, 2014). Also, because the shared source code on Stack Overflow contains only small pieces of code, there is not enough semantic context to derive meaningful topic models from code snippets. Therefore, we clean up the code snippets marked with <code></code> from posts using a similar approach to previous work. Since HTML tags (e.g., <p>, <pre>, and <b>) and URLs do not contribute to the creation of nautical topics in the topic modeling process, we remove all HTML tags and URLs from the text in the corpus (Croft et al., 2022). In step three, we continue to remove numbers, punctuation, and other non-alphabetic characters from the dataset to enable better topic modeling results. As a fourth step, we deleted all English stop words (how, what, and, is, or, the, a, an, for, etc.) that don’t make sense on their own (Gurcan et al., 2022a). Finally, we applied the lemmatization process to the remaining words in order to reduce the words from their derived form to their nominative form (Gurcan & Cagiltay, 2022).

Topic modeling analysis

Our goal is to uncover debate topics related to distinct contexts of information security in our dataset from the ISSE platform. In order to address this issue, we employ a topic model utilizing the latent Dirichlet allocation (LDA) algorithm to investigate and establish specific debate topics within various information security frameworks. LDA is a statistical and generative approach for topic modeling, where topics are represented as probability distributions over words in a collection of texts (Blei, Ng & Jordan, 2003). Additionally, it provides a concise overview of the documentation by representing the detected topics as probability distributions. LDA employs word frequencies and word co-occurrences across documents to construct a semantic topic model consisting of interconnected terms (Gurcan, Dalveren & Derawi, 2022; Gurcan et al., 2023). Words pertaining to a given issue are typically connected in meaning, thereby attributing a distinct significance to the topic in consideration. Conversely, because of the absence of semantic information in the LDA model, the meanings of the linked topics must be ascertained through manual examination of the word set. The presence of phrases such as “encrypt”, “data”, “encryption”, “decrypt”, “unencrypted”, and “plaintext” in a topic suggests that the issue is likely to be connected to encryption in the field of information security (Blei, 2012). LDA has gained widespread adoption as an approach for extracting semantic themes from unstructured textual materials. The topics generated by LDA are less prone to overfitting and are more easily interpretable (Vayansky & Kumar, 2020). Hence, a substantial body of research has surfaced in the domain of software engineering employing topic modeling techniques (Chen, Thomas & Hassan, 2016; Silva, Galster & Gilson, 2021).

We utilize the LDA model implementation offered by the Gensim package to employ LDA on our experimental corpus (Řehůřek & Sojka, 2011). For every topic modeling experiment based on LDA, we increment the number of topics, designated as K, one by one while keeping all other parameters at their default values. The K parameter, denoting the quantity of topics, is a variable typically chosen by the user and determines the level of detail of the topics (Gurcan, Dalveren & Derawi, 2022). Having a K number that is neither too large nor excessively small is considered unsuitable. Excessive K values might result in topic relocation and repetition, while insufficient K values may lead to the identification of broader, partially differentiated topics. We conducted a series of experiments using various K values in order to identify themes that exhibited the greatest level of consistency (Vayansky & Kumar, 2020). In this context, coherence scores are employed to evaluate the significance and coherence of the topics that have been identified. In this empirical process, we systematically varied the K range from 5 to 60, increasing by 1 at each iteration (e.g., 5, 6, 7,…, 60). In this way, we identified the optimal K value that generates topics with high coherence scores (Croft et al., 2022). Concurrently, we computed a coherence score that is tailored to the LDA model for each experiment that is fitted with the K value. We used the four-step topic coherence pipeline and the coherence model built into the Gensim module to do this computation (Mimno et al., 2011; Řehůřek & Sojka, 2011; Röder, Both & Hinneburg, 2015; Gurcan, 2023a). A high coherence score signifies that the themes exhibit greater distinctiveness from one another in the semantic domain. Ultimately, we selected the topic model that had the highest coherence score. The study revealed that the coherence score for K = 38 achieved its highest value (Cv = 0.7189). Hence, after analyzing the findings, we determined that 38 is the most suitable number of topics.

Applying LDA-based topic modeling to our corpus using the sequential processes mentioned above resulted in the generation of 38 topics. The chosen model provides the following metrics and information for each identified topic: (1) Keywords that provide a detailed description: A ranked list of the N most frequently occurring words that describe the topic, along with the probability of each word. This probability indicates the relative dominance of each word in terms of its descriptive power for the topic. We discovered the 20 most prominent words for each topic. (2) Distribution of the topics: Every post is assigned a correlation value ranging from 0 to 1, indicating the degree of association between each post and the themes. A post is deemed more “on-topic” when it has a higher correlation to a specific topic. We allocated a post to a topic based on the highest correlation values between the post and the topic (Uddin et al., 2021; Gurcan, 2023b).

After the postings were allocated to the prevailing topics, we carried out the task of determining topic tags that briefly summarize the conceptual background of each topic. We manually assigned tags to each topic using an open-card sorting approach, taking into account previous research on topic labeling (Katsanos et al., 2019). In open-card sorting, there is no previously determined label for a topic. Instead, this label is determined using the open coding process (Katsanos et al., 2019; Uddin et al., 2021). To classify a topic, we utilize two forms of information: (1) a compilation of the most frequently employed words in the topic, and (2) a selection of 20–30 articles on that topic chosen at random. Based on this information, the label for each topic was derived and assigned. During the card sorting process, the coders, who are the experts in the subject, assigned a label to each topic. Finally, the 38 topics were classified using the open-card sorting method, considering their background (Katsanos et al., 2019; Croft et al., 2022). As a result, a comprehensive taxonomy map was developed to categorize and map the primary themes, contexts, and application areas within the field of information security. The map is organized into seven categories.

What information security issues are addressed? (RQ1)

Our analysis employed LDA-based topic modeling to examine the information security concerns raised on the ISSE platform. The method section provides a comprehensive explanation of our modifications and implementation of LDA-based topic modeling in our corpus. Through the utilization of LDA-based semantic content analysis, a total of 38 topics were identified, with each topic being characterized by 20 descriptive keywords. Table 1 presents the 38 topics (issues) identified by LDA-based topic modeling, along with their corresponding percentages. The themes (problems) are arranged in Table 1 in descending order based on their respective percentages. Furthermore, Table A1 provides the top 20 keywords associated with each topic. The words “topic” and “issue” are utilized interchangeably in the current study due to the fact that the topics listed in Table 1 represent significant issues and challenges related to information security. Topics showed that information security issues cover a wide range of topics, from “Cyber Attacks” to “XSS Attacks”, “Security Testing” to “Code Vulnerability”, “Certification” to “Encryption” (see Table 1).

Moreover, the five most commonly asked topics across the board were “Cyber Attacks”, “Security Testing”, “Certification”, “User Account”, and “Wi-Fi Networks”. The least asked topics were “Virtual Machine”, “SQL Injection”, and “Digital Signature”. Especially in the information technology ecosystem, where even highly secure systems are exposed to cyber-attacks, preventing these attacks and making the systems protected against them has emerged as the main problem focused on by information security experts. The fact that “Cyber Attacks” and “Security Testing” are at the top of the list is a finding that confirms this. The topics discovered also shed light on the priorities and current trends in the ever-growing information security industry. Information security topics cover a wide range of knowledge, skills, and backgrounds in different areas of expertise. The broad spectrum of information security is clearly reflected in the scope of the topics explored.

In order to make the main themes of information security more understandable, a categorization process was carried out by associating these topics with their conceptual background and processes. The 38 information security topics were classified under seven elementary categories, and a structural taxonomy map was developed for information security and presented in Fig. 1. As shown in Fig. 1, information security issues are classified under seven categories: “System Security” (24.25%), “Cryptography” (20.85%), “Network Security” (18.02%), “Web Security” (14.64%), “Data Security” (10.76%), “Software Security” (7.56%), and “Mobile Security” (3.93%). Issues in the “System Security” (24.25%) and “Cryptography” (20.85%) categories account for almost half of the information security issues. This indicates that “System Security” and “Cryptography” categories dominate among information security issues. On the other hand, it was seen that the issues in the “Software Security” (7.56%) and “Mobile Security” (3.93%) categories had relatively the lowest percentages.

How do information security issues evolve over time? (RQ2)

In order to investigate this research question, we analyze the evolution of issues surrounding information security over a specific timeframe to comprehend the persistent and unresolved issues in information security and generate innovative methods to tackle them. In order to accomplish this, we employed the temporal trend calculating methodology that we utilized in our prior study to disclose the movements of each topic over the last decade (Gurcan, 2023a). Table 2 displays the yearly percentages, trend values, and trend directions for the themes arranged according to the general trend. Each row in this table displays the yearly percentage changes for a topic and provides a number of insights regarding the evolution of the topics over time. Based on the movement values in the final column of Table 2, it can be observed that out of the 38 topics, 28 exhibit an upward trend, while 10 show a downward trend. Topics with the most increasing trend emerged as “Certification”, “Encryption Keys”, “Web API”, “Disk Encryption”, and “VPN”. On the other hand, “Corporate Data”, “Cyber Attacks”, and “Security Testing” are the topics with the most decreasing trend. It is a remarkable finding that although “Cyber Attacks” and “Security Testing” are the two topics with the highest percentage, they have the most noticeable downward trend.

How have the difficulty and popularity of information security issues progressed over time? (RQ3)

The results of our investigation in RQ1 unveiled the extensive range and variety of information security issues. The significance, prevalence, and level of complexity of the issues and challenges in information security differ for each specific topic. Certain questions may be presented numerous times, but others may not be presented at all. Assessing the prevalence and complexity of information security issues can aid in the search for effective ways to address these challenges. Various descriptive metrics provide insights into the features of the information security questions shared on ISSE. The aforementioned indicators comprise various parameters, including the number of views, responses, accepted answers, favorites, and comments. During this phase of our analysis, we utilized the metric calculation approach described in our previous study to determine the number of views, answers, accepted answers, favorites, comments, and scores for every topic (Gurcan, 2023a). Subsequently, we computed metrics that ascertain the level of complexity and the level of interest for each topic (Gurcan, 2023a). In order to illustrate all aspects of information security issues, we computed the averages of each topic’s questions, answers, accepted answers, favorites, scores, views, and comments and demonstrated them in Table 3.

The topics in this table are listed in descending order of their percentages. In addition, the difficulty score indicating the difficulty level of the topics is given in the last column of the table. The findings in Table 3 provide important metrics about various characteristics of information security issues. In order to give a visual sense, we also showed how popular the topics were in Fig. 2 and how difficult they were in Fig. 3. Based on Fig. 2, the five topics with the highest number of views are “Cipher Suites”, “Phone Scam”, “Password Hashing”, “Entropy”, and “Wi-Fi Networks”. Conversely, the topics with the lowest viewership include “Web API”, “Security Testing”, “Code Vulnerability”, “Digital Signature”, and “Encryption/Decryption”. Based on the difficulty score shown in Fig. 3, the topic with the highest level of difficulty is “Web API”, with a score of 0.78. The other most difficult topics are “Logging”, “Phone Scam”, “Mobile Apps”, and “Buffer Overflow”. On the other hand, “Cyber Attacks” and “Password Hashing” emerged as the most easily answered topics.

In the field of information security, which tasks, techniques, and tools are the most frequently employed? (RQ4)

Information security entities, encompassing a diverse range of tasks, strategies, and technologies, are currently prevalent in all modern information ecosystems. Trends in information security technologies are constantly evolving simultaneously with technological transformations in information ecosystems. From this perspective, we have extended our investigation to find the linkages between information security challenges and information security-related technologies at this time. Every question post on the ISSE platform is accompanied by user-defined tags that describe the specific context and background of the question. The user selects these tags and includes them in their question. Tags are informative keywords that users assign to their inquiries to indicate the specific techniques, technologies, and tools related to information security. In order to get a particular set of tags correlated to information security, we initially segregated the tags of each post into separate entities and then computed the frequency of each tag over all postings. Consequently, we recognized a grand total of 1,257 unique tags that are frequently employed in the realm of information security. From these tags, we have determined the top 50 tags that appear most frequently. We have calculated their percentages and presented them in descending order in Table 4. According to Table 4, “encryption” is the most prominent tag, followed by “tls”, “authentication”, “passwords” and “web-application”.

Furthermore, in order to offer insight into the time-related patterns of the top 50 tags, we displayed the yearly ratios, trend ratios, and orientations of the tags in Table 5. The general trend values of the tags are demonstrated in the last column of Table 5 and are sorted from increasing to decreasing. As can be seen in Table 5, the prominent tags with a rising trend are “malware”, “tls”, “certificates”, “openssl” and “account-security”. In contrast, the tags seeing a decline in popularity were identified as “appsec”, “web-application”, “network”, “cryptography”, and “penetration-test”.

In the concluding phase of our investigation, we have determined the top 15 tags that appear most frequently for each of the 38 topics. These findings are presented in Table A2, where the topics are arranged in descending order based on their respective percentages. Similarly, these tags in each row are sorted for each topic in descending order. In this sense, Table A2 provides an understanding of which information security issues are related to which tags and how much. For example, for the “Cyber Attacks” topic in the first row of the table, the “encryption” tag has the highest frequency, while the “denial-of-service” tag has the lowest frequency. This categorization of the tags, given in Table A2, reveals both the nature and scope of the issues and the tools and technologies with which they are related. Therefore, categorizing tags under specific problems significantly contributes to a more accurate understanding of the problems and their rapid and effective resolution. It also informs users about the origin and background of the questions, as well as related tasks, technologies, and experiences.

Implications for researchers and practitioners

Information security covers the measures and activities necessary to ensure the safety of users, the safety of the data, and the protection from malicious activities. With the effect of technological developments and the increase in malicious activities, it is more important than ever to be conscious of information security and take the necessary safety measures (Abraham & Chengalur-Smith, 2010). The issues and user experiences shared on ISSE and similar Q&A platforms can be considered an important source of information and motivation for efforts to solve information security issues (Croft et al., 2022). The experimental backdrop, methods, and outcomes of this investigation may have noteworthy ramifications and instructions that will contribute to the understanding of domain-specific issues for information security communities and enable them to take the necessary security measures (McCormac et al., 2017). Our findings may guide us in taking the necessary measures to protect data, ensure the safety of computer networks, and identify and implement security policies (Furnell, Fischer & Finch, 2017). Our research is aimed at assisting a wide range of information security stakeholders, including practitioners, instructors, academics, developers, and enthusiasts.

Developers may drive the evolution of the area of information security by developing more particular apps and devices to solve contemporary information security concerns and demands, as indicated in our outcomes. Tool developers can create useful libraries or tools for popular and difficult issues. Inferences to be drawn from the issues expressed on online sharing platforms such as ISSE can help the research community better understand the issues and difficulties encountered in information security (Croft et al., 2022). Although all of the concerns highlighted are relevant in and of themselves, the results we obtain suggest that information security researchers should focus on the most prominent and problematic challenges. For example, the topics in the “System Security” and “Cryptography” categories are about half of the information security issues. In particular, the issues in these two categories can be considered priorities, and faster solutions can be produced (Silic & Back, 2014). Furthermore, researchers can take advantage of our research methodology to discuss different contexts of information security and thus expand our analysis.

Information security instructors can organize online training programs and webinars to emphasize the importance of being conscious of information security and taking the necessary security measures. In particular, considering the most popular and most challenging security problems, a more effective training curriculum can be prepared for information security trainings (Furnell, Fischer & Finch, 2017). Instructors may maintain their educational programs and curriculum up to date with current developments and give a modern understanding of information security potential hazards (Glaspie & Karwowski, 2018). Our analysis may be utilized by ISSE or other Q&A services to define, label, and better categorize user posts within a systematic taxonomy (Barua, Thomas & Hassan, 2014). Our consequences may be helpful to readers in general and information security enthusiasts to stay updated on the latest advancements and trends in the area.