Evolution of Information Systems Business Value Research: Topic Modeling Analysis Ahad ZareRavasan1 , Anand Jeyaraj2 * 1 Masaryk University, Czech Republic 2 Raj Soin College of Business, Wright State University, U S A Abstract Information Systems Business Value (ISBV) has been a key research topic within the information systems discipline through time. Over the last three decades, ISBV research has dealt with various aspects such as the type of relationships, research methods, theoretical foundations, influencing factors, and measurement issues. This research applies topic modeling on the abstracts of 2824 articles published between 1990 and 2020. Results show that topics such as IS management, IS implementation, and IS public services had endured over time; topics such as IS strategy and IS design had initially gained and then lost traction; and topics such as IS social practices, IS emerging services, and IS sustainability have gained momentum in recent years. Further, ISBV research tends to examine specific and emerging IS tools, technologies, and applications such as Blockchain, Internet of Things (IoT), and business analytics, and seems poised for greater focus on IS sustainability in the near future. Keywords: Business Value, Information systems, Information Systems Business Value, Review, Topic Modeling. 1 Introduction Information Systems1 Business Value (ISBV) discipline has been and is expected to remain one of the major research topics for information systems (IS) researchers [1]. Given the changing role of IS during different digital eras, the way it creates business value has also evolved [2]. SAP Corporation provides the evolution of digital eras from 1960 to 2020 [3]. The first era, "mainframe and Personal Computers (PCs)", lasts from the 1960s to the 1980s. The primary value of IS in this era stands for "industrial automation" with the emergence of PCs and the large-scale adoption of mainframes aimed at plant floor automation. During 1990-2000, the era of "client-server and internet" appeared with the widespread adoption of PCs, broadband internet connection, and the large-scale adoption of enterprise information systems, especially Enterprise Resource Planning Systems (ERPs), aimed at "business process automation", inter and intraorganizational integration, and also as a solution to Y 2 K problem. The emergence of modern concepts and technologies such as smart phones, cloud computing, social networking and Big Data shaped the third era of "cloud, mobile and Big Data" during 2000-2010 triggered the "digital transformation" of business models. Finally, the last era; "intelligent technologies" appears, enabled with advanced technologies such as Artificial Intelligence (AI), Internet of Things (IoT), distributed computing, and Blockchain, shaping more "intelligent enterprises". Not surprisingly, the ISBV body of knowledge has also evolved in line with IS evolution to assist businesses in gaining the most advantage of IS initiatives. Early studies focusing on the financial impacts of IS doubted the economic value of IS [e.g., 4-6]. Later, direct positive 1 IS research has used the term 'Information System (IS)' interchangeably with 'Information Technology (IT)', and 'Information and Communication Technology (ICT)' [1]. In this research a 'holistic' view on IS has been adopted, as suggested in the ATIS Telecom Glossary: "The entire infrastructure, organization, personnel, and components for the collection, processing, storage, transmission, display, dissemination, and disposition of information". In line with this definition, and for consistency, we label IT and ICT as IS throughout the text. associations between IS investments and business value were observed, and IS intangible effects were also considered [e.g., 7-9]. Finally, studies attempted to "unlock the gray box", focusing on ISBV creation mechanisms or indirect effects, such as the effect of IS on business processes [e.g., 10, 11], absorptive capacity [e.g., 12], or strategic alignment [e.g., 13]. The emergence of advanced technologies has also influenced the ISBV trend, as more and more research took examining the ISBV of specific IS applications and technologies such as Big Data, IoT, AI, and Blockchain, rather than studying general IS initiatives [2]. The evolution of the ISBV domain has been of research interest not only from the lenses of the causality and type of relationships (i.e., positive/negative, or direct/indirect effects) and adopted technologies, but also from other perspectives such as research methods, theoretical foundations, contextual factors, and measurement issues [see 14]. Some prior research has attempted to synthesize empirical findings of the literature to reveal ISBV. This is one of the essential tasks for advancing a field of research and integrating the fragmented areas, particularly when characterized by exponential growth involving various research themes and topics, such as ISBV research. Drawing a coherent map of literature in any domain enabled by subjective and qualitative literature reviews is essential to reveal conceptualizations and theory development [15], yet insufficient [16]. Several common qualitative approaches such as literature reviews [1, 17, 18], archival research [19], and meta-review analysis [14] have been leveraged to synthesize the ISBV literature, present the state-of-the-art, and draw the deficiencies and research gaps in the ISBV literature (see ZareRavasan and Krcal [2] for a full review). However, the extensive body of the ISBV literature makes such approaches time and cost-consuming with a lack of replication possibility. Automated quantitative approaches such as topic modeling [e.g., 20-22], computational literature review [e.g., 23], and bibliometric analysis [e.g., 24-27] have been recently adopted in the context of different IS domains. With the exception of Big Data and analytics business value in Batistic and der Laken [28], quantitative approaches to synthesizing the ISBV literature are scarce. Considering the unfilled gap in the literature, we investigate dominant ISBV research topics to provide a clear view of the extant knowledge base. Understanding ISBV key themes is crucial since the topics have been evolving or getting integrated into other topics over time due to the emergence of new technologies and changes in the role of IS. Mapping the evolution of ISBV over time also adds significant value and knowledge to the ISBV literature. We apply the topic modeling approach to discover the dominant topics and analyze the trend of topics over time using slope analysis. Specifically, our study examines the evolution of ISBV research over 30 years (1990-2020) based on journal articles indexed in Scopus. We adopt one main analysis (for the whole period) and three subgroup analyses (for each decade). The main analysis discovers the dominant topics overall, whereas the subgroup analysis identifies the prevalent topics in different periods, the emergence of new topics, and the evolution of topics over time. Our study complements earlier qualitative reviews but employs a broader scope and includes a larger corpus of documents, which provides a more robust, structured, comprehensive, and objective presentation of the evolution of the ISBV research as well as the trends of various ISBV topics. 2 Theoretical Background 2.1 Information Systems Business Value ISBV is "the impact of investments, particularly IS assets, on the multidimensional performance and capabilities of economic entities at various levels, complemented by the ultimate meaning of performance in the economic environment" [1]. Early studies primarily defined ISBV as the contribution of IS to firm performance and used firm-level output or end-product-based measures of ISBV [e.g., 29]. Since the singular focus on firm-level output variables provides a limited understanding of how ISBV was created, later studies adopted a process perspective and included the impact of IS on intermediate business processes [e.g., 30]. This shift to a processoriented approach offers greater potential for meaningful measures of ISBV and provide better insights into how ISBV can be created [31]. Soh and Markus [30] describes how the effects of IS over a chain of interrelated yet uncertain outcomes from 'IS investment' to 'firm performance'. Increases in 'firm performance' require an essential degree of 'IS impacts', which in turn require 'IS assets', which require 'IS investments'. The link from 'IS investments' to 'IS assets' involves the process of IS management/conversion and investment in complementary (non-IS) investments ('IS conversion process'); the link from 'IS assets' to 'IS impacts' relies on the effective US use ('IS use process'); and the link from 'IS impacts' to 'firm performance' depends on the 'IS competitive process' [30]. While [30] continues to serve as a theoretical foundation for ISBV [32, 33], the model has improved and evolved. In particular, the effect of contextual factors and time lag has been embedded in the model later on [see 1, 2]. For a comprehensive view of the ISBV processes, this research adopts an integrated ISBV framework based on Soh and Markus [30] and Schryen [1], as shown in Figure 1. Contextual/environmental factors IS factors Firm factors Industry factors Country factors IS Conversion Process IS Use Process IS Competitive Process IS investments IS assets IS impactsIS investments IS assets IS impacts IS management/ conversion activities IS Effective / ineffective use Competitive dynamics Competitive position Figure 1. ISBV process model 2.2 Prior reviews and syntheses on ISBV literature Prior literature on ISBV has been based on a diverse body of research, including conceptual, theoretical, analytic, and empirical studies [17]. ISBV can be investigated at different analysis levels, such as individual, firm, industry, and economy (also known as country or macro) [34]. ISBV research can be of ex-ante and ex-post nature in which ex-ante evaluation addresses which available IS investment options will best benefit the firm while ex-post research investigates the extent to which IS has created value [35]. ISBV has also been examined taking IS investment, IS capabilities, IS resources, IS staff, IS adoption, and IS use into consideration [2]. Due to such diversity in the ISBV discipline, some aspects of ISBV have been over-researched while others might remain partially unaddressed. This imbalance in the focus on ISBV research spurred detailed investigations to synthesize the ISBV literature over time, as explained next. Among the first literature review studies, Kauffman and Weill [36] review 13 ISBV studies published from 1975 to 1988 and summarize their motivation, focus, and caveats. Brynjolfsson [29], based on an ISBV review, argues that the shortfall in IS productivity is due to deficiencies in measurement and methodology. Brynjolfsson and Yang [37] update Brynjolfsson's [29] review by classifying IS productivity studies into economy-wide, industry-level, firm-level, consumer surplus, and economic growth. Chan [38] classifies ISBV studies according to research methods, measures, and levels of analysis. Dehning and Richardson [39] propose a framework to classify ISBV studies regarding the influence of IS spending, IS strategy, and IS capability on performance. Dedrick et al. [40] review empirical studies examining the economic value of IS on firm, industry, and country levels. Kohli and Devaraj [41] meta-analyze 66 ISBV papers and examine IS payoff determinants. Melville et al. [17] review 202 ISBV papers and develop an ISBV model incorporating the firm, industry, and country environment. Piccoli and Ives [42] synthesize ISBV literature that examines IS role in sustaining competitive advantage. Chau et al. [34] propose a taxonomy on ISBV dimensions and issues in measuring ISBV. Wan et al. [43] analyze 96 papers from 1996 to 2006 that cite Brynjolfsson and Hitt's [44] paper. They categorize empirical research by their results, level of analysis, research methods, and variables included in the model. Kohli and Grover [35] review ISBV literature at the firm level. Pare et al. [18] classify empirical research papers according to their method and goal. Liang et al. [45] conduct a meta-analysis of 42 ISBV studies to discover how different ResourceBased View (RBV) factors impact firm performance. L i m et al. [46] use meta-analysis to examine ISBV research focusing on IS investment's return. Masli et al. [19] synthesize recent empirical ISBV research with a focus on measurement issues. Schryen [1] proposes a new ISBV model. Sabherwal and Jeyaraj [31] extend Kohli and Devaraj's [41] work by meta-analyzing 303 ISBV studies published from 1990 to 2013. They investigate (1) IS investment measurement approaches, (2) study's methodological attributes, (3) value generation, (4) value measures, (5) value enablers, (6) moderating effects, and (7) theoretical approaches. More details regarding the ISBV literature reviews can be found in ZareRavasan and Krcal [2]. A summary of the review period and sample size of prior ISBV literature reviews are illustrated in Table 1. According to the table, prior ISBV reviews primarily used the full text of articles, as they needed to qualitatively code variables and relations manually. Besides, there is no ISBV study aiming to objectively cluster ISBV literature and analyze the stability of topics over time, which is the focus of this research. Table 1. Summary of prior ISBV reviews Citation Review period (sample size) Method Kauffman and Weill [36] 1975-1988 (13 empirical studies) Literature review Brynjolfsson [29] 1986-1991 (17 empirical studies) Literature review Brynjolfsson and Yang [37] 1982-1995 (46 empirical studies) Literature review Chan [38] 1993-1998 (38 empirical studies) Meta-analysis Dehning and Richardson [39] 1996-2001 (31 empirical studies) Meta-analysis Dedrick et al. [40] 1985-2002 (34 empirical studies) Meta-analysis Kohli and Devaraj [41] 1990-2000 (66 empirical papers) Meta-analysis Melville et al. [17] Up to 2002 (202 papers) Literature review Piccoli and Ives [42] 1981-2003 (117 papers) Literature review Chau et al. [34] 1993-2005 (41 papers), and 2000-2005 (49 papers) Literature review Wan et al. [43] 1996-2006 (96 papers that referenced Brynjolfsson and Hitt [44]) Literature review Kohli and Grover [35] -not reported Literature review Pareetal. [18] 1991-2005 (124 empirical papers published in MISQ and ISR) Literature review Liang et al. [45] 1990-2009 (42 empirical papers adopted RBV) Meta-analysis Limet al. [46] 1990-2010 (44 empirical papers) Meta-analysis Masli etal. [19] 2000-2011 (>50 empirical papers) Literature review Schryen [ 1] 1989-2012 (>200 empirical papers) Literature review Sabherwal and Jeyaraj [31] 1990-2013 (265 empirical papers) Meta-analysis ZareRavasan and Krcal [2] 1990-2020 (235 papers) Literature review This research 1990-2020 (2824 papers) Topic modeling 3 Method The main research steps are illustrated in Figure 2. A combination of manual and automated methods was applied for data collection, data cleansing and preparation, topic modeling and visualization, and finally, post hoc analysis. Different tools, programming languages, and libraries are utilized to achieve the intended outcomes. Data Collection Data Cleansing and Preparation - Descriptive Analysis Topic Modeling, and Visualization Post hoc Analysis R matpl tfrb ^ python" GEN 5 IM pyLDAvis R Real Statistics R Figure 2. Research steps, employed applications, and libraries 3.1 Data collection We employed the Scopus search engine to discover the ISBV literature because of its broad coverage and high data quality. To ensure that all the relevant articles to the research topic were included in our final dataset, the Title, Abstract, and Keywords of the published papers in Scopus were queried using Wildcard (*), a Boolean operator (OR), and proximity operator W/n (within the n words) as shown below. TITLE-ABS-KEY ( "Information * technology' OR "Information * system*" OR "enterprise * system") W/l TITLE-ABS-KEY (investing OR investment* OR expenditure OR capab* OR competen* OR use OR usage OR asset OR resource* OR implement* OR adopt* ) AND TITLE-ABS-KEY (performance OR profitability OR productivity OR value OR benefit OR impact OR effect) AND ( LIMIT-TO ( SRCTYPE , " j " )) AND ( LIMIT-TO ( LANGUAGE , "English")) AND ( LIMIT-TO ( DOCTYPE , "ar" )) The search query is based on the underlying concepts of the ISBV framework in Figure 1. To cover the chain from 'IS investments' to 'firm performance,' our search query includes 'information technology', 'information system', and 'enterprise system' to refer to the IS. It also includes 'investment' and 'expenditure,' value generation concepts such as 'assets', 'capability', 'adoption', and 'use' [e.g., 31], and related variants such as 'resources', 'competencies', and 'implement'. To capture only those studies that examined ISBV measures, we included 'performance', 'profitability', and 'productivity' [e.g., 31], 'value', 'benefit', 'impact' and 'effect' [e.g., 1]. We searched for journal articles published in English from 1990 to 2020. Conference papers, textbooks, chapter books, and working papers were excluded as scholars use indexed journals to disseminate quality findings [47]. We also excluded review papers and editorials. The search was conducted in June 2021 and returned 2,839 journal articles. Using the Scopus feature, we downloaded and merged all the metadata (such as journal name, publication year, title, abstract, and keywords) into a single C S V file. After an initial check, 15 papers were removed since the abstract text was unavailable. The remaining 2,824 articles build our corpus, which is considerably large and poses difficulties for manual analysis and interpretation. These articles were published in 1160 journals and authored by more than 6600 authors. 470 papers were single-authored. In Figure 3, we show the distribution of articles over time and by the geographic locations of the authors. Researchers from the USA, China, U K , India, and Canada contributed the most articles to the ISBV literature. l'i'iO 1AH 2010 Figure 3. Distribution of published articles 3.2 Data cleansing and preparation The abstracts of the articles were used for analysis, a common approach adopted in prior IS research [e.g., 48] since well-constructed abstracts contain the essential information of research papers and facilitate quick and accurate identification of the research topics [23]. The following steps are conducted for data cleansing. First, the terms2 were standardized to remove duplicate references to the same term. For instance, "HIS," "HIT," "hospital information system," "health information technology," "healthcare information technology," and "healthcare information system"; "GIS" and "geographic information system"; " E R P " and "enterprise resource planning"; "MIS" and "management information system"; "DSS" and "decision support system"; "PLS" and "partial least squares"; " T A M " and "Technology Acceptance Model," and " R B V " and "resource-based view" were coded as similar words. Second, multi-word phrases such as "business analytics," "social network", "dynamic capability", and "business model" were preserved by replacing spaces with underscore characters. Through this, we tried to keep the meaning behind multi-word terms (n-grams) of our corpus. To objectively discover n-grams, we employed the "terminology extraction, multi-word terms" functionality of Sketchengine3 . This tool provides a C S V file of the frequency of multiword terms of the given corpus. Visually checking the C S V file, we discovered a list of multiword terms relevant to our study and worth preserving in the corpus, which we did by replacing spaces with underscore characters. Third, using the "terminology extraction, single words" functionality of Sketchengine, we extracted the frequent single words. Again, visually checking the output, we defined the list of 2 "Terms" refers to the frequent single word (e.g., firm, business) or multi-word (e.g., social network, management information system) keywords of the corpus. 3 See https://app.sketchengine.eu our stop words (e.g., research, paper, purpose, aim, and objective). Fourth, "information technology," "Information system," and "enterprise system" phrases were removed as we expect to have them in almost all papers, and they do not add any specific value for topic modeling. Finally, actions such as converting words to their singular forms, tokenization, lemmatization, and synonyms were performed (using N L T K library of Python) to prepare the data set for analysis. 3.3 Process of topic modeling Topic modeling is a quantitative method for assessing textual data and uses statistical methods to extract semantic information from a text corpus. Hofmann [49] proposed the first topic model, probabilistic Latent Semantic Indexing (pLSI). Further developments have been presented and implemented to optimize the topic modeling algorithms [50]. There are various open-source programming and commercial software packages available for these models, such as Gensim and Scikit-learn packages in Python, tidytext in R, SAS Text Miner, and Leximancer [51]. Gensim, presented by Rehurek and Sojka [52], is an open-source topic modeling toolkit in Python that aims to extract the semantic topics from the corpus using data streaming and incremental algorithms. It includes multiple algorithms such as L D A , RP, L S A , TF-IDF, hierarchical Dirichlet processes (HDPs), LSI, and singular value decomposition (SVD) [53]. We employed the LdaModel in Gensim for topic modeling and then created visualizations using the "ggplot2", "matplotlib" and "pyLDAvis" libraries. The optimum number of topics is also estimated based on perplexity and log-likelihood. 4 Topic modeling and visualization The topic modeling strategy is illustrated in Figure 4. The main analysis was based on the analysis of abstracts for all papers published in all years. Additional analyses were also conducted based on the publication year to better understand the evolution of ISBV topics over time. Accordingly, all articles were divided into three subgroups based on the publication year: 1990-2000 (#320), 2001-2010 (#874), and 2011-2020 (#1630). Main Analysis Subgroup Analysis Articles in All Years (#2824) Articles in 1990-2000 (#320) Articles in 2001-2010 (#874) Articles in 2011-2020 (#1630) Figure 4. Topic modeling strategy 4.1 Main Analysis Ten topics were uncovered in the main analysis. Some emergent topics seem to be consistent with the dominant themes of IS research and not specific to ISBV, such as 'IS adoption,' 'IS implementation,' 'IS management,' 'IS usage,' 'IS capability,' and 'IS strategy'. Some others, such as 'e-health,' 'e-learning,' and 'IS public services,' refer to specific IS applications in different domains. The top 10 keywords for each topic are provided in Table 2, along with the distribution of papers by publication year. Table 2. Topics in all years, with distribution in years. Topics (% of corpus) Top ten contributing terms to the topic 1990 Publication year* 2000 2010 2020 e-health (12.0) IS capability (11.8) IS adoption & use (10.3) IS strategy (9.7) IS implementation (8.3) IS productivity (7.8) e-learning (6.7) IS Social practices (6.4) IS public services (6.3) IS design (5.8) IS management (5.4) IS evaluation (5.1) IS project management (4.5) healthcare, patient, health_information_system, health, hospital, practice, nurse, medical, clinical, electronic performance, capability, firm, resource, integration, competitive_advantage, competency, collaboration, dynamic, resource_based_view user, adopt, usage, perceive, behavior, intention, satisfaction, perception, acceptance, attitude investment, strategy, firm, market, financial, asset, decision, return, alignment, governance implement, process, project, success, management, enterprise_resource_planning, plan, team, stakeholder, failure productivity, knowledge, human_resource, innovation, technological, efficiency, regression, analysis, capital, economy learn, university, student, experience, science, academic, education, train, content, skill organization, change, work, practice, structure, culture, leadership, commitment, transformation, conflict resource, geographic_information_system, environmental, city, local, water, state, land, region, spatial service, application, process, network, design, integrate, infrastructure, task, requirement, solution management, risk, analysis, small_and_medium_size_enterprise, function, operation, communication, assessment, audit, institutional benefit, evaluation, tool, analysis, evaluate, decision_support, decision, problem, solution, sustainability cost, time, control, product, policy, social, communication, efficiency, human, worker note : * the range of the Y-axis for this column is fixed at 0-40. According to Table 2, 'e-health,' 'IS capability,' and 'IS use' are the three top topics encompassing over one-third of the whole corpus. The popularity of topics such as 'IS project management,' 'IS evaluation,' and 'IS Social practices' has stagnated throughout the period. In comparison, topics such as 'IS adoption,' 'e-health,' 'e-learning,' 'IS capability,' and 'IS use' gained momentum in recent years. Further visualization was done using the pyLDAvis library, which allows the user to look at individual topics while keeping the big picture in view and is thus helpful in interpreting and labeling topics. Figure 5 shows the result for topic#3: e-health. Selected Topic: 0 (Previous Topíc 11 Next Topic Clear Topic! 9 A - 1 o.tj .2 0.4 0.6 Intertopic Distance Map (via multidimensional scaling) Top-30 Most Relevant Terms for Topic 3 (9.1 % oftokens) O 5 4 » 1.000 1.54» 2.000 2.5O0 . PC2 h&HIth_ i nformati s n_syst° m -=ill- ;-actio; et al {2012) Figure 5. Interactive LDAVis presentation of topics (topic#3, e-health) 4.2 Subgroup Analysis 4.2.1 Analysis of articles published in 1990-2000 The analysis includes 320 articles accounting for 11.3 percent of the main corpus. The revealed nine topics are shown in Table 3. Topics of 'IS strategy,' 'IS productivity,' 'IS design,' 'IS management,' 'IS implementation,' 'IS evaluation,' and 'IS public services' are similar to the topics in the main analysis. 'IS adoption' and 'IS use' emerged as distinct topics in the subgroup analysis, but are integrated into a single 'IS adoption & use' topic in the main analysis. 'IS public services' had relatively more articles compared to the main analysis as it incorporates 'e-health' and 'e-learning' related terms that emerged as distinct topics in subsequent years (described later). It makes sense as both 'e-health' and 'e-learning' topics have been immature during this period. Similarly, 'IS implementation' entails terms relate to 'IS project management' topic of the main analysis. Finally, we miss 'IS capability' topic here, as the concept developed later and made popular inspired by the works of Bharadwaj [54], and Santhanam and Hartono [55]. Studies in this period heavily focused on 'IS evaluation' and 'IS productivity' topics as a response to "IS productivity paradox", accounting for nearly one-third of the total articles. There have been contradicting recommendations during this period in terms of ISBV. While some doubted ISBV [e.g., 4, 6], others suggested a direct positive relationship between IS and firm performance [e.g., 7-9]. This paradox led scholars in later years to look for potential sources of such contradicting results [e.g., 17,41]. Table 3. Topics in 1990-2000, with distribution in years Topics (% of corpus) Top ten contributing terms to the topic Publication year* 1990 2000 IS evaluation (15.9) IS productivity (14.4) IS strategy (12.2) IS implementation (11.9) IS adoption (11.6) IS public services (10.9) IS use (9.7) IS management (6.9) IS design (6.6) benefit, cost, process, project, decision, issue, evaluation, problem, investment, decision_support investment, performance, firm, organization, productivity, strategy, return, economic, competitive_advantage, financial strategy, industry, software, market, product, function, customer, competitive, global, demand implement, organization, change, work, process, management, technical, structure, environment, communication adopt, organization, innovation, knowledge, characteristic, practice, individual, diffusion, attitude, behavior healthcare, health, education, service, hospital, alignment, health_information_system, patient, learn, clinical user, computer, application, success, usage, perception, experience, perceive, task, acceptance management, resource, service, access, tool, environmental, agency, requirement, database, library analysis, time, integration, electronic, capability, integrate, analyze, profit, price, produce note : * the range of the Y-axis for this column is fixed at 0-15. 4.2.2 Analysis of articles published in 2001-2010 The analysis includes 874 articles accounting for 31 percent of the main corpus. The revealed ten topics for articles published during the period are shown in Table 4. Compared to the main analysis, we have exact topics here; however, comparing topics for 1990-2000 and 2001-2010 reveals some similarities and differences. Excluding 'IS evaluation' of 1990-2000 period that is missed here and 'IS adoption,' and 'IS use' that integrated into 'IS adoption & use' in 2001-2010 period, we have all similar topics from the previous period. Nevertheless, three additions are 'ehealth,' 'IS social practices,' and 'IS capability,' as a result of advancements in e-health discipline, the increasing importance of social and human factors in IS implementation, and the capabilities required to gain the most business value of IS initiatives. The emerging 'IS capability' topic has gained the most attention during this period with 13.5 percent of the total published articles, followed by 'IS productivity' with 12.2 percent. Taking into account the IS capabilities of IS adopters and also context-related factors (here partially revealed as 'IS social practices'), most studies in this period reported positive impacts of IS initiatives on business performance [e.g., 10, 55, 56]. Table 4. Topics in 2001-2010, with distribution in years Topics (% of corpus) Top ten contributing terms to the topic Publication year* 2001 2010 IS capability (13.5) IS productivity (12.2) IS adoption & use (11.8) e-health (10.4) IS implementation (10.3) IS strategy (10.2) IS design (9.6) IS public services (8.1) performance, firm, capability, market, innovation, product, infrastructure, competitive_advantage, resource, flexibility investment, cost, productivity, capital, evaluation, financial, efficiency, time, return, benefit user, usage, adopt, perceive, individual, behavior, intention, social, perception, acceptance healthcare, hospital, patient, health, practice, health_information_system, service, clinical, nurse, medical implement, knowledge, success, plan, practitioner, challenge, issue, failure, team, executive organization, change, benefit, strategy, network, supply_chain, initiative, supplier, source, collaboration process, management, project, risk, problem, issue, decision_support, control, solution, requirement adopt, human_resource, government, electronic, application, public, policy, state, digital, environment IS management (7.0) IS social practices (6.9) resource, analysis, application, decision, tool, structure, outsource, uncertainty, operation, environmental work, design, culture, communication, global, learn, train, cultural, human, power note : * the range of the Y-axis for this column is fixed at 0-25. 4.2.3 Analysis of articles published in 2011-2020 The analysis includes 1,630 articles that account for 58 percent of the main corpus. The revealed 12 topics are presented in Table 5. Certain topics were similar to the main analysis; however, a comparison of topics that emerged here and in other periods and the main analysis reveals some similarities and differences. Firstly, this period is the only period that 'IS strategy' has not appeared as a distinct topic. A closer look shows that related terms to 'IS strategy,' such as strategy and alignment, are already incorporated into the 'IS capability' topic. Second, the 'elearning' topic emerges during this period thanks to the growing interest in IS value for the education and academic community. This finding is in line with the results of the main analysis that shows only a few 'e-learning' related publications during the early years. Third, two new topics appeared here that did not show up in other analyses as 'IS emerging services' and 'IS sustainability'. This is quite interesting to see that recent ISBV studies examined the role of specific emerging technologies such as cloud computing, business intelligence, RFID, and IoT (top terms according to Table 5) in creating business value. Another interesting emerging topic is 'IS sustainability'. Surprisingly, a significant part of ISBV literature focuses on sustainability issues due to ever-increasing environmental issues and the supporting role of green IT and sustainable business models. Overall, 'IS capability' still stands in the first rank in terms of the frequency of publications, followed by 'e-health' and 'IS adoption & use'. 'IS productivity' is ranked fourth, indicating that "the gray box" of ISBV is not completely unlocked yet, an issue that is highlighted in some of the recent ISBV reviews [e.g., 1]. Table 5. Topics in 2011-2020, with distribution in years Topics (% of corpus) Top ten contributing terms to the topic Publication year* 2011 2020 IS capability (13.3) e-health (12.6) IS adoption & use (11.8) capability, strategy, innovation, dynamic, competitive_advantage, governance, competency, alignment, agility, resource_based_view healthcare, health_information_system, patient, health, hospital, electronic, nurse, medical, clinical, record adopt, user, usage, behavior, perceive, intention, individual, satisfaction, acceptance, attitude IS productivity (9.4) investment, firm, financial, productivity, market, capital, economic, growth, return, accounting IS social practices (8.2) organization, management, human_resource, employee, change, culture, practice, structure, human, leadership IS sustainability (7.9) IS evaluation (6.8) industry, green, sustainability, power, collaboration, sustainable, product, environmental, operational, flexibility analysis, application, tool, risk, evaluation, management, assessment, decision_support, evaluate, solution IS management (6.8) e-learning (6.6) cost, time, efficiency, control, security, asset, standard, operation, monitor, compliance learn, university, student, institution, education, academic, content, library, train, science IS implementation implement, process, project, design, software, successful, (6.1) problem, stakeholder, technical, structure IS emerging services (5.4) service, resource, internet, interaction, online, cloud, emerge, device, rfid, intelligence IS public services (5.3) country, network, plan, policy, government, state, audit, national, public, city note : * the range of the Y-axis for this column is fixed at 0-35. 5 Post hoc analysis 5.1 Trend Analysis The main analysis and three subgroup analyses revealed the dominant topics and their distribution through the periods. We conducted a post hoc analysis of the total number of documents within each topic and the relative weight within the given period. In general, the number of publications increased for most topics over time; however, only a few topics grew in relative popularity based on the slopes4 analyzed. The slope showed whether a topic had an upward (i.e., positive slope) or downward (i.e., negative slope) trend. Table 6 depicts the status of the 17 unique topics in the main and subgroup analyses, in which / and \ denote the upward and downward trends, respectively. Significance levels reported in the table indicate if the observed positive or negative trend is statistically significant or not. Blank cells in the table refer to the topics that did not emerge in the relevant time period. As 'IS adoption' and 'IS use' are integrated into 'IS adoption & use' topic in 1990-2000 period, we grouped them in the table for better visualization. Topics of the subgroups analyses are all observed in the main analysis, excluding 'IS emerging services' and 'IS sustainability' topics of the 2011-2020 period. Simple slope analysis of the linear model is conducted in R (see "lm" function) Table 6. The overall trend of topics across different analyses Topic All Years 1990-2000 2001-2010 2011-2020 IS strategy — — IS productivity — — — — e-health IS project management IS design — — e-learning IS capability — — IS management — — — IS implementation — — — IS evaluation — — IS public services — — — IS adoption & use IS adoption IS use IS Social practices IS emerging services IS sustainability 0.001 '***'; 0.01 '**'; 0.05 '*'; — 'non-significant'; blank cells mean that the topic is not applicable for the associated year. For the main analysis, different trends have been observed. Uptrends are revealed only for three topics of 'e-health,' 'IS capability,' and 'IS adoption & use'. On the contrary, 'IS strategy,' 'IS project management,' 'IS implementation,' 'IS evaluation,' 'IS public services,' and 'IS Social practices' showed a negative slope. The remaining four topics of 'IS productivity,' 'IS design,' 'e-learning,' and 'IS management' demonstrate no statistically significant positive or negative trends. Nevertheless, the slope analysis results for the topics are more consistent across the subgroups. In the analysis for "Articles in 1990-2000" we observe no statistically significant trend. The same is valid for the analysis of "Articles in 2001-2010" excluding 'IS adoption & use' which shows a strong positive trend. For the analysis of "Articles in 2011-2020", only 'ehealth' and 'e-learning,' show upward trends while the others are not significant. 5.2 Topics intra-period analysis We also examined the dominance of topics within the three periods to determine if there are significant differences in terms of the frequency of papers assigned to each topic. This analysis helps determine if the frequency/percentage of each topic in Tables 2-5 indicates a difference in the dominance of topics. We first conducted a one-way A N O V A test to gauge if there is a statistically significant difference between the means of three or more independent topics. We then used the Tukey HSD/Kramer test5 , which compares the means between each pairwise combination of groups, to find which groups are different. We ran this procedure for the main analysis and three sub-group analyses separately using 'Real Statistics'6 . These results are reported in the supplementary materials for the paper. For the main analysis, the derived p-value of the A N O V A test is significant at the 0.01 level, indicating significant differences in the means for at least one of the topics. The Tukey HSD/Kramer test shows that the difference is significant for 'e-health' with 'IS evaluation' and 'IS for Public'. The same holds for the 2011-2020 period analysis, in which the p-value is significant at the 0.01 level and the Tukey HSD/Kramer test shows the difference in a handful number of topics. For the other two subgroup analyses (1990-2000 and 2001-2010), given the achieved p-value greater than 0.05, we concluded no significant differences in the means for the topics, which was also confirmed by the Tukey HSD/Kramer tests. Overall, for the main analysis, the top three topics (i.e., 'e-health,' 'IS capability,' and 'IS adoption & use') have higher means than one or more topics with the lower frequencies in Table 2. Such differences were significant for 'e-health' with 'IS public services,' 'IS design,' 'IS 5 See https://www.real-statistics.coin/one-way-analysis-of-variance-anova/unplann^ 6 https://www.real-statistics.com provides capabilities not found in Excel and easier to use than SPSS. management,' 'IS evaluation,' and 'IS project management'; 'IS capability' with 'IS design,' 'IS management,' 'IS evaluation,' and 'IS project management'; and for 'IS adoption & use' with 'IS project management'. For the subgroup analysis for 2011-2020, all top three topics (i.e., 'IS capability,' 'e-health,' and 'IS adoption & use') show higher means than those for the last six topics (i.e., 'IS evaluation,' 'IS management,' 'e-learning,' 'IS implementation,' 'IS emerging services,' and 'IS public services'). 6 Discussion 6.1 Findings Figure 6 is derived by consolidating the results of all subgroup analyses for three periods. The three digital eras of SAP are also embedded in the figure as client, server, and internet era; cloud, mobile, and Big Data era; and intelligent technologies era. The technologies in each era pose different opportunities for business value creation, including business process automation, digital transformation, and intelligent enterprise. The topics from our analysis are shown for each period. Colored boxes indicate topics with upward trends in Table 6. Arrows indicate the evolution of each topic or how a topic is integrated into another topic (based on the similarity of the terms within topics). For instance, the connection from 'IS public services' in the first era to 'e-health' in the second era shows that the latter topic is emerged or evolved from the first topic. As another example, the arrow from 'IS strategy' in the second era to 'IS capability' in the third era shows that the first topic is integrated into the second one during the period.