Espacios. Vol. 36 (Nº 13) Año 2015. Pág. 11

Meta-analysis of the DeLone and McLean information systems success model at individual level: An examination of the heterogeneity of the studies

Meta-análisis del modelo de éxito de sistemas de información de Delone y McLean a nivel individual: Un examen de la heterogeneidad de los estudios

Patricio RAMÍREZ-CORREA 1, Jorge ALFARO-PERÉZ 2, Linda CANCINO-FLORES 3

Recibido: 19/03/15 • Aprobado: 26/04/2015

Contenido

1. Introduction

2. Literature review

3. Methodology

4. Results

5. Conclusions

References

Appendix: List of Included Studies

1. Introduction

The successful implementation of information technology is considered a key issue in the discipline of information systems (IS). However, perhaps because of the imprecision and broadness of the concept, several authors have operationalized this success in dissimilar manners within their studies. As a way to integrate all these piecemeal, two decades ago the IS success model of DeLone and McLean (D&M) (DeLone & McLean, 1992) arises.

Since its proposal the D&M model has been extensively validated in diverse technological contexts (DeLone & McLean, 2003), becoming recognized as one of the foundations of the body of knowledge of the discipline (Petter et al., 2013). In fact, both the original version of D&M of 1992 and its extension in 2003, appear among the most cited articles of the discipline in the past decade (Stein et al., 2014).

Because of its clear importance in IS, and to identify patterns among multiple studies where the model D&M  has been applied, several meta-analysis (MA) have been conducted (Hwang & Windsor, 1996; Mahmood et al. 2001; Bokhari, 2005; Petter & McLean, 2009).

Despite the fact that the literature emphasizes the importance of analyzing the heterogeneity in studies MA, in general, the authors of the MA on the D&M model do not include or omit the relevance of this analysis, minimizing the consequences of this in their conclusions. Specifically, if there is heterogeneity between studies of a MA, it cannot determine the generalizability of their results, and therefore, this MA study does not provide information relevant to the discipline (Higgins et al., 2003).

In this context, this study aims the analysis of heterogeneity in a MA of D&M model at the individual level. This paper is structured in the following way.  Firstly, a literature review is presented.  Secondly, the methodology of the study is explained. Finally, the results are described and conclusions are presented.

2. Literature review

2.1. D&M model

The original D&M model was proposed in 1992 as a causal-explanatory model for a dependent variable that its authors named the IS success (DeLone & McLean, 1992). This model is the result of detailed research to determine possible constructs that affect the IS success and is based on exploration, categorization and synthesis of dependent variables that were used in 180 studies between 1981 and 1987 . The original D&M model is a multidimensional proposal for the IS success and includes six interrelated dimensions: System Quality, Information Quality, Use/Intention of Use, User Satisfaction, Individual Impact, and Organizational Impacts. In particular, the model suggests that the dimensions of quality are positively related to the use and satisfaction dimensions, and these latter with the dimensions of impact.

Due to the characteristics of the model, a careful definition and measurement of each dimension of this variable is required. In fact, the selection of dimensions and measures of success should be contingent on the objectives and context of empirical research where it is evaluated. However, and as indicated by the authors, it is important to measure the possible interactions between each of these dimensions to isolate the effect of several independent variables with one or more of these dependent dimensions of the success (DeLone & McLean, 1992).

After its proposal, the D&M model has been used by many authors, and diverse studies have validated empirical relationships between their dimensions (Ramírez-Correa, 2004).

In 2003 the authors of the D&M model presented a review of it, in which, firstly, it joins the dimensions of impacts on the dimension Net Benefits, and secondly, added the dimension Service Quality (DeLone & McLean, 2003).

Next, the dimensions of the D&M model are described.

1. SYSTEM QUALITY (SQ): Refers to the quality and overall performance of the IS itself, and can be determined by the quality of its technical (hardware / software), and components that are essential for the capture, processing, storage and retrieval of data.
2. INFORMATION QUALITY (IQ): These are the desirable characteristics of the outputs of a IS, and reflect, inter alia, the accuracy, completeness, currency, and the format of this information.
3. SERVICE QUALITY (SerQ): Quality of support received by users of an information system by a support department.
4. USE (U)/INTENTION OF USE (IU): Degree and manner in which individuals (staff, customers, etc.) use the capabilities of an IS (use), and how they consciously formulate plans to continue using these in the future (intention).
5. USER SATISFACTION (US): The level of user satisfaction with the system globally.
6. NET BENEFITS (NB): Represents the degree to which the IS contributes to the success of individuals, groups, organizations and/or nations.

2.2. MA about D&M model

From the recommendations of one of the authors of the D&M model (Hwang & McLean, 1996), there are several MA about the model. Below some elements of these studies are given.

Hwang and Windsor were based on 82 empirical studies selected from 180 studies that were used to develop the D&M model, determined the size effect on the relationships between the dimensions of the model. The results of this MA validated all dimensions of the model, except for IQ, and although only some relationships between these dimensions were tested, it is concluded that, mostly, the dimensions do not show a large effect (Hwang & Windsor, 1996). In this paper the analysis of the heterogeneity of the studies were omitted.

Subsequently, using the same studies of a previous work, Hwang, Windsor, and Pryor correlate some dimensions of the D&M model (valued through of a MA) to five independent variables of  IS success (Hwang et al., 2000).

One of the results of the MA of Mahmood, Hall, and Swanberg indicates a strong effect between SerQ and U in the D&M model (Mahmood et al., 2001). This MA was based on nine studies and their result rejects the hypothesis of heterogeneity between them.

Moreover, Bokhari in a MA based on a total of 55 empirical studies identifies a medium effect size between U and US (Bokhari, 2005). The result of this meta-analysis could not reject the hypothesis of heterogeneity between studies.

The MA of Sabherwal, Jeyaraj, and Chowa based on a total of 71 studies supports the existence of effects both between SQ and U, and between QS and US (Sabherwal et al., 2006). In this MA was not analyzed the heterogeneity of the studies.

Finally, Petter and McLean through a MA based on 52 empirical studies at the individual level found support most of the relationships in the D&M model (Petter & McLean, 2009). Specifically, their findings indicate that there are strong relationships between the dimensions US-IU, NB-IU, QS-US, US-NB, IQ-US, IQ-IU, and  SQ-IU.

In addition there is moderate relationships between the dimensions IQ-U, U-NB, and SQ-U. And finally, there is a weak relationship between the dimensions U-US. However, although this MA does not explicitly measures the heterogeneity between the studies, the authors suggest that these studies are probably not truly homogeneous.

3. Methodology

3.1. Study Approach

The MA is a quantitative approach that integrates research findings from empirical studies considering the error inherent in any quantitative study (Glass, 1976). In this study, a MA for each relationship proposed in the literature for the D&M model was conducted. In total, 14 MA were carried out. These MA were used to estimate the effect size of the relations between the dimensions of the D&M model. This estimation was performed using the method of Hedges and Olkin (2014), and based on the correlation coefficient. The list of meta-analyzed relationships is as follows:

R1: Between dimensions SQ and IU.

R2: Between dimensions SQ and US.

R3: Between dimensions IQ and IU.

R4: Between dimensions IQ and US.

R5: Between dimensions SerQ and IU.

R6: Between dimensions SerQ and US.

R7: Between dimensions U and US.

R8: Between dimensions US and IU.

R9: Between dimensions U and NB.

R10: Between dimensions US and NB.

R11: Between dimensions NB and IU

R12: Between dimensions SQ and U.

R13: Between dimensions IQ and U.

R14: Between dimensions SerQ and U.

Furthermore, in each MA the heterogeneity of studies through the I2 statistic was assessed. The I2 statistic expresses the percentage of variation across studies that is due to heterogeneity rather than chance (Higgins et al., 2003). StatsDirect software (http://www.statsdirect.com) was used to perform the analyses.

3.2. Collection of Studies

This analysis is based on quantitative empirical studies. These studies analyze the relations contained in the D&M model at the individual level. As a starting point for collecting these studies the MA of Peter and McLean (Petter and McLean, 2009) was used. This database consists of 52 studies published between 1992 and mid 2007. In addition, and through databases Scopus and Web of Knowledge, 27 studies published since mid-2007 to 2014 were identified. In total, this MA collected 79 quantitative empirical studies published between 1992 and 2014.

4. Results

A total of 83 studies contained in the collected studies were analyzed, the appendix shows the tabulation of them. Table 1 shows the results of the MA of each studied relationship (R1 to R14).

r: pooled correlation; K: studies number included in meta-analysis;
N: total sample size for meta-analysis; I2: heterogeneity index in percentage
Table 1. Results of the meta-analyses.

Using heuristics to judge the magnitude of the effect given by Cohen et al. (Cohen et al., 2013), it is possible to indicate that the majority D&M model relationships are strong or moderate.  Specifically, these results indicate that there are seven strong relationships (R2, R3, R4, R6, R8, R10, and R11), six moderate relationships (R1, R5, R7, R12, R13, and R14) and a weak relationship (R9).

However, in all meta-analyses the I2 far exceeds to 75%, which indicates a high heterogeneity in the studies analyzed (Higgins et al., 2003). Therefore, the results of this MA cannot be generalized.

5. Conclusions

This study confirms the potential usefulness of MA to observe the convergence of the investigation of phenomena in the discipline of IS.  

Nevertheless, MA requires following a rigorous process. In this pathway, an essential point, which enables the generalization of the results, is to assess the degree of heterogeneity between meta-analyzed studies. The findings of this study indicate that, due to the high heterogeneity found in empirical studies between 1992 and 2014, it is not possible to generalize the results of the meta-analysis of the D&M model at the individual level.

Given the results of this study and the important of IS success for discipline, we appeal to carefully consider all sorts of generalizations arising from previous meta-analyses of the D&M model, which do not expose explicitly the degree of heterogeneity of the studies collected.

One of the limitations of this study is that it does not consider the statistical corrections associated to the reliability of scales measuring the dimensions of the D&M model; in future studies we hope to resolve this limitation. Additionally, future research should look for moderating variables in the relations contained on the model, allowing to establish subsets of homogeneous studies.

References

ABDULGADER, A. (1997); "Determinants of computer-mediated communication success among knowledge workers in Saudi Arabia"; Journal of Computer Information Systems, 38(1), 55-66.

AGARWAL, R.,  Prasad, J. (1997); "The role of innovation characteristics and perceived voluntariness in the acceptance of information technologies"; Decision sciences, 28(3), 557-582.

AGARWAL, R.,  Prasad, J. (1999); "Are individual differences germane to the acceptance of new information technologies?"; Decision sciences, 30(2), 361-391.

AININ, S., Bahri, S.,  Ahmad, A. (2012); "Evaluating portal performance: A study of the National Higher Education Fund Corporation (PTPTN) portal"; Telematics and Informatics, 29(3), 314-323.

ALADWANI, A. M. (2002); "Organizational Actions, Computer Attitudes, and End-User Satisfaction in Public Organizations: An Empirical Study"; Journal of Organizational and End User Computing, 14(1), 42-49.

AL-DEBEI, D.,  Mutaz, M. (2013); "The quality and acceptance of websites: an empirical investigation in the context of higher education"; International Journal of Business Information Systems, 15(2), 170-188.

AL-DEBEI, M. M., Jalal, D.,  Al-Lozi, E. (2013); "Measuring web portals success: a respecification and validation of the DeLone and McLean information systems success model"; International Journal of Business Information Systems, 14(1), 96-133.

ALMUTAIRI, H.,  Subramanian, G. H. (2005); "An empirical application of the DeLone and McLean model in the kuwaiti private sector"; Journal of Computer Information Systems, 45(3);113+.

ANANDARAJAN, M., Igbaria, M.,  Anakwe, U. P. (2002); "IT acceptance in a less-developed country: a motivational factor perspective"; International Journal of Information Management, 22(1), 47-65.

ARENAS-GAITAN, J., Ramirez-Correa, P. E.,  Rondan-Cataluna, F. J. (2011); "Cross cultural analysis of the use and perceptions of web Based learning systems"; Computers &  Education, 57(2), 1762-1774.

AYYASH, M. M., Ahmad, K.,  Singh, D. (2013); "Investigating the Effect of Information Systems Factors on Trust in E-Government Initiative Adoption in Palestinian Public Sector". Research Journal of Applied Sciences, Engineering and Technology, 5(15), 3865-3875.

BERGERON, F., Raymond, L., Rivard, S.,  Gara, M.-F. (1995); "Determinants of EIS use: Testing a behavioral model"; Decision Support Systems, 14(2), 131-146.

BIN MASREK, M. N. (2007); "Measuring campus portal effectiveness and the contributing factors"; Campus-Wide Information Systems, 24(5), 342-354.

BOKHARI, R. H. (2005); "The relationship between system usage and user satisfaction: a meta-analysis"; Journal of Enterprise Information Management, 18(2), 211-234.

CARSWELL, A. D.,  Venkatesh, V. (2002); "Learner outcomes in an asynchronous distance education environment"; International Journal of Human-Computer Studies, 56(5), 475-494.

CHEN, C.-W.,  Cheng, C.-Y. (2013); "How online and offline behavior processes affect each other: customer behavior in a cyber-enhanced bookstore"; Quality  Quantity, 47(5), 2539-2555.

CHEN, C.-W.,  Cheng, C.-Y. J. (2009); "Understanding consumer intention in online shopping: a respecification and validation of the DeLone and McLean model"; Behaviour  Information Technology, 28(4), 335-345.

CHEN, C.-Y., Shih, B.-Y.,  Yu, S.-H. (2012); "Disaster prevention and reduction for exploring teachers' technology acceptance using a virtual reality system and partial least squares techniques"; Natural hazards, 62(3), 1217-1231.

CHEN, I. Y. (2007); "The factors influencing members' continuance intentions in professional virtual communities- a longitudinal study"; Journal of Information Science, 33(4), 451-467.

CHEN, I. Y. (2009); "Social Capital, IT Capability, and the Success of Knowledge Management Systems"; Knowledge Management & E-Learning: An International Journal, 1(1), 36-50.

CHEN, J. V., Rungruengsamrit, D., Rajkumar, T.,  Yen, D. C. (2013); "Success of electronic commerce Web sites: A comparative study in two countries"; Information  Management, 50(6), 344-355.

COHEN, J., Cohen, P., West, S. G.,  Aiken, L. S. (2013); "Applied multiple regression/correlation analysis for the behavioral sciences"; Routledge.

DELONE, W. H.,  McLean, E. R. (1992); "Information systems success: the quest for the dependent variable"; Information systems research, 3(1), 60-95.

DELONE, W. H.,  McLean, E. R. (2003); "The DeLone and McLean model of information systems success: a ten-year update"; Journal of Management Information Systems, 19(4), 9-30.

DONG, T. P., Cheng, N. C.,  Wu, Y. C. J. (2014); "A study of the social networking website service in digital content industries: The Facebook case in Taiwan"; Computers in Human Behavior, 30, 708-714.

FLOROPOULOS, J., Spathis, C., Halvatzis, D.,  Tsipouridou, M. (2010); "Measuring the success of the Greek taxation information system"; International Journal of Information Management, 30(1), 47-56.

GARRITY, E. J., Glassberg, B., Kim, Y. J., Sanders, G. L.,  Shin, S. K. (2005); "An experimental investigation of web-based information systems success in the context of electronic commerce"; Decision Support Systems, 39(3), 485-503.

GATIAN, A. W. (1994); "Is user satisfaction a valid measure of system effectiveness?"; Information  Management, 26(3), 119-131.

GLASS, G. V. (1976); "Primary, secondary, and meta-analysis of research"; Educational researcher, 5, 3-8.

HEDGES, L. V., Olkin, I. (2014); "Statistical method for meta-analysis"; Academic press.

HIGGINS, J. P., Thompson, S. G., Deeks, J. J.,  Altman, D. G. (2003); "Measuring inconsistency in meta-analyses"; BMJ: British Medical Journal, 327(7414), 557-560.

HSU, M. H., Chiu, C. M.,  Ju, T. L. (2004); "Determinants of continued use of the WWW: an integration of two theoretical models"; Industrial management  data systems, 104(9), 766-775.

HUNG, W.-H., Chang, L.-M., Yen, D. C., Ho, C.-T.,  Chiang, M.-C. (2011); "ERP Success in the SMEs: The Perspectives of Service Quality and Social Cognitive Theory"; Asia Pacific Management Review, 16(4), 503-519.

HUSSEIN, R., Karim, N. S. A.,  Selamat, M. H. (2007); "The impact of technological factors on information systems success in the electronic-government context"; Business Process Management Journal, 13(5), 613-627.

HWANG, M. I.,  Windsor, J. C. (1996); "Validating the DeLone and McLean information systems success model: A meta-analysis"; Proceedings of the 27th Annual Meeting of the Decision Sciences Institute.

HWANG, M. I., Windsor, J. C.,  Pryor, A. (2000); "Building a knowledge base for MIS research: A meta-analysis of a systems success model"; Information Resources Management Journal, 13(2), 26-32.

HWANG, M.,  McLean, E. R. (1996); "The use of meta-analysis in validating the DeLone and McLean information systems success model"; Proceedings of the Twenty-Ninth Hawaii International Conference on System Sciences.

IGBARIA, M.,  Guimaraes, T. (1994); "Empirically testing the outcomes of user involvement in DSS development"; Omega, 22(2), 157-172.

IGBARIA, M.,  Tan, M. (1997); "The consequences of information technology acceptance on subsequent individual performance"; Information  Management, 32(3), 113-121.

JEN, W.-Y.,  Chao, C.-C. (2008); "Measuring mobile patient safety information system success: an empirical study"; International Journal of Medical Informatics, 77(10), 689-697.

JONES, R. A., Jimmieson, N. L.,  Griffiths, A. (2005); "The impact of organizational culture and reshaping capabilities on change implementation success: The mediating role of readiness for change"; Journal of Management Studies, 42(2), 361-386.

JUNGLAS, I., Goel, L., Abraham, C.,  Ives, B. (2013); "The Social Component of Information Systems—How Sociability Contributes to Technology Acceptance"; Journal of the Association for Information Systems, 14(10), 585-616.

KHALIL, O. E.,  Elkordy, M. M. (1999); "The relationship between user satisfaction and systems usage: empirical evidence from Egypt"; Journal of Organizational and End User Computing, 11(2), 21-28.

KIM, C., Suh, K.,  Lee, J. (1998); "Utilization and User Satisfaction in End-User Computing: A Task Contingent Model"; Information Resources Management Journal, 11(4), 11-24.

KIM, H. W., Chan, H. C.,  Gupta, S. (2007); "Value-based adoption of mobile internet: An empirical investigation"; Decision Support Systems, 43(1), 111-126.

KONRADT, U., Christophersen, T.,  Schaeffer-Kuelz, U. (2006); "Predicting user satisfaction, strain and system usage of employee self-services"; International Journal of Human-Computer Studies, 64(11), 1141-1153.

KULKARNI, U. R., Ravindran, S.,  Freeze, R. (2007); "A knowledge management success model: theoretical development and empirical validation"; Journal of Management Information Systems, 23(3), 309-347.

LAI, J.-Y.,  Yang, C.-C. (2009); "Effects of employees' perceived dependability on success of enterprise applications in e-business"; Industrial Marketing Management, 38(3), 263-274.

LAI, J.-Y., Wang, C.-T.,  Chou, C.-Y. (2009); "How knowledge map fit and personalization affect success of KMS in high-tech firms"; Technovation, 29(4), 313-324.

LEE, K. C.,  Chung, N. (2009); "Understanding factors affecting trust in and satisfaction with mobile banking in Korea: A modified DeLone and McLean's model perspective"; Interacting with computers, 21(5), 385-392.

LEE, S.-K.,  Yu, J.-H. (2012); "Success model of project management information system in construction"; Automation in Construction, 25, 82-93.

LIAO, C., Palvia, P.,  Lin, H.-N. (2006); "The roles of habit and web site quality in e-commerce"; International Journal of Information Management, 26(6), 469-483.

LIAW, S.-S.,  Huang, H.-M. (2003); "An investigation of user attitudes toward search engines as an information retrieval tool"; Computers in human behavior, 19(6), 751-765.

LIGHTNER, N. J. (2003); "What users want in e-commerce design: effects of age, education and income"; Ergonomics, 46(1-3), 153-168.

LIN, H.-F. (2007); "Measuring online learning systems success: applying the updated DeLone and McLean model"; Cyberpsychology  Behavior, 10(6), 817-820.

LIN, H.-F. (2008); "Determinants of successful virtual communities: Contributions from system characteristics and social factors"; Information  Management, 45(8), 522-527.

LIN, J.,  Lu, H. (2000); "Towards an understanding of the behavioural intention to use a web site"; International journal of information management, 20(3), 197-208.

LIU, C.,  Arnett, K. P. (2000); "Exploring the factors associated with Web site success in the context of electronic commerce"; Information  Management, 38(1), 23-33.

LU, H.-P., Yu, H.-J.,  Lu, S. S. (2001); "The effects of cognitive style and model type on DSS acceptance: An empirical study"; European Journal of Operational Research, 131(3), 649-663.

LWOGA, E. T. (2013); "Measuring the success of library 2.0 technologies in the African context: The suitability of the DeLone and McLean's model"; Campus-Wide Information Systems, 30(4), 288-307.

MAES, A.,  Poels, G. (2007); "Evaluating quality of conceptual modelling scripts based on user perceptions"; Data  Knowledge Engineering, 63(3), 701-724.

MAHMOOD, M. A., Hall, L.,  Swanberg, D. L. (2001); "Factors affecting information technology usage: A meta-analysis of the empirical literatura"; Journal of Organizational Computing and Electronic Commerce, 11(2), 107-130.

MARBLE, R. P. (2003); "A system implementation study: management commitment to project management"; Information  Management, 41(1), 111-123.

MUN, H. J., Yun, H., Kim, E. A., Hong, J. Y.,  Lee, C. C. (2010); "Research on factors influencing intention to use DMB using extended IS success model"; Information Technology and Management, 11(3), 143-155.

PAI, F.-Y.,  Huang, K.-I. (2011); "Applying the Technology Acceptance Model to the introduction of healthcare information systems"; Technological Forecasting and Social Change, 78(4), 650-660.

PARK, S., Zo, H., Ciganek, A. P.,  Lim, G. G. (2011); "Examining success factors in the adoption of digital object identifier systems"; Electronic Commerce Research and Applications, 10(6), 626-636.

PETTER, S.,  Fruhling, A. (2011); "Evaluating the success of an emergency response medical information system"; International Journal of Medical Informatics, 80(7), 480-489.

PETTER, S.,  McLean, E. R. (2009); "A meta-analytic assessment of the DeLone and McLean IS success model: An examination of IS success at the individual level"; Information  Management, 46(3), 159-166.

PETTER, S., DeLone, W.,  McLean, E. R. (2013); "Information Systems Success: The Quest for the Independent Variables"; Journal of Management Information Systems, 29(4), 7-61.

POELMANS, S., Reijers, H. A.,  Recker, J. (2013); "Investigating the success of operational business process management systems. Information Technology and Management", 14(4), 295-314.

RAI, A., Lang, S. S.,  Welker, R. B. (2002); "Assessing the validity of IS success models: An empirical test and theoretical analysis"; Information Systems Research, 13(1), 50-69.

RAMAYAH, T., Ahmad, N. H.,  Hong, T. S. (2012); "An Assessment of E-training Effectiveness in Multinational Companies in Malaysia"; Journal of Educational Technology  Society, 15(2), 125-137.

RAMÍREZ-CORREA, P. (2004); "Rol y Contribución de los Sistemas de Planificación de los Recursos de la Empresa (ERP)"; Tesis doctoral, Sevilla.

RAMIREZ-CORREA, P., Rondán-Cataluña, F. J.,  Arenas-Gaitán, J. (2013); "Exploration of the factors that affect the adoption of social networking services by generation y in Chile"; Interciencia, 38(9), 628-633.

ROCA, J. C., Chiu, C.-M.,  Martínez, F. J. (2006); "Understanding e-learning continuance intention: An extension of the Technology Acceptance Model"; International Journal of human-computer studies, 64(8), 683-696.

SABHERWAL, R., Jeyaraj, A.,  Chowa, C. (2006); "Information system success: individual and organizational determinants"; Management Science, 52(12), 1849-1864.

SAMBASIVAN, M., Wemyss, G. P., Rose, R. C. (2010); "User acceptance of a G2B system: a case of electronic procurement system in Malaysia"; Internet Research, 20(2), 169-187.

SEDDON, P. B., & Kiew, M.-Y. (1996); "A partial test and development of DeLone and McLean's model of IS success", Australasian Journal of Information Systems, 4(1). 90-109.

SØRUM, H., Medaglia, R., Andersen, K. N., Scott, M., & DeLone, W. (2012); "Perceptions of information system success in the public sector: Webmasters at the steering wheel?"; Transforming Government: People, Process and Policy, 6(3), 239-257.

STEIN, M.-K., Galliers, R. D., & Whitley, E. A. (2014); "Twenty years of the European information systems academy at ECIS: emergent trends and research topics"; European Journal of Information Systems. Advance online publication 26 August 2014; doi: 10.1057/ejis.2014.25.

SUN, J., & Teng, J. T. (2012); "Information Systems Use: Construct conceptualization and scale development"; Computers in Human Behavior, 28(5), 1564-1574.

TEO, H.-H., Chan, H.-C., Wei, K.-K., & Zhang, Z. (2003); "Evaluating information accessibility and community adaptivity features for sustaining virtual learning communities"; International Journal of Human-Computer Studies, 59(5), 671-697.

TEO, T. S., Srivastava, S. C., & Jiang, L. (2008); "Trust and electronic government success: an empirical study"; Journal of Management Information Systems, 25(3), 99-132.

TORKZADEH, G., & Doll, W. J. (1999); "The development of a tool for measuring the perceived impact of information technology on work"; Omega, 27(3), 327-339.

URBACH, N., Smolnik, S., & Riempp, G. (2010); "An empirical investigation of employee portal success"; The Journal of Strategic Information Systems, 19(3), 184-206.

WANG, W.-T., & Lu, C.-C. (2014); "Determinants of Success for Online Insurance Web Sites: The Contributions from System Characteristics, Product Complexity, and Trust"; Journal of Organizational Computing and Electronic Commerce, 24(1), 1-35.

WANG, W.-T., & Wang, C.-C. (2009); "An empirical study of instructor adoption of web-based learning systems"; Computers & Education, 53(3), 761-774.

WANG, Y.-S., Wang, H.-Y., Shee, D. Y. (2007); "Measuring e-learning systems success in an organizational context: Scale development and validation"; Computers in Human Behavior, 23(4), 1792-1808.

WILKINS, M. (1996); "Determinants of microcomputer usage in the Republic of Ireland"; Journal of End User Computing, 8(4), 3-9.

WINTER, S. J., M Chudoba, K., Gutek, B. A. (1998); "Attitudes toward computers: when do they predict computer use?"; Information & Management, 34(5), 275-284.

WIXOM, B. H., & Todd, P. A. (2005); "A theoretical integration of user satisfaction and technology acceptance"; Information Systems Research, 16(1), 85-102.

YANG, H.-d., & Yoo, Y. (2004); "It's all about attitude: revisiting the technology acceptance model"; Decision Support Systems, 38(1), 19-31.

YOON, Y., & Guimaraes, T. (1995); "Assessing expert systems impact on users' jobs"; Journal of Management Information Systems, 12(1), 225-249.

ZHENG, Y., Zhao, K., Stylianou, A. (2013); "The impacts of information quality and system quality on users' continuance intention in information-exchange virtual communities: An empirical investigation"; Decision Support Systems, 56, 513-524.

Appendix: List of Included Studies  

1. Doctor de la Universidad de Sevilla, Profesor Asociado, Escuela de Ciencias Empresariales, Universidad Católica del Norte (Chile), patricio.ramirez@ucn.cl

2. Doctor de la Universidad de Politécnica de Madrid, Profesor Asociado, Escuela de Ingeniería, Universidad Católica del Norte (Chile), jalfaro@ucn.cl

3. Ingeniero Comercial, Ayudante de Investigación, Escuela de Ciencias Empresariales, Universidad Católica del Norte (Chile), ldcancino@gmail.com