Throughout the design, implementation, and deployment of software, different quality attributes, such as availability, variability, performance, maintenance, testability, modifiability, scalability and usability, are considered with some having more preference over others, depending upon the nature of the application. Usability is one of those quality attributes of a software architecture design, which is always there to be considered, no matter what type of system is going to be designed. This defines the scope of this study. We are focusing on user requirements Throughout the design, implementation, and deployment of software, different quality attributes, such as availability, variability, performance, maintenance, testability, modifiability, scalability and usability, are considered with some having more preference over others, depending upon the nature of the application. Usability is one of those quality attributes of a software architecture design, which is always there to be considered, no matter what type of system is going to be designed. This defines the scope of this study. We are focusing on user requirements and the usability of open source software (OSS). The International Organization for Standardization and the International Electro Technical Commission ISO/IEC 9126-1 [1] places software quality attributes into six categories: functionality, reliability, usability, efficiency, maintainability, and portability. In the standard, usability is defined as “
The use of free and open source software has increased significantly in recent years, mainly due to the accessibility and availability of the Internet. However, software developers and programmers are not the sole users of OSS anymore; the number of non-technical and novice computer users is growing at a fast pace, highlighting the necessity of understanding and addressing their requirements and expectations [2]. Expectations and requirements of these users might not be the same as those of experts and need to be addressed in further studies. OSS is no longer a limited arena for OSS developers and technical contributors alone.
Raza et al. [3], in their empirical study, maintained that the requirements and expectations of OSS users are not the same as before, when software developers were the only OSS users. According to them, in order to improve OSS product quality and usability, OSS designers and developers should comprehend that they are not the eventual users of their project. Bodker et al. [4] also suggest that the right user requirements can help avoiding “
Online forums are the main communication platforms for the users, developers and other contributors of the OSS community. In this study, the role of online forums is empirically examined in relation to the identification and management of user requirements. A dataset of 100 OSS projects of different categories is used to study the research model of this empirical study.
According to Hepting et al. [6], “an
Schwartz and Gunn [9] maintained that for the OSS community, “
Lee et al. [12] carried out an empirical study to measure OSS success. In their proposed model, they identified five determinants for OSS success: software quality, community service quality, OSS use, user satisfaction and individual net benefits. The authors maintained that “
Stam [14] examined the implication of user activities over innovative performance of the organizations using OSS. It was observed that organizations’ contribution to OSS community development is dependent upon end user requirements in addition to other factors, such as source code and technical knowledge. Scacchi [15] comparatively studied the requirements engineering process to characterize how the requirements for OSS are carried out in different OSS projects of different communities. It was found that OSS development involves a social and technical relationship which is not there in the case of traditional software development. The study advocated that “
Breu et al. [16] focused on collaboration between the developers and the users of OSS through bug tracking systems. While studying 600 bug reports, the authors found out that in OSS, the role of users is not limited to simply bug reporting. They believe that
Based on the above discussion, it is evident that online forums have a crucial role in the development and maintenance of OSS. Online forums provide a major platform as far as users’ involvement is concerned. In particular, we intend to explore whether users’ requirements are being addressed by OSS developers. It is also investigated whether public online forums assist in identifying and addressing users’ requirements.
III. RESEARCH MODEL AND HYPOTHESES
In this study, we present a research model for analyzing the relationship between user requirements and the online forums of OSS projects. The theoretical model that will undergo empirical testing is presented in Fig. 1. Although it is generally believed that OSS developers refine their work through other contributors’ input, in this study, we are particularly focusing on whether Support Request Forums and Feature Request Forums assist in identifying and addressing end users’ demands. The term “Support Requests Forum” refers to a forum that deals with any activity, such as installation help, documentation, information retrieval, usability, etc., whereas the phrase “Feature Requests Forum” is used for a platform that deals with changes in requirements to add more requested functionality [18].
Our aim is to investigate the answer to the following research question (RQ):
The term “Open Requirements” is used as a user request or requirement that has been reported but has not yet been addressed, whereas “Closed Requirements” refers to a user requirement that has been reported and taken care of. In order to empirically investigate the research question, five hypotheses are presented in Table 1.
[Table 1.] User requirements hypotheses
User requirements hypotheses
Data was collected from 100 projects on sourceforge.net, a popular open source software project repository. The dataset covers the ten most downloaded projects from each of the categories of communication, software development, Internet, games & entertainment, text editors, scientific/engineering, database, education, multimedia, and formats & protocols. The data was collected from statistics of tracker traffic of each project for the time period of one year (September 2010–August 2011). The first filtration activity removes the data of all projects which have either total requirements of zero or no online forums. This reduces the dataset to 72 projects.
The maximum total open requirements/user requests (7,070) were found in the category of scientific/engineering but none of them were closed. The maximum requests (54.10%) addressed were observed in a text editor project. The maximum open support requests (2,804) were also found in the category of scientific/engineering, whereas a project in the Internet category had the highest percentage (63.64%) of the closed support requests. The category of “games/entertainment” had the maximum number of open feature requests (163) in one project. The maximum closed feature requests (51.85%) were observed in an Internet project. Fig. 2 presents the one-year tracker traffic of open and close requests in online forums of two sample projects.
We conducted tests for hypothesis H1, and its subhypotheses H1a, H1b, H1c, and H1d. Initially, we divided the data analysis activity into three phases. In Phase I, we conducted tests for the hypotheses using parametric statistics by computing the Pearson correlation coefficients, and in Phase II, we utilized non-parametric statistics through the Spearman correlation coefficients. Due to the relatively small sample size, both parametric as well as non-parametric statistical approaches were used to reduce the threat to external validity. The results are presented in Table 2.
[Table 2.] Empirical analysis results
Empirical analysis results
The Pearson correlation coefficient and t-test were examined between variables involved in the hypotheses H1a, H1b, H1c, and H1d. The Pearson correlation coefficient between open requirements and support requests in the public forums was positive (0.985) at p = 0.000, and thus provided a justification to accept the hypothesis H1a. The hypothesis H1b was also accepted based on the Pearson correlation coefficient (0.591) at p = 0.000, between open requirements and number of feature requests in the online forums. The correlation coefficient of 0.019 at p = 0.846 was observed between the closed requirements and number of support requests in the online forums, and hence H1c was rejected. The hypothesis H1d was accepted based on the Pearson correlation coefficient (0.646) at p = 0.000, between closed requirements and feature requests in the online forums. Hence, it was observed and that hypotheses H1a, H1b, and H1d were found to be statistically significant and were accepted. However, the hypothesis H1c was not supported by our parametric analysis and was hence rejected.
In Phase II, we conducted a non-parametric statistical technique using the Spearman correlation coefficient to test our hypotheses. Hypothesis H1a was statistically significant at p = 0.000 with a Spearman correlation coefficient of 0.421. A positive association was observed between open requirements and feature requests (H1b) in the online forums (Spearman 0.701 at p = 0.001). H1c, which deals with the closed requirements and support requests in the online forums, could not be accepted due to the observed Spearman coefficient of 0.272 at p = 0.006. The Spearman correlation of 0.688 at p = 0.000 was observed for H1d and was thus accepted. Hence, it was observed that hypotheses H1a, H1b, and H1d were statistically significant and accepted. However, like in Phase I, H1c was rejected due to the statistical analysis results.
Finally, in Phase III of hypotheses testing, we used the partial least square (PLS) technique to cross validate the results of Phase I and Phase II. We tested the hypotheses H1a, H1b, H1c, and H1d by examining their direction and significance. The hypothesis involves two variables, so in PLS, we placed one variable (support requests or feature requests) as the response variable and another (open requirements or closed requirements) as the predicate. Table 3 reports the results of the structural tests of the hypotheses. It contains observed values of the path coefficient, R2, and F-ratio.
[Table 3.] Structural test results
Structural test results
The results are displayed in Table 3. The PLS technique is especially useful in situations involving complexity, non-normal distribution, low theoretical information, and small sample size [19, 20]. All statistical calculations were performed using Minitab 16 software.
The path coefficient of open requirements (H1a) was found to be 0.27 with R2 (0.177), and the F-ratio (21.15) was significant at p = 0.000. Open requirements (H1b) also had a positive path coefficient of 0.59 with R2 (0.491) and an F-ratio of 94.47 at p = 0.000 with the number of feature requests. Closed requirements with support requests had a path coefficient of 0.19, R2 (0.074), and Fratio (7.83) at p = 0.006, which did not support hypothesis H1c. Closed requirements and the feature requests had a path coefficient of 0.64, R2 (0.473), F-ratio (88.13) at p = 0.000, and had the same direction as proposed in H1d. Overall, the hypotheses H1a, H1b, and H1d showed significance at p < 0.005 with positive path coefficients and were in the same direction as proposed, but H1c was not supported in this phase as well.
V. STUDY DISCUSSION AND CONCLUSION
The open source community is not limited to developers and programmers any more. Users from all over the world are using OSS, with their different cultural backgrounds and unique requirements and expectations. In this research, we studied the relationship between OSS users’ requirements for different supports and features and their response through public online forums.
The results of our study demonstrate positive correlation between the open user requirements, support requests, and feature requests in the online forums. Similar correlation has also been observed between the closed user requirements and the number of feature requests. Our analysis, however, did not support the correlation between the closed user requirements and the support requests. This necessitates addressing support requests per requirements and expectations of OSS users.
The overall results of our statistical investigation indicate a positive correlation between user requirements and the role of online forums. The outcome of the study supports hypothesis H1 and our assumption that online public forums help in addressing user requirements in OSS projects. This, in turn, provides an answer to the RQ as well.
As with any empirical studies, this research study has limitations of its own. Construct validity, internal validity, external validity and reliability are four criteria of validity in an empirical study [21]. Threats to the external validity limit the researcher’s ability to generalize the experimental results to industrial practice [22]. Furthermore, there have been ethical issues related to empirical studies in software engineering [23, 24].
In this study, we used a random sampling technique to address threats to external validity. The data has been collected from the most downloaded projects. Additionally, we retrieved data from the most active and well-known OSS reporting website, sourceforge.net, which includes a large number of projects. Moreover, we followed recommended ethical principles to avoid violation of any of the recommended experimental ethics.
Addressing users’ requirements is one of the major challenging options for open source projects. The objective of this study was to empirically analyze the association between user requirements in OSS and support through online public forums. The empirical results of this study support the hypotheses that public online forums assist in addressing user requirements in OSS projects. The study results, however, suggest that user requirements need to be more thoroughly addressed by using support request forums.