Leveraging Human Computation for Quality Assurance in Open Source Communities

Sprinz, Johannah (29. January 2022): Leveraging Human Computation for Quality Assurance in Open Source Communities. Bachelor, Faculty of Mathematics, Computer Science and Statistics, Ludwig-Maximilians-Universität München.

Abstract

Software developed under the open source development model (OSSD) has risen to significant importance over the recent decades. With more and more critical components being developed under the OSSD, the need for extensive quality assurance (QA) increases. This thesis investigates any potential for conducting formalized user testing through inexperienced volunteer community members under the OSSD. A human computation platform to aggregate such test results was designed and named open crowdsourced user-testing suite (OPEN-CUTS). A usability study of a prototype of OPEN-CUTS confirms the viability of this approach and points to potential future research questions.

Abstract

Software, die unter dem quelloffenen Entwicklungsmodell (Open Source Development Model, OSSD) entwickelt wird, hat in den letzten Jahrzehnten massiv an Bedeutung gewonnen. Immer mehr kritische Komponenten werden im Rahmen des OSSD entwickelt und der Bedarf für tiefgehende Qualitätssicherung (Quality Assurance, QA) steigt. Diese Arbeit untersucht, ob das OSSD Potenzial für formalisierte Nutzertests durch unerfahrene freiwillige Community-Mitglieder bietet. Eine Human Computation Plattform zur Sammlung solcher Testergebnisse wurde entworfen und trägt den Namen open crowdsourced user-testing suite (OPEN-CUTS). Eine mit einem Prototyp von OPEN-CUTS durchgeführte Benutzbarkeitsstudie bestätigt diesen Ansatz und zeigt mögliche zukünftige Forschungsfragen auf.

• Item Type: LMU Munich: Thesis
• Keywords: Crowdsourcing, Human Computation, Software Quality Assurance, Software Testing, Ubuntu Touch, UBports
• Faculties: Mathematics, Computer Science and Statistics > Computer Science > Selected Theses
• Institut or Department: Institute of Informatics, Teaching and Research Unit Programming and Modelling Languages
• Subjects: 000 Computer science, information and general works > 000 Computer science, knowledge, and systems; 000 Computer science, information and general works > 004 Data processing computer science; 300 Social sciences > 360 Social problems and social services; 600 Technology > 600 Technology
• Language: English

References

[1] A. M. Turing, “Computing Machinery and Intelligence,” Mind, no. 49, pp. 433–460, 1950, Accessed: Jan. 22, 2022. Available: http://www.jstor.org/stable/2251299

[2] S. Kumar, “Fundamental Limits to Moore’s Law,” arXiv, Nov. 2015, Accessed: Jan. 22, 2022. Available: http://arxiv.org/abs/1511.05956

[3] T. M. Mitchell, Machine Learning. New York: McGraw-Hill, 1997. Accessed: Jan. 22, 2022. Available: https://www.cs.cmu.edu/~tom/mlbook.html

[4] E. Law and L. von Ahn, Human computation. Morgan & Claypool, 2011. doi: 10.2200/S00371ED1V01Y201107AIM013.

[5] R. Mühlhoff, “Human-aided artificial intelligence: Or, how to run large computations in human brains? Toward a media sociology of machine learning:” New Media & Society, pp. 1868–1884, Nov. 2019, doi: 10.1177/1461444819885334.

[6] A. Doan, R. Ramakrishnan, and A. Y. Halevy, “Crowdsourcing systems on the World-Wide Web,” Communications of the ACM, vol. 54, no. 4, pp. 86–96, Apr. 2011, doi: 10.1145/1924421.1924442.

[7] R. Ardila et al., “Common Voice: A Massively-Multilingual Speech Corpus,” arXiv, Mar. 2020, Accessed: Jan. 22, 2022. Available: http://arxiv.org/abs/1912.06670

[8] K. Huotari and J. Hamari, “Defining gamification: A service marketing perspective,” in Proceeding of the 16th International Academic MindTrek Conference, 2012, p. 17. doi: 10.1145/2393132.2393137.

[9] L. von Ahn and L. Dabbish, “Labeling images with a computer game,” in Proceedings of the 2004 conference on Human factors in computing systems - CHI ’04, 2004, pp. 319–326. doi: 10.1145/985692.985733.

[10] R. Mühlhoff, “Big Data Is Watching You Digitale Entmündigung am Beispiel von Facebook und Google,” in Affekt Macht Netz, R. Mühlhoff, A. Breljak, and J. Slaby, Eds. Bielefeld: transcript Verlag, 2019, pp. 81–106. doi: 10.14361/9783839444399-004.

[11] G. G. Schulmeyer, Ed., Handbook of software quality assurance, 4th ed. Boston: Artech House, 2008.

[12] S. S. Bahamdain, “Open Source Software (OSS) Quality Assurance: A Survey Paper,” Procedia Computer Science, vol. 56, pp. 459–464, Jan. 2015, doi: 10.1016/j.procs.2015.07.236.

[13] M. Aberdour, “Achieving Quality in Open-Source Software,” IEEE Software, vol. 24, no. 1, pp. 58–64, Jan. 2007, doi: 10.1109/MS.2007.2.

[14] E. S. Raymond, “The cathedral and the bazaar,” First Monday, vol. 3, no. 2, Mar. 1998, doi: 10.5210/fm.v3i2.578.

[15] D. Wahyudin, A. Schatten, D. Winkler, and S. Biffl, “Aspects of Software Quality Assurance in Open Source Software Projects: Two Case Studies from Apache Project,” Aug. 2007, pp. 229–236. doi: 10.1109/EUROMICRO.2007.19.

[16] D. Kulesz and I. Bogicevic, “SystemTestPortal - A Web-Application for managing Manual System Tests,” Nov. 2017. Accessed: Jan. 22, 2022. Available: https://archive.fosdem.org/2018/schedule/event/systemtestportal/attachments/paper/1969/export/events/attachments/systemtestportal/paper/1969/iste_fosdem2018_v2.pdf

[17] N. Haines, “The Future of Ubuntu,” in Beginning Ubuntu for Windows and Mac Users, Berkeley, CA: Apress, 2015, pp. 205–209. doi: 10.1007/978-1-4842-0608-9_8.

[18] B. R. Prathap, “Operating Systems for Mobiles until late 2018 A Comparative Survey and Overview,” International Journal of Computer Sciences and Engineering, vol. 6, no. 12, pp. 576–581, Dec. 2018, doi: 10.26438/ijcse/v6i12.576581.

[19] J. Jenkins, “Ubuntu Touch beats Firefox OS to win best of MWC from CNET,” CNET, Feb. 2013, Accessed: Jan. 22, 2022. Available: https://www.cnet.com/news/ubuntu-touch-beats-firefox-os-to-win-best-of-mwc-from-cnet/

[20] M. Shuttleworth, “Growing Ubuntu for cloud and IoT, rather than phone and convergence.” Apr. 2017. Accessed: Jan. 22, 2022. Available: https://ubuntu.com/blog/growing-ubuntu-for-cloud-and-iot-rather-than-phone-and-convergence

[21] J. Sprinz, “One year after the world ended - Ubuntu Touch today.” Ubucon Europe 2018, Gijón, 2018, doi: 10.13140/rg.2.2.25859.78886.

[22] M. Gripsgård and J. Bots, “Stiftungsgeschäft (unter Lebenden) über die Errichtung der UBports Foundation.” Senatsverwaltung für Justiz, Verbraucherschutz und Antidiskriminierung des Landes Berlin, Abteilung II, Stiftungsaufsicht, May 2019.

[23] J. Sprinz, “State of the Touch: Ubuntu on phones and tablets.” Ubucon Europe 2019, Sintra, 2019, doi: 10.13140/rg.2.2.19148.90248.

[24] J. Sprinz, “Ubuntu Touch & Co - GNU/Linux in der Hosentasche.” 36th Chaos Communication Congress, Leipzig, 2019, doi: 10.13140/rg.2.2.15793.45921.

[25] UBports Foundation, “Release Schedule,” UBports Documentation. 2019. Accessed: Jan. 22, 2022. Available: http://docs.ubports.com/en/latest/about/process/release-schedule.html

[26] UBports Foundation, “Contributing: Quality Assurance,” UBports Documentation. 2019. Accessed: Jan. 22, 2022. Available: http://docs.ubports.com/en/latest/contribute/quality-assurance.html

[27] C. Győrödi, R. Győrödi, G. Pecherle, and A. Olah, “A comparative study: MongoDB vs. MySQL,” in 2015 13th International Conference on Engineering of Modern Electric Systems (EMES), Jun. 2015, pp. 1–6. doi: 10.1109/EMES.2015.7158433.

[28] Ecma International, “The JSON Data Interchange Syntax Standard,” Ecma International, Geneva, ECMA-404 2nd Edition, Dec. 2017. Accessed: Jan. 22, 2022. Available: https://www.ecma-international.org/wp-content/uploads/ECMA-404_2nd_edition_december_2017.pdf

[29] B. Terlson, B. Farias, and J. Harband, “ECMAScript® 2019 Language Specification,” Ecma International, Geneva, ECMA-262 10th edition, Jun. 2019. Accessed: Jan. 22, 2022. Available: https://262.ecma-international.org/10.0/

[30] W3C, “Web Services Architecture,” World Wide Web Consortium, Working Group Note 11, Feb. 2004. Accessed: Jan. 22, 2022. Available: https://www.w3.org/TR/2004/NOTE-ws-arch-20040211/wsa.pdf

[31] M. Bryant, “GraphQL for archival metadata: An overview of the EHRI GraphQL API,” in 2017 IEEE International Conference on Big Data (Big Data), Dec. 2017, pp. 2225–2230. doi: 10.1109/BigData.2017.8258173.

[32] Facebook, Inc., “GraphQL Specification: June 2018 Edition,” Facebook, Inc., Jun. 2018. Accessed: Jan. 22, 2022. Available: http://spec.graphql.org/June2018/

[33] D. Michie, “‘Memo’ Functions and Machine Learning,” Nature, vol. 218, no. 5136, pp. 19–22, Apr. 1968, doi: 10.1038/218019a0.

[34] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. New York, NY: Springer New York, 2009. doi: 10.1007/978-0-387-84858-7.

[35] MongoDB, Inc., “Aggregation Pipeline,” MongoDB Manual. 2021. Accessed: Jan. 22, 2022. Available: https://docs.mongodb.com/manual/core/aggregation-pipeline

[36] MDN contributors, “Introduction to client-side frameworks,” MDN Web Docs. Nov. 2021. Accessed: Jan. 22, 2022. Available: https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Client-side_JavaScript_frameworks/Introduction

[37] C. M. Lonvick and T. Ylonen, “The Secure Shell (SSH) Connection Protocol,” Internet Engineering Task Force, Request for Comments RFC 4254, Jan. 2006. doi: 10.17487/RFC4254.

[38] E. Rescorla, “The Transport Layer Security (TLS) Protocol Version 1.3,” Internet Engineering Task Force, Request for Comments RFC 8446, Aug. 2018. doi: 10.17487/RFC8446.

[39] J. H. Birns, K. A. Joffre, J. F. Leclerc, and C. A. Paulsen, “Getting the Whole Picture: Collecting Usability Data Using Two Methods - Concurrent Think Aloud and Retrospective Probing,” Proceedings of UPA Conference, pp. 8–12, 2002, Accessed: Jan. 22, 2022. Available: https://www.researchgate.net/publication/228865154_Getting_the_Whole_Picture_Collecting_Usability_Data_Using_Two_Methods--Concurrent_Think_Aloud_and_Retrospective_Probing

[40] S. Jamieson, “Likert Scales: How to (ab) Use Them,” Medical education, vol. 38, pp. 1217–1218, Jan. 2005, doi: 10.1111/j.1365-2929.2004.02012.x.