What happens when 57 researchers tackle the same task? VŠE took part in a “crowdscience” experiment that businesses can also learn from
What results do you get when you give 57 researchers the same data and the same assignment? This question was examined by the authors of a study published in the prestigious Journal of International Business Studies (JIBS). The experiment, which also involved a pair of researchers from the Faculty of Business Administration at the Prague University of Economics and Business (VŠE), showed to what extent individual scholars’ conclusions can differ—and what this means for both business practice and academic research.
The authors organized a so-called “crowdscience” experiment: 57 scientists from two dozen countries, specializing in research on the business environment, received the same dataset containing information on more than 26,000 foreign subsidiaries of Japanese firms from the years 1991–2009. Each participant was also given four identical research questions, focusing on relationships between ownership form (100% ownership vs. joint ownership) and performance, the link between intangible assets and ownership share, and the impact of political uncertainty. However, the choice of measurement and modeling methods was left entirely to each researcher. It was precisely the consequences of this freedom that most interested the study’s authors.
Different methods, different results
The experiment showed that freedom in data processing leads to significantly different—sometimes even contradictory—conclusions. “Even when participants in the experiment start from the same theoretical principles, the way they apply them in practice can differ considerably,” confirms Ondřej Machek from VŠE, who took part in the experiment together with Aleš Kubíček.
“The outcome is inevitably influenced by what the analyst puts into the variables. Some measured firms’ intangible assets using brand-related indicators, such as advertising expenditures, while others understood them more as innovation and technological know-how, measured for example through R&D spending. In short, the final output of individual analysts was shaped by legitimate but subjective choices and judgments,” adds Aleš Kubíček.
Making sense of the mess
However, the authors were not merely interested in a vivid demonstration of the principles of open science or in highlighting the unreliability of business research and analysis. Although participants arrived at markedly different conclusions on many issues, in two of the four research questions it was possible—after averaging the results—to identify a consensus among researchers.
Although the effect was only slightly above the threshold of statistical convention, the overall body of results suggests that the firms studied generally perform better when operating fully owned subsidiaries than when they only co-own them. It was also confirmed that a greater level of intangible assets at the parent company is associated with a higher ownership share in foreign affiliates.
Don’t be misled by a single result
What can business and practice take away from this experiment? According to Machek, it clearly illustrates that in important business decisions it is unwise to rely on just one analysis or a single analyst. At the same time, he notes that one should not expect multiple sources to produce identical results. “Combining multiple perspectives and synthesizing several sources often leads to a more reliable estimate,” he adds.
Just as in this study, managers may encounter situations where analysts disagree on the same issue—and that, too, is a valuable outcome, according to Machek. “An investor or a company can then conclude that it is not rational to cling to a single interpretation as the ‘one truth,’ but rather to prepare for different scenarios that may arise,” Machek explains.