NSSI (Not Such a Silly Idea… but do it properly) #3

By Catherine Webb and Nicola Gujer

The adventures in data-sleuthing continue. In this blog series, two summer students examine the National Statement of Science Investment 2015-2025, and appraise its use of data. So far we have found one case of partially-correct-yet-misleading-ness, and another of axis-labelling-deficit. This time we take a look at Academic-Corporate Collaboration.

On page 17, the NSSI makes an intriguing statement: “Only 3.2 per cent of New Zealand publications have academic and corporate affiliations, suggesting scope for more collaboration.” We asked, where did this information come from? By what standard do we have ‘scope for more collaboration’? And is that a good standard to use?

Implicit in that statement is the assumption that an abundance of academic-corporate collaboration is a good thing – a claim to be investigated later on.

Firstly, we found that the “3.2%” used in the NSSI is found under ‘Academic-Corporate Collaboration in New Zealand’ on SciVal. The first thing we noticed was that this statistic does change over time, although it fluctuates less than field-weight citations. From the time it was retrieved for the NSSI published in October and the time we checked it in November, it had already fallen to 2.6% (that’s around a 19% drop). Hence, we wanted to check out how stable this measure is over a longer period of time.

We didn’t find anything very remarkable in that regard: on average, the academic-corporate collaboration rate of each small advanced economy deviated about 17% from its initial value over five years, with New Zealand squarely on the mean (+16.7%).

This also helps us answer the second question, ‘compared to what do we lack collaboration?’ The graph shows how our nation’s academic-corporate collaboration measures up to that of other small advanced economies (SAEs); Denmark, Finland, Ireland, Israel and Singapore (widely accepted as comparative countries due to similarities in factors that affect science performance). Using the same measure, this is the data as it stands in Nov/Dec 2015:

Percentage of publications with both academic and corporate affiliations:

We see that by this standard, NZ is below average, but not markedly. We are still above Singapore and Finland, and with our ‘3.2%’ measured earlier in the year, we would have been above the present average!

Presumably, when the NSSI claims that New Zealand is lacking collaboration, they are using the small advanced economies as a reference – they cannot be referring to the worldwide percentage of academic-corporate collaboration, as that is only 1.34%. And yet, if they are comparing us to other SAEs, we are near the average and certainly not significantly underperforming.

Finally, however, we found a significant problem with the New Zealand statistics on SciVal. Academic-corporate collaboration is defined as at least one academic institution and one corporate (business) institution being affiliated with the same publication. On SciVal we found evidence that Crown Research Institutes (which are government entities and not private businesses) are being counted as corporate organisations. Here is one example of a paper listed as an academic-corporate collaboration:

As you can see, the only contributors to this paper are universities and two Crown Research Institutes; AgResearch and Landcare Research. Although our Crown Research Institutes have been ‘corporatised’, meaning that they are managed much like businesses, New Zealand is unique in this respect. Many countries have government science research organisations – the equivalent of our CRIs – but which are treated as purely public government institutes, such as the CSIRO in Australia. This presents an obstacle in drawing conclusions from this data set: comparing academic-corporate collaborations between countries is problematic when New Zealand calls corporate what other countries call government.

This inclusion of CRIs as corporations is skewing the total stats on collaboration in New Zealand, but by how much it is difficult to tell. Unfortunately, it is not possible to find the collaboration data adjusted to exclude CRIs on SciVal. CRIs cannot be excluded from the affiliation search without excluding papers with genuine collaboration between universities and corporations. SciVal’s lack of Boolean operators makes more nuanced searches impractical. Thus, we cannot provide a more correct number for New Zealand’s academic-corporate collaboration percentage than that published in the NSSI. But what we can say is that the NSSI’s number is not accurate, and when CRIs are excluded, NZ’s true academic-corporate collaboration percentage should be, in fact, lower than the NSSI reports.

We have to trust that a similar mistake has not been made in SciVal’s database for any other of the small advanced economies. Without a better dataset we cannot draw any conclusions about the potential for improving academic-corporate collaboration in New Zealand. If anything, this project has highlighted the need for comprehensive data-keeping, as well as taking care in how it is used.

*Featured image by XKCD comics.