Media
NSSI (Not Such a Silly Idea… but do it properly) #1
By Catherine Webb and Nicola Gujer
In October, the MBIE published an interesting document: the National Statement of Science Investment 2015-2025. It is intended to assess New Zealand’s strengths and weaknesses in research and development, and inform science funding for the next decade. As scientists, this document could impact our future, so it’s understandable that it has generated dialogue, especially about its use of data.
Among the NSSI’s many attractive graphics, this graph (Figure 1) in particular, which compares the impact of New Zealand research with that of a number of similar countries, has generated active discussion. A number of people have commented on social media about how poorly we do in multidisciplinary science compared to Denmark, Israel, Finland, Ireland and Singapore. We wanted to look a little deeper into what is going on.
Figure 1. Academic output quality of small advanced economies by field (NSSI, p. 19, 2015)
This data is taken from SciVal, Elsevier – a tool for analysing publications and citations – which is part of the Scopus database. The data is also field-weighted, meaning it is normalised to make different fields comparative.
The “percentage of publications in the top 10% of citations by field” is used here as a measure of the excellence of a country’s academic output. The greater the proportion of Denmark’s maths papers that make it into the most cited 10% of maths papers worldwide, the more excellent Denmark is at maths.
This seems like a reasonable measure, and it’s easy to see New Zealand has a concerningly left-heavy position on the scale. But does the graph really say, as the NSSI claims, that we need to up our game in almost every field to be as ‘excellent’ as our comparative countries?
We set out to pull apart this data, go back to the source, and check out how reliable this information is.
The first issue we encountered was the field labelled “Multidisciplinary”. As you can see, in all six countries, this field out-streaks the others where citation excellence is concerned. This made us wonder what “Multidisciplinary” actually means here – are these small countries all so good at collaborating across different fields?
As it turns out, “Multidisciplinary” publications seem to be labelled according to the nature of the journal they are published in, not because of any cross-field collaboration within the paper. For instance, this category includes papers that are published in the multidisciplinary journals Nature and Science, as well as more unusual choices such as Vaccines and the Journal of Chaos and Bifurcation Theory. Because a few multidisciplinary journals like Nature and Science are extremely highly cited, the citation impact distribution of “Multidisciplinary” publications is skewed to the right, so field-weighting (which is basically an arithmetic mean) does a poor job of normalising the top 10%. Thus, the field receives a confusingly high excellence score.
The second issue is more significant. We wanted to know how stable and consistent these field rankings were across the years, so we went back to SciVal to check. We discovered two things: firstly, due to the dynamic nature of citations, the data we retrieved this November on publications from 2013 was already different to the data on the very same 2013 publications that was gathered earlier this year (for use in the NSSI).
Figure 2. Change in 2013 data on output quality ranking of science fields in New Zealand between data retrieval in November 2015 (right) and earlier in 2015 (left). (SciVal November 2015, NSSI)
In a matter of months, the ranking of many fields has changed significantly. This caused us to question whether the trend stabilises over a matter of years. Discovery number 2: no, it doesn’t.
Figure 3. Change in academic output quality ranking of New Zealand science fields over five years. (SciVal November 2015)
This is a graph to show how the order of which fields New Zealand is most ‘excellent’ in has changed unpredictably, and has not stabilised over 5 years. We can infer that the graph on page 19 of the NSSI may be an accurate snapshot of the data at the time it was taken (except for that remarkable multidisciplinary field), but the data moves so quickly that this type of graph cannot reflect any meaningful trends in the order of fields.
That said, the SciVal data does have some information to offer.
Figure 4. Percentage of small advanced economies’ academic output falling within the 10% most cited academic output worldwide, by field, averaged over five years. (SciVal November 2015)
This is a remake of the NSSI graph, using averaged data over five years (2010-2014). The order of fields is different from the graph that MBIE published, as we would expect from what we have seen so far. However, this is similar to the NSSI graph in that New Zealand on average scores notably lower than other small advanced economies.
So does the graph say what it claims to say? With a bit of tweaking, it could. It cannot give us any meaningful information about which fields NZ is doing best or worst in, because that can change within months. However, if expanded to more years than 2013, it does show that NZ consistently produces a lower percentage of globally top-quality papers than comparable countries.
Whether papers being in the top 10% of citations in the world is a good way to measure science excellence or not – that’s a question for another day.
*Featured image by XKCD comics.
Adam Jaffe at ISPIM15
Adam Jaffe, our Theme Leader in Complex Economic and Social Systems is giving one of the keynote speeches at the ISPIM Innovation Summit in Brisbane.
The Summit runs 6th – 9th in December
The talk is titled: “Golden Goose or Sacred Cow: Evaluating the Impact of Public Research Support Programmes”
Public support of scientific research and technology development is common in advanced economies. There is a good theoretical case for such support, because the social rate of return to investments of this kind is high, and there are a number of reasons why private firms under-invest. But there is little evidence as to how much beneficial effect such programmes have. Simply pointing to scientific or commercial successes funded by the programme is not evidence of policy success, unless we can know or infer the extent to which the observed success would not have occurred with public support. We should be undertaking systematic evaluation of these programmes. This raises difficult issues of measurement and inference. This talk will discuss those issues and provide examples of attempts to overcome them.
Innovation NZ Symposium
Our Director Shaun Hendy will be attending the Innovation NZ Symposium 2015. It’s a symposium on Growing more Innovative and Productive Kiwi firms.
Catch Shaun on the panel December 1.
Royal Society of New Zealand Honours Dinner
by Kate Hannah
“Without those conversations, we couldn’t have done this research.” Amidst the sparkle and reflection of a Langham ballroom bedecked for a research honours dinner that celebrated International Year of Light, Professor Edwin Mitchell, reflecting on receiving the Beaven Award Medal from the Health Research Council. Recognised for a career dedicated to discovering the causes of sudden infant death syndrome, and to developing public health interventions to prevent infant death, he spoke quietly of the gift he and his colleagues are given by grieving families – the gift of their story, their medical history, their family’s grief and trauma. “Their contribution enabled us to save babies’ lives.”
Last night featured many a reference to the contribution of communities, the partnership that exists between science and the society it serves. Associate Professor Ruth Fitzgerald, winner of the Te Rangi Hiroa Medal for her work in the field of medical anthropology was cited for the immense importance she places on the relationship between researcher and subject, on the balance between investigation and privacy, the critical tipping point of needing to know versus the need to tell one’s own story.
We at the Te Pūnaha Matatini table were delighted to share with the celebrations of this year’s medallists, particularly the outstanding selection of investigator Dr Michelle Dickinson as the 2015 Callaghan Medal recipient. There may have been a standing ovation from table 19 as Michelle went up to receive her award! Michelle too, spoke of the connection between the work she does with children, families, and communities, and her research, thanking her head of department for appreciating the value of public engagement and science communication.
There was a sense of contentment too, at the prevalence of awards for people who are deeply concerned with the impact of their research within communities, the need for partnership, collective approaches, and teamwork. Professor Margaret Mutu, awarded the Pou Aronui Medal for her contributions to indigenous scholarship in New Zealand, thanked the University of Auckland for its support of her, even when she’s enacting her role as critic and conscience – a acknowledgement of the importance of our own, scientific or research community.
It was a time for our community to celebrate some changes too – why? As Justin Trudeau might say, “because it’s 2015.” Of the fifteen people celebrated last night – eleven Royal Society of New Zealand medallists, two Health Research Council medallists, two Gold Crest Award winners – five were women. Professor Margaret Hyland, who was awarded the Pickering Medal for her work to reduce fluoride emissions from the aluminium industry, was the first woman to ever win that particular medal. There was a sense of more women present too, in the people asked to present awards, and the citation videos and the celebration of twenty five years of the HRC. Also new – Professor Michael Walker welcomed us in te reo Māori, and his mihimihi was followed by Society President Professor Richard Bedford also speaking at length in Te Reo.
A night, then, of light, and a collection of important words: team, collaboration, community, sharing, support. Society Chief Executive Dr Andrew Cleland alluded to the need for the Royal Society to remain relevant, to reflect the values of the scientific community, and where necessary to take leadership in modelling those values. It felt, last night, like a beginning.
Pierre talks Soil
David Farrier talks with our Soil investigator Pierre Roudier on TV3’s Newsworthy.
You can watch it here
IP Statistics for Decision Makers
Our Director Shaun Hendy attended the 2015 IP Statistics for Decision Makers (IPSDM) Conference.
The conference took place in Vienna, Austria, to celebrate the beginning of the 10th year of PATSTAT, the EPO Worldwide Patent Statistical Database. PATSTAT was launched on 23 and 24 October 2006 in Vienna, at one of the first IPSDM conferences.
Shaun presented a paper on “The Regional Structure of Technological Innovation” co-authored with Dion O’Neale.
Asia-Pacific Innovation Conference
Adam Jaffe, Director at Motu and our Theme Leader in Complex Economic and Social Systems is presenting two papers at the 2015 Asia Pacific Innovation Conference in Hangzhou, P.R. China.
The School of Management, National Institute for Innovation Management and Institute for Intellectual Property Management at Zhejiang University is hosting the conference this month.
The papers Adam is delivering are:
- The effect of public funding on research output: the New Zealand Marsden Fund.
- The impact of R&D subsidy on innovation: a study of New Zealand firms.
We’re at SCANZ
In November, at the Science Communicators Association of New Zealand (SCANZ). Three of our people are presenting. They are:
Kate Hannah
(Te Pūnaha Matatini)
Presenting:
Using Emily Dickinson to upskill the new Michelle
Rhian Salmon – Plenary Speaker.
(Victoria University of Wellington)
Presenting:
Developing an engagement strategy for a National Science Challenge
Tulele Masoe along side Sarah Morgan
(COMET Auckland, Te Pūnaha Matatini Intern)
Presenting along side:
Setting up a participatory science platform pilot in South Auckland
Ten by Ten with Dr Siouxsie Wiles
Complexity and the Biosphere second Hui
Report By Hafiz Mohd
The second TPM Complexity and the Biosphere theme meeting, was held on Aug 27th 2015 at the University of Canterbury. This meeting brought together researchers in mathematics, science and other disciplines to discuss problems involving complex systems and networks. This year we had an exciting range of speakers ranging from talks on biological problems to discussion on economic welfare. Distinguished Professor and prominent theoretical ecologist from the University of California Davis, Alan Hastings, kicked off the meeting with exciting talk on modelling resource pulses. He discussed how theoretical results are used to provide important insight into the dynamics of resource pulses in the ecosystems. Another great talk was given by William Godsoe on predicting the distributions of species. As an ecologist at the Lincoln University, he specialises in employing mathematical models to understand different factors that can shape species’ distributions. We also enjoyed interesting talks from Stephen, Andrea, Simone, Mubashir, and Hafiz.
One of the talks during the meeting (Photo: Rachelle Binny)
Overall, meeting attendees enjoyed a productive day filled with great talks, delicious food and numerous opportunities to communicate and network in a collaborative environment. We would like to thank the organisers, attendees, presenters and sponsors for helping to make this second meeting a success. See you next time!