By Shaun Hendy
The MacDiarmid Institute’s Kate McGrath created a bit of a stir on Friday with a blog post that took a critical look at the Marsden Fund’s decision to invite 15% fewer proposals into the second round of its funding process than it did last year. The Marsden Fund runs a two-round process: applicants submit a one-page expression of interest in February, with about a fifth of these then invited by expert panels to submit a more comprehensive proposal in May.
In 2014 20% of first round applicants were invited to submit a second-round proposal. This year this figure fell to just 17%. Below, I’ve plotted the fraction selected to submit second round proposals since 1998. After a spike in 2003, the fraction selected at the second round has returned to the levels seen in fifteen years ago. So, while not unprecedented, a 17% throughput at the first round is something not seen for a while.The advantage of running a two-round system is that it can reduce the burden placed on the sector in the writing of comprehensive proposals. If we had a one-round system, everyone who wanted to apply would have to write a comprehensive proposal that was detailed enough for peer review. The more proposals that get through to the second round, the more work there is for the researchers that write them and the selection panels that evaluate them. The Marsden Fund Council chose to reduce the number of proposals selected for the second round this year because they wanted to reduce the workload for the sector.
McGrath is conscious of this issue, having blogged about compliance costs in late 2013: “The compliance costs of everyone applying for everything is costing our country real money that could be used producing real outcomes from the scientists utilising their time to ply their trade; scientific research rather than applying for funding that invariably they will never get.” In other words, the amount of effort we spend writing proposals should not be allowed overwhelm the effort we put in to our research.
Expectations
A slightly different way to look at this is to compute the expected value of writing a second-round proposal i.e. what is the expected pay-off in funding from submitting a second-round proposal? You can compute this by multiplying the likelihood that a second-round proposal will be funded by the average amount awarded. I’ve plotted this quantity below from 2001 to 2014, with a fantasy figure for 2015 based on the Marsden Fund’s indication that this year they will fund 90-95 proposals.By reducing the number of proposals at the second round, the Marsden Fund Council can increase the expected value of second-round proposals to the applicants. A standard second-round proposal (as opposed to a fast-start) this year is expected to return $315,000-$319,000 compared to $287,000 in 2014. As there is less money available overall this year, if the Marsden Council had okayed a similar proportion of second-round proposals to what it did in 2014, then expected value of a second-round proposal would have fallen to around $250,000. This would have been well below the long-run average of $290,000 for the expected value of a second-round proposal.
However, in her recent post, McGrath says that is now “more difficult to get your foot in the door of arguably the most prestigious grants in the country”. Her concern is that panels might not be expert enough to make good decisions at the first cut, so that the country might be missing out on funding the best science. The assumption that McGrath makes is that decisions made at the second round, when panels have access to expert peer review, will be of significantly higher quality than those at the first round.
Picking winners
Yet recent work by Te Pūnaha Matatini investigator Adam Jaffe suggests that Marsden panels are not very effective at predicting the eventual success of the second round proposals they view. Jaffe’s findings (not yet published, but which were commissioned by the Marsden Fund Council) are broadly consistent with the findings of many similar studies in other countries: panels and peer reviewers are not very good at picking winners (although see this recent article).
These findings do not to say that panels aren’t doing their best. Rather, the evidence suggests it is simply an almost impossible challenge to predict the outcomes of cutting-edge scientific research. It is also far from clear that letting more proposals through to the second round would improve matters. In my opinion, when we know that panels have such a difficult job to do, it is better to acknowledge their limited effectiveness and to reduce the burden on the system accordingly rather than to double down as McGrath suggests.
One-round system?
One of the key disadvantages of a two-round process is that, all things being equal, it will have a much lower success rate than a one-round process. More people will be prepared to chance their arm in round one if they only need to complete a short application. Filling out a single-round application for an Australian Research Council Discovery Grant, for instance, is considerably more onerous than the preparation of even a second-round Marsden proposal. The Australians are rewarded for the extra work involved with a success rate of 18%.
Despite this, I still prefer our system. If the ability of expert panels to predict funding success is poor, then better to have a system that minimises transaction costs. And as I noted in a blog post in March, I think we also need to acknowledge the increased research intensity in universities that has driven the oversubscription of the Marsden Fund.
I think the Marsden Fund Council is well aware of the trade-offs described above and has chosen a sensible course. In fact the Council should be commended for commissioning work to investigate the effectiveness of its decision-making process. The real fix for the system – increasing the Marsden Fund to match the increased level of research activity in our universities – is not within the Council’s control.
wouldn’t another counter-factual be to ask what level of guaranteed funding would scientist’s prefer over large, risky Marsden funds? what if scientists could choose to forego applications altogether but be able to count on some figure of funding that they can use at their discretion? if my expected value for a 10% chance overall is about $250k, wouldn’t i be indifferent at $25k per annum risk free but with the upside of being able to save/collaborate etc year in year out with an eye on long term results?
Shaun, I agree with you on this topic. Having been an applicant for the ARC and an applicant for Marsden as well as a panelist for several national research funds including Marsden, I think I’ve got a good basis for comparison between the two systems. The ARC system wastes massive amounts of academic time and hence money, in fact it has been estimated that the cost of investigator time to prepare ARC proposals is 66 million AUD per annum [1] – that is more funding that is given out by the Marsden fund annually. Now the ARC receives roughly 6000 proposals a year (it was slightly less in 2015) where Marsden now is getting 1200 so a factor of 5 less (eg the same as you would expect per capita for the size of the countries). However, it’s not that the ARC make you waste masses amounts of time in applying that gives a better success rate in the ARC, it’s simply that there is more money in the system in Australia – a lot more money. At present the ARC gives out about 350 million AUD a year and naively you could compare that to the total Marsden funding of just over 50 million AUD ie a factor of 7 BUT that would be missing the fact that ARC is not calculated on full cost funding as much of the ancillary costs for doing research in Australia is paid through other mechanisms such as RIB grants. It’s little bit hard to do a direct comparison with absolute conversions because it depends on how the money is to be spent. However since NZ overheads are highest on salary components and the ARC stipulate that salary based on-costs can only be 28%, not 120% and salary for chief investigators is not a budget-able item that makes a very large difference. As an example my 2012 Marsden which cost 870K in NZ and which had a fraction of my time and a 3 year postdoc would have been only 450K AUD in the ARC system which is roughly half of what you need in NZ. So really you are not looking a an increase of 7 times the funding in Australia compared to NZ but more like 14 times the funding in Australia. This is for an increase of only 5 in the number of applicants, so in real terms there is 2.8 times more funding per proposal available in Australia than in NZ. That Marsden has a 7% success rate and ARC is roughly 19% is entirely accountable by the difference in the level of funding in each system.
What is different though is that, yes, the ARC process wastes a vast amount of investigator time and NZ don’t. The ‘solution’ for Marsden is not as some would suggest that we move to an ARC-like proposal system, because we will still ultimately have the same success rates it’s just now we too will be wasting large amounts of money (~10 million a year) preparing proposals. In the NZ case because the funding pool is so small, such waste becomes insane when compared to the total amount of money on offer and so such an approach would, in my opinion, be insane [2]. (Plus this analysis disregards that the total funding pool will be eroded as the RSNZ will likely dip into it to cover higher assessment costs of dealing with full proposals).
So bottom line; NZ shouldn’t follow the ARC in assessment, but we should encourage the Government to follow Australia in the levels of funding.
[1] http://www.nature.com/nature/journal/v495/n7441/full/495314d.html
[2] Such thinking is a feature of the NZ system elsewhere and should be stamped out – you often see ridiculous participation costs for individuals and institutions for incredibly limited funding pools in NZ. I saw one recently where a 6K travel grant wanted a 10 page application followed by a 6 page report on completion!
If proposal evaluation cannot reliably pick “winning” science and has a cost that is (perceived as) too high, then use a lottery in which each applicant’s “ticket” is weighted in proportion to criteria for past research success.
Preparing a full grant application should not be considered a total waste of time. After all, one expects at least one highly qualified person in the field will read it.
So I tend towards Kate McGrath’s position. I would even recommend moving closer to the ARC system in which a full proposal was required at the preliminary stage, and filtered to maybe 50% to be refereed. This might reduce the incentive for applicants to apply without a big effort, which decreases the number of proposals to assess compared with the current preliminary round. Admittedly they are longer, but could be assessed similar to a preliminary proposal, but with more information. And people who rate their chances would still want to submit a full application, as they do in ARC, even if the expected reward in terms of $ is low.
I think is useful for me to place some thoughts here because I agree with Kate McGrath’s view. It’s not something new. I have just had that view since arriving in New Zealand 15 years ago. Having a Marsden grant has been a great experience I’m very thankful for, but it has only reinforced my view.
Why do I have such a view? I think it is because I come from a science system in the US where there were a diversity of funding sources and you can see what really works (and doesn’t work).
The most fundamental comment is that full proposals are the heart of a functioning science system. They’re not something to minimise for efficiency. Watching colleagues work through their Marsden full proposals at the moment underlines this point. There’s no comparison to the fashion show of pre-proposals. Seriously exploring an idea or collaboration really requires a full proposal – not a one-page “sell” and a bunch of CVs. Connections – whether they are between already published literature and new work, or between collaborators – are what allows ideas to develop into something that makes a difference. (Connections are, by the way, also the heart of Te Pūnaha Matatini.) A proposal system should incentivise the best possible mechanisms to encourage proposal writers to combine insights and connections in ways that have the greatest potential to lead to major discoveries. Well developed proposals should minimise strain on peer-review.
It appears that Marsden’s “triage” approach is based on the emerging work that the full proposal round is not very effective in picking winners. The argument that it is more efficient to evaluate less proposals is valid. But it seems equally valid to suggest that the “triage” approach is a temporary solution, and in the long-term we’ve got to come up with something better – rethink the whole system as Kate McGrath suggests.
Before we move on to what more full proposals would look like, and what they might achieve, is there any data to suggest what sort of funding rate we might have if all proposals were handled as a full proposal process? Yes, there is. In 2012, the US NSF’s Division of Environmental Biology (DEB) shifted to a pre-proposal round not unlike Marsden’s from what I can tell. Due to the US stimulus injecting extra funding into science from 2009, it became slightly difficult to estimate the effect. Now things seem to have settled out showing that having a pre-proposal round roughly doubled the number of proposals. Perhaps best of all, NSF DEB staff blogged on this. And the DEB data can be compared to directorates that did not implement pre-proposals – although it may pay to wait a few more years to get a full view.
So then, it looks like our funding rate and ARC are probably similar in real terms?
Well, maybe not. Throughout NSF, it has typically been the case that most funded proposal go through roughly between 2 and 3 submission/peer-review cycles before they are successful. So the funding rates are actually higher than you’d think – by a factor of 2 or 3. And in contrast to a sense that this might inefficient, most scientists firmly believe their proposal are (on average) greatly improved by this process, improving the quality of the science that is actually carried out once it is funded.
So, I’m saying it can be argued that the efficiency generated by the Marsden preliminary round simply means less ideas get properly explored.
One last problem is that Marsdens, wonderful opportunities that they are, represent a very small proportion of total science funding in New Zealand. We’re a small country, and Marsden grants cover the entirety of science and social science. There’s not much likelihood of 2 or more Marsden grants connecting to each other. This comment is already too long, so I’d simply say that connections are good. Fundamental research is important, but it is a fallacy to believe it cannot be useful connected to areas of applied research. See my commentary on this. Why not target areas of national importance and strength, where more connections might occur and be very useful. Given Shaun's case that Marsden has only kept pace with inflation, this may be the only way to get the government to commit to increasing funding for fundamental science. And it is easy to show targeting works, at the level of producing proposals that intersect where funders want them to.
To sum up, the two things a American scientists knows are that: 1) a good full proposal process works – they see it work individually, and not in aggregate efficiency stats; and 2) well-designed targets for proposals inspire good proposals that intersect usefully to build knowledge.
Both produce connections that work. (Measuring how well they work is good, hard goal.)
Last, I have a sense I should say this clearly. Marsden is the best thing we've got going. We should be very careful tinkering with it. But I do think a redesigned system could be much better.