As a graduate student, one of my greatest challenges is determining what to read, how much to read, and how skeptical I should be of what I’ve read. There are so many journals, so many publications, so many people wanting to gain the academic currency of published work. If I do a quick and dirty google scholar search for “decision making” since 2014, over 100,000 results pop up. If I do a similar search for “Aging” since 2014, over 130,000 results pop up. If I search for “Aging and Decision Making” 40,000 results rise from the masses. I can specify it even more by adding “Aging, Decision making, Neuro*” and I still get close to 3,000 hits…. From the last year.
Now, even with the added specificity, this is still a fairly vague search. But as a neuroscientist, I need to relate information from the very specific forms of decision making I study, to decision making in general, to complex cognitive capacities, etc…. The list could go on. Because honestly, how can I understand the neural circuitry involved in decision making without also understanding attention and memory?
So when I go to read journal articles, how do I choose what to read and how do I know whether or not it carries merit? One strategy would be to only read articles from the highest impact journal articles. But this would ignore a large portion of the research out there that could be more relevant. I could primarily read from particular authors that I trust. But this would lead to a unilateral viewpoint of the research.
As for whether or not the articles carry merit. This is a tricky question. The peer review process, from my experience, seems suspicious. In my time as a graduate student, I have reviewed and rejected a paper that ultimately was published, I have submitted papers where the editor had to review the paper him/herself because no reviewers responded, I have had only one individual review my paper because no one else responded, I have twice had a manuscript rejected in less than 48 hours by the editor at one specific journal (for which my research was quite relevant). I have also read papers with furrowed brows, wondering “how….just…how did this get published?” because every so often something is so poorly written, basic, or unoriginal that even a youngster like me questions the reasoning behind the publication.  I think from this evidence, we can suspect at least two potential problems underlying current publication practices.
First, science and the peer-review process is not immune to politics. Friends publish friends work. Name matters. It seems that some editors use name and prestige as a heuristic for deciding which manuscripts get sent out for a review. This isn’t necessarily the worst thing in the world – those individuals with a well known name and prestige didn’t get there by accident. But it does prevent potentially important and relevant work from being published in some cases.
Second, scientists do not value reviewing. Often, a manuscript will get sent out to a P.I. and the P.I. will pass it on to a student, without giving the manuscript a glance. Therefore, it is up to a young, possibly naive, individual to review work. That is, if the P.I. makes the effort to send the review along to the student in the first place. This is a bigger problem than the first one. We depend on peer review to advance our science, but those individuals most qualified to review and provide quality feedback on manuscripts are too busy or unwilling to expend the time.
The sociopolitical environment and undervalued review process only amplify the challenge of sifting through publications to find the most relevant and thoughtful articles to inform my own work. But of course, this is only the tip of the iceberg. Other factors that need to be considered include the fear of publishing null results (yes, I said fear), the use of the p-value as a measure of “significance,” circular reasoning and conflated discussion sections, the hypercompetitive and oversaturated market, and a lack of research pre-registration as a common practice. And I’m sure there are more.
So what does this mean for our future? I personally think this is a problem that prevents us from achieving fundamental goals of science: to acquire knowledge and to discover underlying truths and organizing principles of the universe and those things that exist within it. If we don’t know where to seek valid and reliable knowledge to build upon, or aren’t able to effectively and efficiently determine which knowledge is sound, how can we productively progress?
So you want to be a scientist 