A guide to research metrics and their importance for young researchers.
As an early career researcher you are likely to pay little attention to citations, impact factors, h-index or i-index metrics. Furthermore, if you do, it may be confusing when deciding on what index to choose that best describe you. Do you use h, i-, g-, s-, e-index etc, and actually what do they really mean? Why bother? And how do you go about collecting this information?
While the area of research based metrics is a fiercely debated subject, as an ECR it is important to understand the fundamentals and plan ahead. Journal and publication metrics will ultimately affect your career at some stage, whether it is related to a job application, promotion or grant success and funding. This article gives you a brief introduction to key metrics and how they relate to you.
Your choice of journal and the use of impact factors
One of the most frequently used metrics related to journal 'impact' is the impact factor. At its core, the ISI impact factor is simply the average number of citations per article over a two-year period for any given journal. Thus, for the journal Nature, the 2012 impact factor (reported in 2013) was 38.6. This means that an 'average' article in the journal received 39 citations per year, in 2010 and 2011.
In general, impact factors are used to compare journals in a similar discipline and are not designed to compare individual cited works or researchers. Additionally, impact factor calculations are based on a set of rules outlined by ISI, and require a journal to be listed in its database and ultimately is an ’average’. This last point is important, since the distribution of citations across all articles in a journal are inherently not normally distributed and thus the average is usually skewed to a small number of highly cited articles. While I point out that journal impact factors are for comparing relative impact of journals and not researchers, like all things, when metrics are created they are inevitably used inappropriately. Impact factors are routinely used to compare researchers by employers, grant reviewers and peers. While this is clearly inappropriate, and as a research community we should push for change, as an ECR you need to carefully decide where to publish your valuable research.
So, publish everything in Nature right? Good luck! When considering where you publish your work the first step is to rank a series of journals in your field. This is relatively easy, and if you use ISI journal citation reports you can rank by discipline. The next step is to look at the relevance of your work with respect to each of the journals in the list. One important thing to consider is that you may want to look at other potential disciplines to publish your work. For example, if you work in the area of tissue engineering, does your work fit into a medical field, engineering field or physics discipline? You will find that an impact factor of 2 in one field may be classified as a moderate journal, yet a top journal in another. This aspect is quite important, since at some point you want to be in a position where you can highlight your impact within a discipline.
Another thing to consider is the journal itself and how it relates to your peers. Publishing your work in a high impact journal does not mean that you will receive a high number of citations yourself (remember, impact factor is only a mean of a non-normal distribution right?). Choose a high impact factor journal in the field but also ensure your peers publish in the same journal. This will enhance the likelihood of your work being cited in future and thus influence your personal publishing metrics.
While journals may ‘cheat’ the system to increase there relative impact factor and a number of alternatives are available (including page rank algorithms, half-life etc) as an ECR, the impact factor and 5-year impact factor are still the most important factors to consider when choosing where to submit your research article. This may change in future with the continuing evolution of the publishing paradigm, so spreading your work across journals that rank highly in the majority of metrics is best. Finally, as an ECR, always consider publishing a review or two. Review articles, if comprehensive and specific, will always achieve a high number of citations since they can be used to highlight the 'state of play' by researchers in your own field as well as others.
Your metrics
Impact factor allows comparison of journals within a given discipline so it's use in determining your research and your ’ranking’ is unrealistic. So, you need to identify a number of metrics that define you. At their core, your metrics will be a function of papers published and number of citations.
The simplest of calculations is your total number of citations and total number of publications. These are unrelated to journal impact factor and are simply reflective of your productivity output and peer citations. Many researchers quote citations that exclude self-citations (i.e. citing your work in a subsequent paper), since it reduces the potential for gaming the system. However, it may be good to include both total and non-self citations, since it reflects the relevance and continuity of your work.
The total number of papers is equally simple to calculate, however, for an ECR it is always good to report total number of works (along with citations) across a specific time frame. As an ECR you will be generally competing for positions and funding against peers at a similar point within their career. The majority of funding agencies, for example, take ’record-to-opportunity’ in to consideration so don't get despondent or even compare yourself to a professor with 200 research articles and 10,000 citations!
Ultimately, publishing takes time as it reflects the pace of your research (which will also vary dependent on discipline) and citations have a lag time with respect to publications and your peers research pace and publications. As an ECR, a good pace to start is by publishing additional review articles which will potentially be highly cited. You are likely to be challenging new avenues of research as an independent researcher and thus you need to conduct due-diligence by reviewing the literature. Don't put that to waste, publish it!
In addition to simply summing the number of publications or citations you can use additional calculations to report your impact. The most common of these is Hirsch's h-index. The h-index was originally developed for evaluating theoretical physicists relative standing with the discipline but is now widely used by most researchers. h-index is calculated by collecting all your papers with associated citation data and then ranking them by number (n) as a function of descending number of citations (c). You can then easily identify your h-index, where n=c. Thus a person with an h index of 5 has 5 papers with at least 5 citations. This removes bias for singularly high-cited papers or total number of publications. As with all metrics, the h-index should be reported with respect to peers of a similar level, as higher values will always be biased to established and senior researchers since citations grow with time. Many additional citation indexes exist which will also reflect the impact of your work, however the h-index is still the lead in the majority of disciplines. Of note, the g-index and i10-index are sometimes reported. The g-index, takes into account the lack of weighting for highly cited articles and is calculated in a similar way to the h-index however it considers the square of the citation number and thus bias towards higher cited articles. Another recent index is Google’s i10-index. The i10-index is simply the number of articles with 10 citations or more. Importantly all these indexes can be calculated within a given timeframe and for a ECR it is worth taking the time to calculate such metrics overall and over the past 2 or 5 years, thus giving the 'reader' a reflective snapshot of your career status. The use of a recent set time frame also allows you to not only show a more realistic productivity value but also levels the playing field with respect to more senior researchers. You never know, you may have a higher 5-year citation rate or i10-index than your more senior colleges.
Tools, calculations and final thoughts
There are many other metrics that describe your research productivity. The metrics discussed here are generally regarded as the most popular, however this article is meant to highlight the importance of thinking about metrics as an ECR and to generate discussion. All these metrics require journal articles being registered within a citation database and do not even begin to consider the impact of other non-cited work. Researchers are continually pushing the boundaries of science but also digital media and outreach. Many researchers have blogs, videos and write commentary and articles for popular magazines or newspapers. This clearly has impact but is not measurable using conventional metrics and is something we should all consider in future.
Finally, how do you go about calculating and representing your metrics? In general all of the metrics discussed here can be calculated manually providing you have access to a citation database. ISI Web of Knowledge and SCOPUS databases have tools that make this easy. More recently, Google Scholar has added citations and can also be used as has a easy to use and free interface that can be used by researchers where individual profiles can be created and published online. Obviously, these database collate and collect information in different ways so your metrics will never be identical across each tool, however providing you indicate your method of calculation, this is not so much of a problem and they are generally similar.
Ultimately, metrics are important for ECRs and if you haven't thought about this yet, you should do now. Finally, the age old 'publish or perish' paradigm is unlikely to change soon. So get to it!
This article was brought to you by ECR2STAR. If you would like to contribute to the ecr2star.org community by submitting an article, please contact us via info@ecr2star.org
