Sunday 26 August 2012

How to bury your academic writing



Inappropriate use of journal impact factors has been much in the spotlight. The impact factor is not only a poor indicator of research quality but it is also blamed for delaying publication of good science, and even encouraging dishonesty.  My own experience is in line with this: some of my most highly-cited work has appeared in relatively humble journals. In the age of the internet, there are three things that determine if a paper gets noticed: it needs to be tagged so that it will be found on a computer search, it needs to be accessible and not locked behind a paywall, and it needs to  be well-written and interesting.
While I'm not a slave to metrics, I am, like all academics these days, fascinated by the citation data provided by sources such as Google Scholar, and pleased when I see that something I have written has been cited by others. The other side of the coin is the depression that ensues when  I find that a paper into which I have distilled my deepest wisdom has been ignored by the world. Often, it's hard to say why one article is popular and another is not. The papers I'm proudest of tend to be those that required the greatest intellectual effort, but these are seldom the most cited. Typically, they are the more technical or mathematical articles; others find them as hard to read as I found them to write.  Google Scholar reveals, however, one factor that exerts a massive impact on whether a paper is cited or not: whether it appears in a journal or an edited book.
I've had my suspicions about this for some time, and it has made me very reluctant to write book chapters. This can be difficult. Quite often, a chapter for the proceedings is the price one is expected to pay for an expenses-paid invitation to a conference. And many of my friends and colleagues get overtaken by enthusiasm for editing a book and are keen for me to write something. But statistical analysis of citation data confirms my misgivings.
Google Scholar is surprisingly coy in terms of what it allows you to download. It will show you citations of your papers on the screen, but I have not found a way to download these data.  (I'm a recent convert to data-scraping in R, but you get a firm rap over the knuckles for improper behaviour if you attempt to use this approach to probe Google Scholar too closely). So in what follows I treated rank order of citations, rather than absolute citation level as my dependent variable. I downloaded a listing of my papers, ranked by citations, and coded them according to whether the article appeared in a journal or as a book chapter. Book chapters tend not to be empirical – they are more often review papers, or conceptual pieces - so to control for that I subdivided the journal articles into empirical and theoretical/review pieces. I also excluded papers published after 2007, to allow for the fact that recent papers haven't had a chance to get cited much, as well as any odd items such as book reviews. To make interpretation more intuitive, I inverted the rank order, so that a high score meant lots of citations, and the boxplots showing the results are in the Figure below.
Citation rank by Publication type. High rank indicates more citations. There is no significant difference between journal reviews and empirical papers, both of which have significantly higher citation rank than book chapters (p < .001)

Because I'm nerdy about these things, I did some stats, but you don't really need them. The trend is very clear in the boxplot: book chapters don't get cited. Well, you might say, maybe this is because they aren't so good; after all, book chapters aren't usually peer reviewed. It could be true, but I doubt it. My own appraisal is that these chapters contain some of my best writing, because they allowed me to think about broader theoretical issues and integrate ideas from different perspectives in a way that is not so easy in an empirical article. Perhaps, then, it's because these papers are theoretical  that they aren't cited. But no: look at the non-empirical pieces published in journals. Their citation level is just as high as papers reporting empirical data. Could publication year play a part? As mentioned above, I excluded papers from the past five years;  after doing this, there was no overall correlation between citation level and publication year.

Things may be different for other disciplines, especially in humanities, where publication in books is much more common. But if you publish in a field where most publications are in journals, then I suspect the trend I see in my own work will apply to you too. Quite simply, if you write a chapter for an edited book, you might as well write the paper and then bury it in a hole in the ground.

Accessibility is the problem. However good  your chapter is, if readers don't have access to the book, they won't find it. In the past, there was at least a faint hope that they may happen upon the book in a library, but these days, most of us don't bother with any articles that we can't download from the internet. 
I'm curious as to whether publishers have any plans to tackle this issue. Are they still producing edited collections? I still get asked to contribute to these from time to time, but perhaps not so often as in the past. An obvious solution would be to put edited books online, just like journals, but there would need to be a radical rethink of access costs if so. Nobody is going to want to pay $30 to download a single chapter. Maybe publishers could make book chapters freely available one or two years after publication  - I see no purpose in locking this material away from the public, and it seems unlikely this would damage book sales. If publishers don't want to be responsible for putting material online, they could simply return copyright to authors, who would be free to do so.

My own solution would be for editors of such collections to take matters into their own hands, bypass publishers altogether, and produce freely downloadable, web-based copy. But until that happens, my advice to any academic who is tempted to write a chapter for an edited collection is don't.

Reference
Eve Mardera, Helmut Kettenmann, & Sten Grillner (2010). Impacting our young Proceedings of the National Academy of Sciences, 107 DOI: 10.1073/pnas.1016516107

33 comments:

  1. Nice analysis! Do you put your book chapters online via your website at all? That might be one way to have the best of both worlds (book chapter, but still openly available). I'm not sure what the contracts are like with publishers for book chapters, but the journals certainly seem to disprove of this whilst also allowing it, so it may be the same.

    ReplyDelete
  2. thanks John. That's a very good question. I have recently started doing so for a handful of articles - partly in a spirit of curiosity to see if they attract any interest. I use a personal slideshare site rather than my University web page. I posted one of my chapters there recently and it received 132 views in one day. Obviously, views don't equate to citations, but I feel it at least has a chance of getting read. For most of these writings, I have no memory of the copyright arrangements, but I suspect I would have signed it away. It's hard to see that anything would be gained by publishers strictly enforcing copyright after several years, but I guess I may yet get asked to take it down.

    ReplyDelete
    Replies
    1. why not put it on arXiv? It's not just for physics papers and it's indexed by Google Scholar.

      Anyway, spot on blogpost. Even if my library has the book in question it is not instant access like the papers and I may not bother with it.

      Delete
  3. Re. book chapters. Accessibility is the problem, you're absolutely right. You're normally not even allowed to deposit postprints when it comes to book contributions. One problem with your solution is the way that publishing with a recognized academic house, or in a ditto book series (possibly with additional req.s, such as editorial committees of some note being attached), is used by universities/governments/funders to grade publications output of individuals for further use in the allocation of funds.

    Funny enough, for myself, my most cited piece according to GS is in fact a book chapter... :)

    ReplyDelete
  4. Thanks for this excellent post. I've never published a book chapter in an edited book, and after reading this I'm not likely to!

    I guess I always harboured the suspicion that it would be poorly read, mainly because of the accessibility problem. It's interesting to see that hunch borne out empirically as you've done. I'm sure I'm not alone in having had problems tracking down PDF versions book chapters written by others...

    I think there is also a sense in some cases that book chapters can be dumping grounds for data that wouldn't cut it for peer review - a bit like journal special issues where the bar for manuscript acceptance can be lower than normal. During my PhD I came across some dreadful book chapters in which the authors basically just regurgitated/reprinted their previous published work in a less coherent/rushed form. It put me off the whole idea, to be honest.

    I'm sure that's not true in your case of course :)
    And I think the main problem, as you say, is accessibility rather than quality.

    Thanks for the really interesting and useful analysis.

    ReplyDelete
  5. I still like the idea of edited books, i.e., bringing together leading authors to address some targeted topic. It is a great shame if people use this forum as a dumping ground for poor quality work. It is extremely disheartening if people so readily lower their standards for work they know will not be peer reviewed.

    I agree that collected works could be better served by distribution methods that circumvent the main publishing houses. It is difficult to see the benefit of the standard publishing route, especially as the peer review tends to be internal and organised by the guest editor anyway. The only obvious reason to attract a publisher is the stamp of approval they can provide, but that seems a pretty large cost if it effectively renders the work inaccessible! Also, I am not sure how much difference it makes to the reader which publishing house has endorsed the material. I am guessing that the academic reputation of the editors/authors is more important.

    ReplyDelete
  6. Hi Chris, Mark
    I really don't think the publisher is all that important. Some of my neglected stuff is with v. good quality editors/publishers.
    I agree about the value of themed work, but my preference these days would be to do this as a special issue of a journal, just so people can get hold of the papers on the internet.

    ReplyDelete
  7. Has anything changed since Google started putting edited volumes on line with Google books? They leave some pages out, but you can often get the gist... Did you break it down by year-of-citation?

    ReplyDelete
  8. Hi Joanna - I didn't find any correlation with publication year. But remember this is just an account of my own papers. To check out the effect properly, would need to look at a lot more authors.

    ReplyDelete
  9. escribid en cristiano, coñe
    cagondios

    ReplyDelete
  10. Some book chapters are invaluable, in a way that citations don't always measure- as tutorial pieces and for extended theoretical development. Someone who really wants to understand deeply the issues involved in your empirical articles probably can't get it from the terse style we typically use in journal articles. But this role may result in your readers citing your empirical articles more rather than the actual book chapter, so that the true impact is obscured.

    I only agree to write book chapters for which the publisher allows me to post my preprint on the university repository, so that solves the access problem (for my particular chapter).

    ReplyDelete
  11. Excuse my only having spotted the graph (v. interesting) and not read what everyone has written (a bit too busy right now), but a loose thought:- some book chapters are available on-line as single chapters. I'm guessing the cost of accessing these might be too high to be a game-changer, but it's something to think about - ?

    I like Alex's thought about publishing chapter preprints. Must ask after that myself. (I've "recently" written a book chapter on bioinformatics of methods investigating genome 3D structure.) It occurs to me that this might be particularly useful for fields that are moving fast.

    Alex: some journals give more space and allow more relaxed writing for reviews; there are special review journals that will accept book chapter-sized efforts.

    ReplyDelete
  12. My book chapters average about 1/10 of the citations my journal articles get.

    ReplyDelete
  13. My publications are all in literary criticism. I haven't done this kind of citation analysis, but all the books in which I have published chapters are indexed at the chapter level in the Modern Language Association database -- so they are pretty findable, though they still have to be requested and accessed in print. But I also self-archive on Academia.edu.

    ReplyDelete
  14. "own solution would be for editors of such collections to take matters into their own hands, bypass publishers altogether, and produce freely downloadable, web-based copy. But until that happens, my advice to any academic who is tempted to write a chapter for an edited collection is don't. " Well written.

    ReplyDelete
  15. Thanks for the very nice blog post. Indeed, a valuable use of Gscholar, which I too find fascinating. However, I was surprised by your usage of "suprisingly", when trying to download/export from Gscholar. Isn't it at the very core of Google products (and facebook, twitter etc...) that you can't export data (beyond a very limited amount though an ever-changing API).

    Thus, hard to do large-scale studies using R, in the same way that studies on social networks are de facto very hard to do for us computer science/data analysis researchers. These companies want to keep the knowledge that may be mined from their data.

    ReplyDelete
  16. In my experience, book chapters are great for satisfying NSF broader impacts. Pretty valuable to me, even if they don't get cited.

    Anyway, it's not usually a choice between writing a low-citation book chapter or a highly-cited journal article. I write the book chapter when I want to consolidate several lines of research I've been working on. I find it helpful for identifying new research directions.

    That is to say, citations are a very rough measure of value.

    ReplyDelete
  17. Thanks for all the comments. Quite a lot of interesting points have come out of the discussion, both here and on Twitter
    1. How much weight should be put on citations, esp. as chapters may be much read by students? In general, I am not in favour of rating work by metrics such as journal impact factor, but I do see citations of specific articles as an realistic indicator of a work's influence. And I have, for instance, an article in Psychological Bulletin (a journal that publishes reviews) that it is widely used in student teaching but it also has numerous citations. Quite a lot of journals do have options for publishing reviews or comment pieces, and I'd now always try one of those if I decided to write that kind of article.
    2. The value of thematic works, with contributions from several people. Agreed, but here I think a special journal issue is the answer
    3. Just how inaccessible are book chapters? It is true that you can usually get these either from inter-library loan or via a direct request to the author, and I reckon I make such a request perhaps 5 or 6 times a year. But in the current internet age we have all got very used to immediate access of material at our desks, and I am likely to disregard a chapter if I find that a lot of effort needs to be expended to get hold of a copy. Few chapters are really essential, so what I want to do is to have a quick look to decide just how relevant a work is to what I am writing. I am old enough to remember the days when there were no electronic journals, and I used to haunt the library regularly where I could browse. Now I can't remember when I last visited my University library.
    4. Self-archiving as an option. This is a very important point, I think. And it is a solution that is in our hands. Too many academics are still not using their university repository. I liked Alex's idea of agreeing to write a chapter only if deposit of the preprint in a repository was allowed by the publisher. You still have the problem that people may not know your chapter exists, but at least those who do know can get hold of it, which is an improvement on the usual situation.
    5. The final comment by Anonymous says it's not an either/or between book chapter or journal article. For me it is! The older you get, the more you realise you have a finite time on earth, so if you spend 3 months writing a book chapter, it's 3 months not writing something else. Agreed, I increasingly am interested in writing the kind of longer more thoughtful piece that consolidates information - but, as noted above, I'll try to find a journal that will publish it because I have seen too much good stuff buried in a book.

    ReplyDelete
    Replies
    1. "I used to haunt the library regularly where I could browse. Now I can't remember when I last visited my University library." You do recognize that this is a choice YOU make?

      Delete
  18. Adam Wagstaff from the World Bank contacted me to draw my attention to similar analyses he has done on a much larger scale,
    He wrote "Your results chime with mine from a couple of different literatures. The first WP below was published in the Review of International Organizations, and the second was published in the Journal of Health Economics. "
    see these summaries:
    http://go.worldbank.org/3VBZSAT010
    http://go.worldbank.org/R3C0OM4481

    ReplyDelete
  19. "my advice to any academic who is tempted to write a chapter for an edited collection is don't."

    That's good advice in general, but perhaps there should be an exception for books that deal with methods. I wrote three chapters for Single Channel Recording (Eds Neher & Sakmann) and that book became the bible for anyone in the business. One of them is my only publication with over 1000 citations. That may have been helped by pdf versions that have appeared on the web, no doubt illegally.

    ReplyDelete
  20. Great post - I would concur both with regard to the general quality of edited volume pieces, and their visibility. One shift in the US is for publishers to create "on demand" volumes, at least for classroom use, to allow chapters to have second and third lives. One can pick out chapters from all the volumes that Sage/Congressional Quarterly Press, e.g., have produced, and use the result as a class reader (or, presumably, for your own research.)

    ReplyDelete
  21. Hi
    As you mention, edited books are indeed more common in Humanities and an academic publisher does ensure the whole manuscript, ie including individual chapters is peer-reviewed,
    All the best
    Tricia

    ReplyDelete
  22. Interesting analysis. One thing to think about is whether the book publisher is assigning CrossRef DOIs to the chapters. Increasingly, this is the case, especially among publishers who already have strong journal programs. (More than 5.2 million DOIs have been assigned at CrossRef to book chapters, and this is our fastest growing content type.)

    The University presses are not quite so likely to do this, but they are getting the message. The major new book hosting platforms from JSTOR, Project MUSE, Oxford University Press and Cambridge University Press, are all committed to assigning DOIs to this important content.

    Many book publishers are starting to digitize their backfiles and assiging DOIs to this older book content as well.

    How does assigning DOIs help book chapter citations? By making chapters more visible (i.e. actively linked from references where they ARE cited.) By making them more discoverable. The bibliographic metadata that is deposited at CrossRef is distributed to a number of discovery services such as search engines, web and local library catalogs, abstracting and indexing databases. It can also be used for citation analysis. That means that when a researcher uses these tools to search, if a book chapter has a CrossRef DOI, that chapter will be found.

    One suggestion in for authors of book chapters to ask, of course, if the chapter will be available electronically, and if so, t whether a CrossRef DOI will be assigned to the content, and if not, explain the importance.

    ReplyDelete
  23. This comment has been removed by a blog administrator.

    ReplyDelete
  24. This comment has been removed by a blog administrator.

    ReplyDelete
  25. Its a great article on academic writing .... good work by the writer!!

    ReplyDelete
  26. Great analysis. You can download some data using Harzing's Publish or Perish, which sits on top of Google Scholar and is a free download.

    ReplyDelete
  27. We took your advice and bypassed the publishers a while back. You can see the results at http://ontogenesis.knowledgeblog.org. The rapidity and ease of publication compared to conventional book publication was notable. A very positive experience.

    ReplyDelete
  28. Very nice and impressive blog about Academic writing.
    Thanks for this great information.


    Academic Writing

    ReplyDelete
  29. What years did you include in your analysis? More specifically, if you look at citations in the last couple years, has it changed? I'd think with google books & green open access the difference should be fading.

    ReplyDelete
    Replies
    1. I haven't written any chapters in recent years, so I can't test your idea!
      I do agree that if book chapters become available electronically, the problem should disappear.

      Delete
  30. Very Interesting article. This is making me think about the purpose of writing a journal and writing a book. If we say that the purpose is to announce new scholarly research then I can imagine that the journal article is a better platform than a book chapter.

    I am wondering if we can list different objectives or purposes of writing and then we can compare whether a book chapter is more suited or a journal article. For example, from above comments we see that journal articles are written tersely. Maybe this is not an ideal platform for a more reflective writing, or writing with an objective to give a more complete overview of the subject.

    Wonder what you think Dorothy?

    ReplyDelete