Today’s Open Science Reading: the Open Science Reviewer’s Oath

Well this certainly is interesting: The Open Science Peer Review Oath – F1000Research.  This emerged apparently from the AllBio: Open Science & Reproducibility Best Practice Workshop.  The “Oath” is summarized in the following text from a box in their paper:

Box 1. While reviewing this manuscript:

  1. I will sign my review in order to be able to have an open dialogue with you
  2. I will be honest at all times
  3. I will state my limits
  4. I will turn down reviews I am not qualified to provide
  5. I will not unduly delay the review process
  6. I will not scoop research that I had not planned to do before reading the manuscript
  7. I will be constructive in my criticism
  8. I will treat reviews as scientific discourses
  9. I will encourage discussion, and respond to your and/or editors’ questions
  10. I will try to assist in every way I ethically can to provide criticism and praise that is valid, relevant and cognisant of community norms
  11. I will encourage the application of any other open science best practices relevant to my field that would support transparency, reproducibility, re-use and integrity of your research
  12. If your results contradict earlier findings, I will allow them to stand, provided the methodology is sound and you have discussed them in context
  13. I will check that the data, software code and digital object identifiers are correct, and the models presented are archived, referenced, and accessible
  14. I will comment on how well you have achieved transparency, in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards, such that your experiments can be repeated independently
  15. I will encourage deposition with long-term unrestricted access to the data that underpin the published concept, towards transparency and re-use
  16. I will encourage central long-term unrestricted access to any software code and support documentation that underpin the published concept, both for reproducibility of results and software availability
  17. I will remind myself to adhere to this oath by providing a clear statement and link to it in each review I write, hence helping to perpetuate good practice to the authors whose work I review.

I note – I reformatted the presentation a tiny bit here.   The Roman numerals in the paper annoyed me.  Regardless of the formatting, this is a pretty long oath.  I think it is probably too long.  Some of this could be reduced.  I am reposting the Oath below with some comments:

  1. I will sign my review in order to be able to have an open dialogue with you.  I think this is OK to have in the oath. 
  2. I will be honest at all times. Seems unnecessary.
  3. I will state my limits. Not sure what this means or how it differs from #4.  I would suggest deleting or merging with #4.
  4. I will turn down reviews I am not qualified to provide.  This is good though not sure how it differs from #3. 
  5. I will not unduly delay the review process. Good. 
  6. I will not scoop research that I had not planned to do before reading the manuscript. Good. 
  7. I will be constructive in my criticism. Good. 
  8. I will treat reviews as scientific discourses.  Not sure what this means or how it is diffeent from #9. 
  9. I will encourage discussion, and respond to your and/or editors’ questions.  Good though not sure how it differs from #8. 
  10. I will try to assist in every way I ethically can to provide criticism and praise that is valid, relevant and cognisant of community norms. OK though this seems to cancel the need for #7. 
  11. I will encourage the application of any other open science best practices relevant to my field that would support transparency, reproducibility, re-use and integrity of your research.  Good.  Seems to cancel the need for #13, #14, #15, #16. 
  12. If your results contradict earlier findings, I will allow them to stand, provided the methodology is sound and you have discussed them in context. OK though I am not sure why this raises to the level of a part of the oath over other things that should be part of a review. 
  13. I will check that the data, software code and digital object identifiers are correct, and the models presented are archived, referenced, and accessible.  Seems to be covered in #11. 
  14. I will comment on how well you have achieved transparency, in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards, such that your experiments can be repeated independently. Seems to be covered in #11. 
  15. I will encourage deposition with long-term unrestricted access to the data that underpin the published concept, towards transparency and re-use. Seems to be covered in #11. 
  16. I will encourage central long-term unrestricted access to any software code and support documentation that underpin the published concept, both for reproducibility of results and software availability. Seems to be covered in #11. 
  17. I will remind myself to adhere to this oath by providing a clear statement and link to it in each review I write, hence helping to perpetuate good practice to the authors whose work I review.  Not sure this is needed.

The paper then goes on to provide what they call a manifesto.  I very much prefer the items in the manifesto over those in the oath:

  • Principle 1: I will sign my name to my review – I will write under my own name
  • Principle 2: I will review with integrity
  • Principle 3: I will treat the review as a discourse with you; in particular, I will provide constructive criticism
  • Principle 4: I will be an ambassador for good science practice
  • Principle 5: Support other reviewers

In fact I propose here that the authors considering reversing the Oath and the Manifesto.  What they call the Manifesto shoud be the Oath.  It is short.  And works as an Oath.  The longer, somewhat repetitive list of specific details would work better as the basis for a Manifesto.

Anyway – the paper is worth taking a look at.  I support the push for more consideration of Open Science in review though I am not sure if this Oath is done right at this point.

The future of Google Scholar

There is an interesting interview out in Nature where Richard van Noorden interviewed Anurag Acharya from Google Scholar: Google Scholar pioneer on search engine’s future : Nature News & Comment.  Definitley worth a look.  It has tidbits on the past, present and future of Google Scholar.

There are also some follow ups to this.  For example on Twitter I saw the following exchange:

I am in general agreement here that the cmmnity needs to start thinking about an open alternative.  Yes, I like Google Scholar (e.g., see my post on the Google Scholar blog: Using Google Scholar in Scholarly Workflows that I wrote in honor of the 10th Anniversary og GS.  But the lack of an API interface and the givng in to publishers demands seems lame.  So I do think we need to start to build up new strategies.  

Crosspost: Using Google Scholar in Scholarly Workflows

I wrote this for the Google Scholar blog where it was posted yesterday.  I am reposting it here, in part because comments are not allowed on the GS Blog (not sure why) and some people have asked me about that.  So here it is


When Anurag Acharya asked me recently if I would be interested in writing a guest post for the Google Scholar blog in relation to the 10th anniversary of Google Scholar I immediately responded “Yes.” Literally, that was the full content of my email response email response to his request. Why did I answer so enthusiastically? Well, I can put this down to three main reasons:

So – in thinking about what to write for this post, I came up with three main topics I thought would be good to cover – how I got interested in topics of searching for and sharing scholarly papers, how I use Google Scholar, and some ideas about future possible uses of Google Scholar.

Part 1: Some Background 

One day, in ancient history, my wife came home from work (at a biotech startup up focusing on bioinformatics) raving about this new search engine “Google” that people at her company were talking about. As someone who thought of himself as on the cutting edge of web technology, I was a bit dismayed that I had not somehow discovered this myself. But I got over that and tried it out. And, after searching for my name (and being impressed with how well this new search engine worked on such an important topic) I immediately started playing around with searching for scientific papers and data. I did this, I guess, because ever since I was in college, I had been becoming more and more interested in (or some would say obsessed with) issues relating to finding and sharing scientific knowledge.

Without going into two much detail some of the factors that contributed to my obsession included:

  • Working as a shelver and then assistant in the Museum of Comparative Zoology library in college and seeing how people struggled to find papers of relevance to their work;
  • Spending many years in graduate school (in the 1990s) working on projects that had been largely unstudied since the 1960s, including one (so called adaptive mutation) where researchers claimed to have discovered something new in the 1990s but had in fact missed a rich literature on the topic from the 1950s and 1960s (e.g., see this from 1955).
  • Building and sharing databases where I was trying to include a description of every paper that had been published about specific genes. I note – thanks to the Wayback machine my Stanford website from when I was a PhD student is still available – although alas the specific linked databases are not. I have reposted some of them for people to see what they were like (though many of the links in them are busted). See for example my sites on RecASNF2MutS and more
  • Working on projects to catalog everything known about specific organisms in association with work I was doing to characterize the genomes of these organisms

In these and other projects I had seen and experienced just how much time could be spent on searching for papers and data about a particular topics. I am not sure I had a well-defined strategy in every case but I came to rely upon some preferred methods including:

  • “Citation walking” where one takes a paper of interest and then asks “how has this paper been cited?” and traverses across the literature via citations
  • Searching for keywords in abstracts and titles
  • Browsing through specific journals
  • Looking for papers by specific authors
  • For data, I mostly would look in specific centralized data repositories such as Genbank for DNA sequence information and PDB for three-dimensional structural data on proteins.

And of course many other approaches. Nothing really novel or brilliant here though I do think I got pretty good at how to carry out such searches. But one of the challenges was each approach had to be done in a different system and some of the systems were only available for a fee and some were not even online. And even with lots of time and pain, many things could be missed.

Thus when my wife introduced me to this new fangled Google thing my thoughts rapidly turned to – how can I use this new tool to help in finding and then sharing scientific papers or data about these genes and organisms I was studying? Did Google searches solve all my “issues” in this regard? Alas, no. But jump forward ~15 years to today and I am quite amazed in retrospect how much of my scholarly workflow flows through Google Scholar. But rather than try to recall and write about how my workflow changed with the advent of Google Scholar I thought I would just jump to the present time and discuss some ways that I use Google Scholar now.

Part 2: Using Google Scholar today

When working on this post I started to look around at how I use Google Scholar and I confess I was amazed at how many different ways I use it in my work. Here are some examples:

Tracking and using citations. One major general use of Google Scholar lies in tracking of citations to specific scholarly works. Here are some ways that I use such information:

  • Citations to individual works. A key aspect of scholarly work in many fields is examining how specific works are cited. Such information has many uses include discovering new works on a topic by seeing how specific papers from the past are cited, assessing impact of works, ego satisfying, and more. For many years, information on how a specific work was cited was nearly impossible to come by without paying for access to citation tracking databases. Now, with Google Scholar I (and others) can very rapidly gather such information.
  • Citation from diverse sources. One aspect of using Google Scholar to track citations to individual works is the way GS finds citations in diverse sources – not just in the peer reviewed scholarly literature. Now, in some ways this can be viewed as a limitation (some may not want to count or even know about citations from self published white papers, for example). But in others ways this is a wonderful thing as one can find citations to one’s work from very diverse sources outside of the “normal” mold.
  • Citation metrics. It is not a large conceptual leap to go from the ability to track citations to individual works to the ability to create summary statistics about citations across many works. There are many indices for such purposes – some useful and some not. But whatever you think of such indices – Google Scholar has opened up the ability for people to calculate such metrics for oneself or to offer services to calculate metrics for others. Such indices can be used in many ways but perhaps the most common is to summarize the citations for one individual researcher. Which leads into my next topic …

Google Scholar Pages. Perhaps my favorite development from Google Scholar in the last 10 years has been the introduction of Google Scholar Pages for individuals. I make use of my Google Scholar page and pages of others for dozens of things including these:

  • Citation metrics for myself. See above for a discussion of citation metrics in general. I use Google Scholar pages to examine citation metrics for myself and my papers all the time (right now GS shows two summary statistics H-index and I-10 index). And I use this information in many ways including putting it on my CV, including it in grant reports, and examining which of my scholarly works have had more “impact”.
  • As landing page for my publication list. Once one has a GS page, GS automatically adds new publications to one’s list and also updates citation counts and other information regularly. Thus I now include a link to my GS page on my blogs, my work web sites, and in my email signature.
  • To keep track of my coauthors. I have been blessed (and perhaps a bit cursed) to work in a field (genomics) where many projects involve large-scale collaborations across many institutions, involving many researchers. And I have found that a nice way to track these coauthors is via GS (although – note to GS folks – there used to be a way to show, publically, all coauthors in a list but I cannot seem to figure out how to do this anymore).
  • Author disambiguation. For people like myself with a relatively unique name, when others search for my scholarly works, they are pretty easy to find (although I note the fact that there is another Jonathan Eisen out there who publishes some works with a bit of a conspiracy theory angle has been both good and bad for me at times). But for many others, their name is not a perfect way to find their work. This may be because they have a name that is relatively common, or it may be because they have changed their name (e.g., after marriage). For such people creating a GS page can be very useful because once one trains GS with a set of works, it can find new works by that same person quite well (I first found out about this author disambiguation by GS when Anurag gave a talk at a meeting I organized last year). GS is certainly not the only tool in author disambiguation and others – like author UIDs (e.g., ORCIDare almost certainly better long term options. I note – author disambiguation may seem like a esoteric topic to many but it has major implications on important issues such as gender equity in academia, since women are much more likely to change their names during their career than men are.
  • Automated updates of new papers by specific authors. One option associated with GS author pages I use extensively is the ability to “follow” specific authors and get notified of new publications of theirs.
  • To keep track of a collection of people. Most researchers do not regularly update their individual publication pages on their websites. However, if those researchers have GS pages one can keep track of their new papers quite easily (either by the follow option mentioned above or just by browsing occasionally). For example, for my microBEnet project I curate a list of GS pages for researchers in the whole field with connections to studies of “microbiology of the built environment” and thus (hopefully) help others keep up with what is going on in the field.
  • Who is in a specific field? One feature of GS author pages that is not used a lot as far as I can tell, but which has some value is the “areas of interest” tag one can add to one’s profile. Though not everyone uses such tags, I have found they are a useful tool in finding researchers working on specific topics. For example, I list “symbiosis” as one of my areas of interest and if I click on the link for that on my page I get a list (sorted by citation counts – which is both useful and annoying) of others who have listed that same area of interest. And many of the people in this list I am not familiar with yet they do work on topics in which I am very interested.

Automated discovery of new papers by topic. Pretty much all scholars these days are drowning in information and in keeping up with scholarly works. There are many reasons for this of course, and there are also some solutions. I find, for example, that social media is a great way to keep up to date on what new papers are coming out or have come out recently. But social media does not find everything and as someone who is responsible for keeping others up to date on various fields (e.g., this is one of my jobs at microBEnet) I also rely on both manual and automated searchers of the scholarly literature to find new papers or old papers I have missed. GS has two key ways to help in this regard. The first is relatively simple in concept but takes advantage of the power of Google indexing – which is just directly searching GS for papers on particular topics. And the advanced search options allow some customization of such searches. But as someone who is quite busy, I do not actually end up searching GS for new papers all that often. Instead I rely upon automated searches through various services including PubmedPubchase, and GS. I use GS in two ways for such automated searches:

  • Create an alert. When one does a search on GS, in addition to results one is presented with an option to “Create an alert” (which I think may only come up if one is logged in with a Google account). I now have dozens of such alerts in operation. To avoid getting drowned by the results I set them up to send only once a week and I filter them into a separate mail folder that I only look at when I have time. But I frequently find interesting new papers this way.
  • GS Updates. Another option now available, if one has a GS profile, is to use the GS Updates system (which I have written about before here and here for example). This system uses one’s publication list to scan for new papers that are related in some way to one’s prior work.

Many other uses of GS. I have gone on perhaps way too long here so I am only going to briefly mention a few other uses of GS.

  • Finding online versions of papers. Unquestionably one of the most valuable uses of GS is to find online versions of scholarly works. But since others have written extensively about this I will just say the following: if you publish any scholarly work I recommend you make it freely and openly available AND that you make sure that it gets indexed by GS.
  • Full text searches of the literature. Another critically important aspect of GS is that it facilitates full text searching of the scholarly literature which is important for many reasons.
  • Finding works outside of the “normal” places to publish. Another key feature of GS is that it indexes much more than just publisher’s sites. If one posts a preprint on one’s own web server, that paper may show up in GS (which I think is a good thing). GS also indexes many diverse sources of scholarly works and thus helps in finding works that may otherwise not see the light of day.

Part 3: Where do we go from here?

As an active user of Google Scholar I of course have many comments, complaints, ideas and thoughts about what it could do better and where it might go in the future. And there are SO many things that could be added or improved upon – things like better figure and table searching, better exporting of information, better abilities to curate and create collections and to then use such collections as training sets for automated searchers, and more and more and more.  I have written about some such issues and suggestions from time to time in my blog (see for example, this and this and this).  There is certainly lots of work to be done.

But in thinking about this I realized that making a list of issues and suggestions is only of limited value. What I think GS really needs is a better public forum where GS can discuss what their plans are for the future and also where users and developers can discuss what they would find useful. And though I see some places for such discussions on the Google Scholar blog and in related sites, I don’t see a lot. So – I would like to end with a call for GS to create a better site for such discussions of the future of GS …


Update – Adding some comments and responses from Twitter

A year ago, I came up with a great acronym for our project – ICIS – what do I do now?

About a year ago, when we starting meeting and discussing our new project on the future of scholarly communications, we decided it would be important to have a title and if possible a brief acronym. Now – I love coming up with such project names and acronyms and think I have done a good job with such tasks in the past.  Somehow I wanted to find a cool acronym for a project on scholarly communication, and publishing, and openness, and social media and more.  And I just could not come up with anything great.  

And this was starting to get semi urgent since we had a meeting coming up and needed to make a web site and needed names and titles and such.  So the group of us involved in the project (myself, Mackenzie Smith and Mario Biagioli) started sending around ideas to each other. Among the ideas first proposed:

  • IFHA Innovating Academic Publishing Project
  • The UC Davis IFHA project on Innovating Academic Publishing” with a project acronym of IAP

But I did not like this so I started doodling:

Screenshot 2014-10-23 14.36.48

Screenshot 2014-10-23 14.35.45

So I compiled some ideas and sent around a list

  • ISP: Innovating Scholarly Publishing
  • iSP: Innovating Scholarly Publishing
  • i2SP: Innovations in Scholarly Publishing  (with the 2 being a superscript)
  • i2AP: Innovations in Academic Publishing  (with the 2 being a superscript)
  • i2SP2: Innovations in Scholarly Publishing Project  (with the 2 being a superscript)
  • i2AP2: Innovations in Academic Publishing Project  (with the 2 being a superscript)
  • IN-A-PUB: Innovating Academic Publishing
  • SP2.0: Scholarly Publishing 2.0

Other ideas circulated (some mine, some others)

  • LIAP: Lab for Innovating Scholarly Communication, pronounced LEAP?
  • Innovating Academic Communication – IACOM
  • Innovating Academic Publishing – IAPUB
  • Innovating Scholarly Publishing – ISPUB
  • Innovating Scholarly Communication – ISCOM or INSCOM
  • COMMIS: communications innovation in scholarship

And then I came up with one I loved and sent around a suggestion:

“ok here is my favorite so far – innovating communication in scholarship – ICIS (pronounced isis —)”

Other ideas circulated:

  • pubs 2.0
  • innopubs
  • scholar 2.0

But I said I still liked ICIS best and, well, it won.  Yay.  We had a project name.   And we used it for our meetings and our website: icis.ucdavis.edu.  I was so proud of this name.  It sounded nice.  It conjured up images of ISIS the Goddess.  And every time we discussed the project I could remember the struggle to come up with a name and how happy I was when it came to me.

And then the ICIS name got, well, polluted.  Or, at least, the sound of the name got polluted by the group known variously as the Islamic State of Iraq and the Levant (aka ISIL), Islamic State of Iraq and Syria (aka ISIS).  And though I would hope it would be clear that our ICIS project is not connected in any way to ISIL/ISIS.  But yet, when I mention the project name to people, I invariable get some comment.  And Alison Fish, the first post doc working on the project just told me that she also gets some murmuring when she mentions the name during talks.

So we are in the midst of our annual retreat for the project now.  We have six people working on the project – myself, Mario Biagioli, Mackenzie Smith, Allison Fish, Alessandro Delfanti, and Alexandra Lippman.  And we all liek the ICIS name but are not sure what to do now?  So – in the interest of openness and communication I proposed to everyone (and they at least seemed to agree) that it might be good to ask the broader community for input here.

So – any suggestions?  What do you recommend we do?  Should we change the name?  Change how we say it?  Should we stick to our name and pronunciation?  And ideas, thoughts, or suggestions would be welcome.

Curating a Storify About this

TODAY: UC OA Policy & what it means for you (UC Davis Library 10/22)

WED. OCTOBER 22nd from 1:30-3PM

Panel organized by Library at UC Davis, Shields Library, Nelle Branch Room

The UC Open Access Policy (http://osc.universityofcalifornia.edu/open-access-policy/ or http://uc-oa.info) was passed by the UC Academic Senate on July 24, 2013, and is going into effect for all UC campuses, including UC Davis, on November 1, 2014. The policy grants UC faculty the right to make their articles freely available to the public by depositing a pre-publication copy in an open access repository. What does this policy mean for faculty at UC Davis?

Come to this talk by Catherine Mitchell of the California Digital Library (CDL), who will describe the tools and services that CDL is developing to support the policy, and Dr. Robert Powell of Chemical Engineering, who will give background on the policy and its passage through the UC Senate.  Afterwards a Q&A panel will be held with the speakers, UC Davis librarians and open access researchers to answer questions and discuss the implications of the policy and open access.

This talk is being held during Open Access Week 2014, an annual international event to raise awareness about open access issues.

  • Catherine Mitchell and Dr. Robert Powell on the UC OA policy: talk and discussion
  • Wednesday, October 22, 2014
  • Shields Library, Nelle Branch Room, 2nd floor (at the far end of the main reading room)
  • 1:30-3:00pm

Questions? Contact Phoebe Ayers, psayers@ucdavis.edu

Talk tomorrow on A2K for Visually Impaired: Ruth Okediji @ King Hall, UC Davis

Talk by Prof Ruth Okediji
Date and time: Tues, 10/21, Noon to 1pm
Location: King Hall, Room 1001
Title: The Road To Marrakesh: Toward a new global synthesis of intellectual property and the public interest?

Abstract: The Marrakesh treaty to facilitate access to Published Works by Visually impaired Persons and Persons with Print Disabilities (ViP treaty) is the first intellectual property agreement establishing minimum, mandatory exceptions to the rights granted to authors and owners of knowledge goods. the ViP treaty aims to eliminate copyright-related barriers to access to copyrighted works for over 285 million blind, visually impaired or otherwise print-disabled people around the world; accordingly, it differs from all other intellectual property agreements as a humanitarian endeavor. But the ViP treaty also suggests an explicit reshaping of the structural role of the state in relation to the public welfare objectives of national intellectual property policies. this lecture will highlight the political economy of the negotiating process, and the various justifications that shaped the final provisions of the ViP treaty. the lecture will also explore future prospects for achieving socially-responsible intellectual property outcomes in the global regulatory framework for knowledge goods.