Tag Archives: peer review

Video for Gaming Metrics Conference

GamingMetrics

Please follow the links to watch videos of the conference proceedings of “Gaming Metrics: Innovation and Surveillance in Academic Misconduct” held at UC Davis February 3-4th, 2016.

DAY 1

Welcoming remarks (Ralph Hexter, Provost, UC Davis)

FROM PUBLISH OR PERISH TO IMPACT OR PERISH (Mario Biagioli, STS & Law, UC Davis)

GAMING THE GAME ACROSS THE BOARD

  • Timothy Lenoir (UC Davis, Cinema and Digital Media & Science and Technology Studies) (Chair)
  • Alex Csiszar (Harvard University, History of Science) “(Gaming) Metrics Before the Game”
  • Paul Wouters (Leiden University, Science and Technology Studies) “The Mismeasurement of Quality and Impact”
  • Karen Levy (NYU, Media, Culture, and Communication) “Networks of Resistance in Trucking”

UNIVERSITY RANKINGS: GAMING OR COOKING?

  • Martin Kenney (UC Davis, Human Ecology) (Chair)
  • Barbara Kehm (University of Glasgow, School of Education Robert Owen Centre for Education Change) “Global University Rankings: Impacts and Applications”
  • Lior Pachter (UC Berkeley, Mathematics) “How King Abdulaziz University Became a ‘Better’ University than MIT in Mathematics”
  • Daniele Fanelli (Stanford University, METRICS) “Institutional Pressures to Publish: What effects do we see?”

PERSONAL V. INDUSTRIAL CHEATING

  • MacKenzie Smith (UC Davis, University Librarian) (Chair)
  • Finn Brunton (NYU, Media, Culture, and Communication) “Making People and Influencing Friends: Citation Networks and the Appearance of Significance”
  • Sarah de Rijcke (Leiden University, Science and Technology Studies) “System Identity: Predatory publishing as socio-technical disruption”
  • Jeffrey Beall (University of Colorado, Denver, Information Science) “Fake Impact Factors and the Abuse of Bibliometrics”
  • Dan Morgan (University of California Press, Collabra Project) “Cui Bono? Judging Intentions (and Outcomes) of Personal and Industrial Cheating”

META GAMING, META CHEATING

  • Anupam Chander (UC Davis, Law) (Chair)
  • Johan Bollen (Indiana University, School of Informatics and Computing) “From Bibliometric Metrics to Crowd-Sourced Science Funding Systems”
  • Carl T. Bergstrom (University of Washington, Biology) “It’s All a Game: The twin fallacies of epistemic purity and a scholarly invisible hand”
  • Jennifer Lin (Crossref) “Trust through Transparency: O brave new world/ That has such data in’t!”
  • Michael Power (London School of Economics, Accounting) “Research Impact and the Logic of Auditability: Solicited testimony as a case of meta-gaming”
  • James Griesemer (UC Davis, Philosophy) “Taking Goodhart’s Law Meta: Gaming, Meta-Gaming, and Hacking Academic Performance Metrics”

Day 2:

Watch HERE.

Welcoming Remarks (Kevin Johnson, Dean, UC Davis School of Law)

MISCONDUCT WATCHDOGS (I)

  • Jonathan Eisen (UC Davis, Genome Center) (Chair)
  • Ivan Oransky (Retraction Watch & NYU) “Retraction Watch: What We’ve Learned Since 2010”
  • John Bohannon (Science Magazine) “Grey Hat Hacking for Science”
  • Elizabeth Wager (Sideview) “Why Do We Need a Committee on Publication Ethics and What Should It Do?”

MISCONDUCT WATCHDOGS (II)

  • Jonathan Eisen (UC Davis, Genome Center) (Chair)
  • Darren Taichman (Executive Deputy Editor, Annals of Internal Medicine Vice President, American College of Physicians) “A False Sense of Security?”
  • Debora Weber-Wulff (University of Applied Sciences Berlin, HTW, Media and Computing & VroniPlag Wiki) “Documenting Plagiarism in Doctoral Theses: The Work of the VroniPlag Wiki Academic Community in Germany”
  • Brandon Stell (The PubPeer Foundation & CNRS) “Introducing PubPeer”
  • Emmanuel Didier and Catherine Guaspare (EPiDaPo, UCLA) “The Voinnet Affair: New Norms in High-Pressured Science”

COUNTERFEITING BRANDS V. FAKING PRODUCTS

  • Madhavi Sunder (UC Davis, Law) (Chair)
  • Marie-Andree Jacob (Keele University, Law) “Template, Creativity and Publication Ethics”
  • Alessandro Delfanti (University of Toronto, Institute of Communication, Culture, Information and Technology) “ArXiv or viXra? Physics and the quest for the true archive”
  • Sergio Sismondo (Queen’s University, Philosophy) “Leveraging Academic Value in the Pharmaceutical Industry”

CARNIVALESQUE RESPONSES

  • Alexandra Lippman (UC Davis, Innovating Communication in Scholarship) (Chair)
  • Cyril Labbé (Joseph Fourier University – Grenoble I) “Ike Antkare, His Publications and Those of His Disciples”
  • Burkhard Morgenstern (Universität Göttingen, Bioinformatics) “Virtual Editors Can Significantly Improve the Visibility of Junk Journals – A case study”
  • Paul Brookes (University of Rochester, Medicine) “Crossing the Line – Pseudonyms & Snark in Post-Pub Peer Review”

Today’s Open Science Reading: the Open Science Reviewer’s Oath

Well this certainly is interesting: The Open Science Peer Review Oath – F1000Research.  This emerged apparently from the AllBio: Open Science & Reproducibility Best Practice Workshop.  The “Oath” is summarized in the following text from a box in their paper:

Box 1. While reviewing this manuscript:

  1. I will sign my review in order to be able to have an open dialogue with you
  2. I will be honest at all times
  3. I will state my limits
  4. I will turn down reviews I am not qualified to provide
  5. I will not unduly delay the review process
  6. I will not scoop research that I had not planned to do before reading the manuscript
  7. I will be constructive in my criticism
  8. I will treat reviews as scientific discourses
  9. I will encourage discussion, and respond to your and/or editors’ questions
  10. I will try to assist in every way I ethically can to provide criticism and praise that is valid, relevant and cognisant of community norms
  11. I will encourage the application of any other open science best practices relevant to my field that would support transparency, reproducibility, re-use and integrity of your research
  12. If your results contradict earlier findings, I will allow them to stand, provided the methodology is sound and you have discussed them in context
  13. I will check that the data, software code and digital object identifiers are correct, and the models presented are archived, referenced, and accessible
  14. I will comment on how well you have achieved transparency, in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards, such that your experiments can be repeated independently
  15. I will encourage deposition with long-term unrestricted access to the data that underpin the published concept, towards transparency and re-use
  16. I will encourage central long-term unrestricted access to any software code and support documentation that underpin the published concept, both for reproducibility of results and software availability
  17. I will remind myself to adhere to this oath by providing a clear statement and link to it in each review I write, hence helping to perpetuate good practice to the authors whose work I review.

I note – I reformatted the presentation a tiny bit here.   The Roman numerals in the paper annoyed me.  Regardless of the formatting, this is a pretty long oath.  I think it is probably too long.  Some of this could be reduced.  I am reposting the Oath below with some comments:

  1. I will sign my review in order to be able to have an open dialogue with you.  I think this is OK to have in the oath. 
  2. I will be honest at all times. Seems unnecessary.
  3. I will state my limits. Not sure what this means or how it differs from #4.  I would suggest deleting or merging with #4.
  4. I will turn down reviews I am not qualified to provide.  This is good though not sure how it differs from #3. 
  5. I will not unduly delay the review process. Good. 
  6. I will not scoop research that I had not planned to do before reading the manuscript. Good. 
  7. I will be constructive in my criticism. Good. 
  8. I will treat reviews as scientific discourses.  Not sure what this means or how it is diffeent from #9. 
  9. I will encourage discussion, and respond to your and/or editors’ questions.  Good though not sure how it differs from #8. 
  10. I will try to assist in every way I ethically can to provide criticism and praise that is valid, relevant and cognisant of community norms. OK though this seems to cancel the need for #7. 
  11. I will encourage the application of any other open science best practices relevant to my field that would support transparency, reproducibility, re-use and integrity of your research.  Good.  Seems to cancel the need for #13, #14, #15, #16. 
  12. If your results contradict earlier findings, I will allow them to stand, provided the methodology is sound and you have discussed them in context. OK though I am not sure why this raises to the level of a part of the oath over other things that should be part of a review. 
  13. I will check that the data, software code and digital object identifiers are correct, and the models presented are archived, referenced, and accessible.  Seems to be covered in #11. 
  14. I will comment on how well you have achieved transparency, in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards, such that your experiments can be repeated independently. Seems to be covered in #11. 
  15. I will encourage deposition with long-term unrestricted access to the data that underpin the published concept, towards transparency and re-use. Seems to be covered in #11. 
  16. I will encourage central long-term unrestricted access to any software code and support documentation that underpin the published concept, both for reproducibility of results and software availability. Seems to be covered in #11. 
  17. I will remind myself to adhere to this oath by providing a clear statement and link to it in each review I write, hence helping to perpetuate good practice to the authors whose work I review.  Not sure this is needed.

The paper then goes on to provide what they call a manifesto.  I very much prefer the items in the manifesto over those in the oath:

  • Principle 1: I will sign my name to my review – I will write under my own name
  • Principle 2: I will review with integrity
  • Principle 3: I will treat the review as a discourse with you; in particular, I will provide constructive criticism
  • Principle 4: I will be an ambassador for good science practice
  • Principle 5: Support other reviewers

In fact I propose here that the authors considering reversing the Oath and the Manifesto.  What they call the Manifesto shoud be the Oath.  It is short.  And works as an Oath.  The longer, somewhat repetitive list of specific details would work better as the basis for a Manifesto.

Anyway – the paper is worth taking a look at.  I support the push for more consideration of Open Science in review though I am not sure if this Oath is done right at this point.