Gaming Metrics: Innovation & Surveillance in Academic Misconduct

GamingMetricsUC Davis, February 4-5, 2016

The event is open to the public. Please register here.

Follow on Twitter at #GamingMetrics.

This conference explores a recent evolution of scholarly misconduct connected to the increasing reliance of metrics in the evaluation of individual faculty, departments, and universities.  Misconduct has traditionally been tied to the pressures of “publish or perish” and, more recently, to the broadened opportunities enabled by electronic publications. The conference takes the next step and asks whether the modalities of misconduct have changed in time to adapt themselves not just to the general demands of “publish or perish” but to the specific features and techniques of the modern processes of academic evaluation variously connected to the notion of “metrics.” Have we moved from “publish or perish” to “impact or perish”? If so, are metrics of evaluation now creating new incentives for misconduct? Are metrics also helping the evolution of forms of misconduct in specific and innovative directions? And, crucially, can we reliably draw a clear separation between gaming the metrics game and engaging in misconduct?  Traditional discourses and policies of misconduct were rooted in oppositions between truth and falsehood, right and wrong, honest mistake and fabrication, but new metrics-based misconduct seem to be defined less by opposition than by degree — the amount of gaming involved.  In sum, are new metrics-based forms of misconduct asking us to rethink what misconduct means?

DAY 1 (Vanderhoef Studio Theatre, Mondavi Center)

9:00-9:15 Welcoming remarks (Ralph Hexter, Provost, UC Davis)

9:15-9:30 “FROM PUBLISH OR PERISH TO IMPACT OR PERISH” (Mario Biagioli, STS & Law, UC Davis)
A brief discussion of the conference themes and working hypothesis concerning the relation between academic metrics and misconduct. Current scenarios exemplify a vast increase of kinds of misconduct compared to traditional definitions (fabrication, falsification, plagiarism), but also point to a shift in the very goals of misconduct.  Initially driven by “publish or perish,” misconduct has become geared toward maximizing more complex metrics of academic credit encapsulated in a new imperative: “have impact or perish.”

9:30-10:30 “GAMING THE GAME ACROSS THE BOARD”
This session is meant to provide a baseline for the conference’s subsequent discussions by casting a wider net on metrics-gaming well beyond the specific field academic publishing, looking at how different communities and professions construe the line between acceptable and unacceptable gaming.  Mapping a wide range of gaming scenarios will then allow us to contextualize the specific forms of academic misconduct involving metrics gaming concerning academic credit.

  • Timothy Lenoir (UC Davis, Cinema and Digital Media & Science and Technology Studies) (Chair)
  • Alex Csiszar (Harvard University, History of Science) “(Gaming) Metrics Before the Game”
  • Paul Wouters (Leiden University, Science and Technology Studies) “The Mismeasurement of Quality and Impact”
  • Karen Levy (NYU, Media, Culture, and Communication) “Networks of Resistance in Trucking”

10:45-11:00 BREAK

11:00-12:00 “UNIVERSITY RANKINGS: GAMING OR COOKING?”
As university rankings are gaining increasing importance across the globe, they have been praised as agents of democratization against traditional academic “brands” living off reputational rent, but also criticized for the substantial ranking distortions that their easy gaming allows for.  When can these practices be treated as ranking gaming, and when do they cross over into institutional misconduct?

  • Martin Kenney (UC Davis, Human Ecology) (Chair)
  • Barbara Kehm (University of Glasgow, School of Education Robert Owen Centre for Education Change) “Global University Rankings: Impacts and Applications”
  • Lior Pachter (UC Berkeley, Mathematics) “How King Abdulaziz University Became a ‘Better’ University than MIT in Mathematics”
  • Daniele Fanelli (Stanford University, METRICS) “Institutional Pressures to Publish: What effects do we see?”

12:00-1:30 LUNCH

1:30-3:00  “PERSONAL V. INDUSTRIAL CHEATING”
One conspicuous difference from the days of “traditional” misconduct is the shift between misconduct as the work of individual scientists and scholars to scenarios in which misconduct is a more “collaborative” endeavor, as in the case of citation rings among journals to maximize their impact factors. (The production of fake alternative impact factors may be another example).  In addition to these novel conspiracies (which typically involve editors and publishers rather than traditional individual cheats like scientists and scholars), modern misconduct also involves businesses and organizations providing tools, platforms, and opportunities to academics interested in misconducting themselves.  These include so-called “predatory” journals, fake conferences, fake prizes, etc., that is, tools that enable and entice academics to meet the demands of their institutions’ evaluation metrics by gaming/cheating them.  Also, while these activities concern publications, they are not limited to the production of a fraudulent text (as “traditional” misconduct typically was), but aim at facilitating its publication.  They may be perhaps termed “postproduction” misconduct.

  • MacKenzie Smith (UC Davis, University Librarian) (Chair)
  • Finn Brunton (NYU, Media, Culture, and Communication) “Making People and Influencing Friends: Citation Networks and the Appearance of Significance”
  • Sarah de Rijcke (Leiden University, Science and Technology Studies) “System Identity: Predatory publishing as socio-technical disruption”
  • Jeffrey Beall (University of Colorado, Denver, Information Science) “Fake Impact Factors and the Abuse of Bibliometrics”
  • Dan Morgan (University of California Press, Collabra Project) “Cui Bono? Judging Intentions (and Outcomes) of Personal and Industrial Cheating”

3:00-3:15 BREAK

3:15-5:00 “META GAMING, META CHEATING”
This session has a double goal.  First, to analyze the kind of gaming that involve not the manipulation of a metric but the construction or adoption of a metrics – not gaming an established game, but the gaming that goes into defining the game itself. Is the competitive market of academic metrics (from faculty performance to university rankings) a form of gaming the game itself?  And where/when/how can it become misconduct?  Second, this session aims at engaging with Goodhart’s law, which is taken to show not only that the introduction of any kind of metric creat, es a market for gaming it, but that by so doing it invalidates the significance of that metrics.  If so, one could argue that any metrics will create the possibility of misconduct, but that the articulation of forms of misconduct specific to that metric will eventually “crowd” that market, thus creating an incentive to change the metrics, which in turn will usher in the next generation of innovative misconduct.  Or can we argue, against Goodhart, that it is possible to find a metrics of academic evaluation that can break the nexus with gaming/misconduct?

  • Anupam Chander (UC Davis, Law) (Chair)
  • Johan Bollen (Indiana University, School of Informatics and Computing) “From Bibliometric Metrics to Crowd-Sourced Science Funding Systems”
  • Carl T. Bergstrom (University of Washington, Biology) “It’s All a Game: The twin fallacies of epistemic purity and a scholarly invisible hand”
  • Jennifer Lin (Crossref) “Trust through Transparency: O brave new world/ That has such data in’t!”
  • Michael Power (London School of Economics, Accounting) “Research Impact and the Logic of Auditability: Solicited testimony as a case of meta-gaming”
  • James Griesemer (UC Davis, Philosophy) “Taking Goodhart’s Law Meta: Gaming, Meta-Gaming, and Hacking Academic Performance Metrics”

————————————————————————————————————
DAY 2 (Kalmanovitz Appellate Courtroom, King Hall)

9:15-9:30 Welcoming Remarks (Kevin Johnson, Dean, UC Davis School of Law)

9:30-10:30 “MISCONDUCT WATCHDOGS (I)”
The emergence and pervasiveness of new forms of misconduct exceed the
reach, resources, and conceptual framework of traditional governmental watchdog organizations typically connected to funding agencies like, in the US, the ORI.  This has spawned a new generation and new figures of misconduct surveillance, detection, and prosecution. Among these is a new breed of “watchdogs” — new actors who are often institutionally unaffiliated. These “watchdogs” have assumed an important role and a credible voice, often by creating new “ecologies of support” for themselves — websites, blogs, wikis, social media, etc.  Does their somewhat unique role indicate something about the specific nature of modern academic misconduct?  Does it suggest that the “battlefield” of misconduct is moving away from governmental agencies (acting according to traditional and possibly outdated definitions of misconduct) and toward journals and the watchdogs who monitor their publications?

  • Jonathan Eisen (UC Davis, Genome Center) (Chair)
  • Ivan Oransky (Retraction Watch & NYU) “Retraction Watch: What We’ve Learned Since 2010”
  • John Bohannon (Science Magazine) “Grey Hat Hacking for Science”
  • Elizabeth Wager (Sideview) “Why Do We Need a Committee on Publication Ethics and What Should It Do?”

10:30-10:45 BREAK

10:45-12:00 “MISCONDUCT WATCHDOGS (II)”

  • Jonathan Eisen (UC Davis, Genome Center) (Chair)
  • Darren Taichman (Executive Deputy Editor, Annals of Internal Medicine Vice President, American College of Physicians) “A False Sense of Security?”
  • Debora Weber-Wulff (University of Applied Sciences Berlin, HTW, Media and Computing & VroniPlag Wiki) “Documenting Plagiarism in Doctoral Theses: The Work of the VroniPlag Wiki Academic Community in Germany”
  • Brandon Stell (The PubPeer Foundation & CNRS) “Introducing PubPeer”
  • Emmanuel Didier and Catherine Guaspare (EPiDaPo, UCLA) “The Voinnet Affair: New Norms in High-Pressured Science”

12:00-1:30 LUNCH

1:30-2:30  “COUNTERFEITING BRANDS V. FAKING PRODUCTS”
This session looks at a specific form of fakery rooted in “brand appropriation.”  While the preceding session considers generally fake journals, conferences, etc., here we want to look more specifically at imaginary journals whose titles (as well as the look and feel of their websites) are made to resemble those of well-known and respectable journals.  One could perhaps add to this list certain “academic” conferences that take place in prestigious locations (say, Oxford) but are not actually affiliated with the university, or the appropriation of the names of respected academics that are then listed (without authorization) on editorial boards of fake journals or organizing committees of fake conferences.  Similarly, fake universities who sell degrees without any attempt at educating their students (not even online) tend to assume names with an Ivy League ring to them.  The common denominator here is an attempt at the mimicry of a “brand” rather than just the copying/pirating of a product.

  • Madhavi Sunder (UC Davis, Law) (Chair)
  • Marie-Andree Jacob (Keele University, Law) “Template, Creativity and Publication Ethics”
  • Alessandro Delfanti (University of Toronto, Institute of Communication, Culture, Information and Technology) “ArXiv or viXra? Physics and the quest for the true archive”
  • Sergio Sismondo (Queen’s University, Philosophy) “Leveraging Academic Value in the Pharmaceutical Industry”

2:30-3:30  CARNIVALESQUE RESPONSES
While misconduct “watchdogs” (discussed in a previous session) expose through public communication and denunciation, this session focuses on other actors who reveal misconduct and poor oversight through a carnivalesque approach.  Humor and absurdity—submitting profane papers and computer-generated gibberish articles that “sound” academic, or whistleblowers using clever anagrams as aliases–become a mode of critique and unmasking. Neither clearly “predatory” journals, “fake” conferences nor “legitimate” journals are immune to being the subject of a joke–a joke that, in some cases, may be more powerful than punishment. In a way, carnivalesque responses to misconduct continue the logic of an older history of art forgery-as-prank in which the forgery reveals through a kind of satire.  Are these cases telling us, perhaps, that satire is the best approach to both metrics and the gaming they elicit?

  • Alexandra Lippman (UC Davis, Innovating Communication in Scholarship Project) (Chair)
  • Cyril Labbé (Joseph Fourier University – Grenoble I) “Ike Antkare, His Publications and Those of His Disciples”
  • Burkhard Morgenstern (Universität Göttingen, Bioinformatics) “Virtual Editors Can Significantly Improve the Visibility of Junk Journals – A case study”
  • Paul Brookes (University of Rochester, Medicine) “Crossing the Line – Pseudonyms & Snark in Post-Pub Peer Review”

 

LOCATION: Please note that the conference will be held at two different locations on the UC Davis campus.  On Thursday, February 4 we will convene at the Vanderhoef Studio Theatre, Mondavi Center.  On Friday, February 5 our proceedings will take place in the Kalmanovitz Appellate Courtroom at King Hall (UC Davis Law School) (located here).

CAMPUS MAP: Can be found at http://campusmap.ucdavis.edu/

ADDITIONAL INFORMATION: Please email Alexandra Lippman (alippman at ucdavis dot edu)

INNOVATING COMMUNICATION IN SCHOLARSHIP — a UC Davis IFHA Project
CENTER FOR SCIENCE AND INNOVATION STUDIES

Open Sourcing Religion– Workshop, December 3, 2015

OpenSourceReligion-2015-10-30

Open Sourcing Religion

Thursday, December 3, 2015 / 9:00AM to 5:00PM

UC Davis King Hall, Room 2100A

A one day workshop on digital transformations in religious studies scholarship

RSVP if you plan to attend.

AGENDA

9:00am to 9:15am: Introduction: Andrew Ventimiglia (UC Davis)

9:15am to 11:15am: FUTURES

Moderator: TBD

Panelists:

  • Heidi Campbell (Texas A&M)
  • Pauline Hope Cheong (Arizona State University)
  • Eric Schmidt (UC Press)
  • Allison Fish (Indiana University)

11:15am-12:15pm: LUNCHTIME KEYNOTE: Flagg Miller (UC Davis)

12:15pm-1:15pm LUNCH BREAK

1:15pm-2:45pm HISTORIES

Moderator: David Biale (UC Davis)

Panelists:

  • Brian Hochman (Georgetown University)
  • Steven Jones (Loyola University, Chicago)
  • Andrew Ventimiglia (UC Davis)

2:45pm-3:00pm COFFEE BREAK

3:00pm-5:00pm PRACTICES

Moderator: Meaghan O’Keefe (UC Davis)

Panelists:

  • Daniel Schwartz (Texas A&M)
  • Mairaj Syed (UC Davis)
  • Nazmus Saquib (MIT)
  • Behnam Sadhegi (Stanford University)

ICIS Event: Software for Science — Getting Credit for Code

CreditForCodeFlyer-Oct-30-2015

Software for Science: Getting Credit for Code

Date & Time: October 30, 2015 from 9:30 am – 5:00 pm

This event was recorded and is now available for viewing

Software for scientific modeling, simulations, analytics, etc., have become commonplace in many research disciplines, requiring years of effort by scientists and researchers to create and maintain. Yet the scholarly credit system does not reliably recognize this effort and the importance of this type of contribution to scientific progress. Getting software integrated into the scholarly credit system of publication and citation is an important step towards correcting the problem. At UC Davis, the Software Attribution for Geoscience Applications (SAGA) project has brought together an interdisciplinary team from the Geological, Social, Library, and Computer Sciences to illuminate the technological and cultural barriers to effective software citation. The UC Davis Innovating Communication in Scholarship (ICIS) project is sponsoring a one-day conference to discuss these issues, what the SAGA project is discovering, what publishers and other players are doing, and what needs to happen next.

AGENDA

9:30-10:00 || Registration, Breakfast

10:00-10:30 || Welcome and Overview of Scientific Software Citation Needs and Culture
The use of software in science has become pervasive throughout the research cycle. If we agree that software is critical to research and scholarship, why have we been slow to change our practices and rewards its creation and ongoing development and support? This session will frame the issues and provide an overview of the day.
Speakers:

  • MacKenzie Smith, University Librarian, UC Davis
  • Louise Kellogg, Professor, Earth & Planetary Sciences, and Director, Computational Infrastructure for Geodynamics, UC Davis

10:30-12:00 || SAGA project findings: current infrastructure and survey results
The UC Davis SAGA project is exploring the problem of academic credit for software used in the geological sciences. This session will report out on findings for both infrastructure and practices available to projects that generate scientific software, and attitudes about what it needed and possible from a range of stakeholders working in geophysics.
Moderator: MacKenzie Smith, UC Davis

Speakers:

  • Laura Soito, Librarian, University of New Mexico University Libraries
  • Allison Fish, Professor, School of Informatics & Computing, Indiana University
  • Lorraine Hwang, Associate Director, Computational Infrastructure for Geodynamics, Earth & Planetary Sciences, UC Davis

12:00-1:00 || Lunch

1:00-2:30 || Existing Infrastructure & Needs – Credit, Discoverability, and Altmetrics
Several interesting projects have explored this topic over the past few years, involving key organizations like GitHub, CrossRef, PLoS, and Zenodo. This session will describe and discuss those efforts and what we’ve learned from them.
Moderator: Allison Fish, Indiana University

Speakers:

  • Abigail Cabunoc Mayes, Lead Developer, Mozilla Science Lab, Mozilla Foundation
  • Jennifer Lin, Director of Product Management, CrossRef
  • Arfon Smith, Chief Scientist, GitHub (via Skype)

2:30-3:00 pm || Coffee Break

3:00-4:30 || The Emerging Role of Publishers and Information Resource Companies
Clearly progress will be hindered unless publishers and information resource companies are involved in making software count, both as a recognized form of scholarship and as part of the scholarly credit ecosystem (e.g. included in citation databases and metrics systems). This session will focus on how publishers and other companies are thinking about software in relation to publications and other forms of scholarship beyond the research article.
Moderator: Joe Dumit, UC Davis

Speakers:

  • Patricia Brennan, Product Manager, Thomson-Reuters, Web of Science
  • Dan Morgan, Digital Science Publisher, University of California Press and Publisher, Collabra
  • Brooks Hanson, Director, Publication, American Geophysical Union (via Skype)

4:30-5:00 || Summary and Directions
If the first step is to admit we have a problem, what are the next steps we must follow? Where do we want to go from here? Are there specific changes or experiments to be done that would advance things right now? What specific measures can be undertaken? What research needs to be done? The closing session will discuss these questions and develop a plan of action.

Location: Student Community Center, Multipurpose Room, UC Davis

Welcoming New Graduate Student Researcher, Kevin Schenthal


Kevin SchenthalThe University Library’s Scholarly Communications Program is pleased to welcome Kevin Schenthal to the program.  Kevin is a sixth-year graduate student in the mathematics department.  Kevin joins the University Library’s open access team and will help conduct faculty outreach and support of the 2013 UC Academic Senate Open Access policy. These UC policies follow other changes in scholarly communications such as federal mandates to archive and share research publications, data, and software; rapidly changing business models for journal and book publishing; and new digital formats for online publications.  Kevin will also collaborate with ICIS, the Software Attribution for Geoscience Applications, and the forthcoming Hard Science Software: Credit for Code workshop.

Notes from Art of the Archive: Rethinking Archival Practices in a Digital Era

This re-post from the Committee on the Anthropology of Science, Technology, and Computing (CASTAC) blog describes a workshop on archival practices that took place on May 21, 2015, at the University of California, Davis. The essay is co-authored by Alessandro Delfanti, Allison Fish, and Alexandra Lippman. Delfanti, Fish, and Lippman are postdocs with UC Davis’ Innovating Communication in Scholarship (ICIS) project.

On May 21, 2015, the Innovating Communication in Scholarship project at the University of California, Davis held a one-day workshop on Art of the Archive. Papers given by the fifteen invited speakers explored the changing nature of the archive given the emergence of new information and communication technologies. These presentations largely focused on how these new digital archives are not merely technical creations, but are also constructed through social processes, have social impacts, and are not seamlessly implemented in everyday life. Instead, these digital storehouses are vibrant spaces for curating, organizing and publishing cultural heritage and expressive culture in new ways. In taking up this discussion three primary topics emerged and are described below: questions about access, circulation, and research design.

Image: Specimen Tray of Hans Sloane (1660-1753). Source: The Natural History Museum, London.

Continue reading Notes from Art of the Archive: Rethinking Archival Practices in a Digital Era

Imogen Clarke “Peer Review in the Ether”

Imogen Clarke  holds a PhD on the transition from classical to modern physics in early twentieth century Britain from the University of Manchester in 2012. She currently works in academic publishing in Oxford, UK. Clarke’s guest post is related to her recent article, The Gatekeepers of Modern Physics: Periodicals and Peer Review in 1920s Britain Isis 106 (1), (March 2015) 70-93.

Peer review – can’t live with it, can’t live without it. That’s the basic conclusion to be drawn from the myriad of discussions currently surrounding the topic, with a new peer review crisis apparently popping up every other day. Peer review’s ubiquitous place in scientific funding and publishing might lead the casual observer to assume it had been commonplace for centuries. In fact, while the Royal Society established a form of peer review as early as 1753, the practice did not become widespread until after World War II and, as of yet, nobody really knows why. But peer review’s surprising youth is a treat to the historian of twentieth century science. To try and picture a publishing landscape without omnipotent peer review, one does not have to go back too far.

Sir_Oliver_Lodge's_ether_machine
Sir Oliver Lodge’s ether machine

Continue reading Imogen Clarke “Peer Review in the Ether”