Category Archives: Alt Metrics

Notes from “Advancing Research Communication & Scholarship”

4dfd167454e9f4745360a9550e189323

I’m currently at the Advancing Research Communication & Scholarship conference. Hackathon, presentations on open data,  open access, and alt metrics. You can follow what’s happening via Twitter at #arcs2015. I’ll add some notes from the conference soon and am excited to present on “Beyond Open: Global Perspective on Research Communication and Knowledge Production.” The panel’s chaired by Brian Rosenblum (University of Kansas Libraries) and I am joining Jane Anderson (NYU) Mónica I. Feliú-Mójer, (Ciencia Puerto Rico and iBiology) and John Willinsky (Stanford). Below is the description:

Current scholarly communication initiatives are focused on expanding access, use, and reuse. This session will explore the relationship between these issues and the needs and goals of the developing world and marginalized communities.  We will consider how new models and expectations affect and address knowledge distribution structures in the developing world, and the control local research communities have over their own legacies and outputs.

Continue reading Notes from “Advancing Research Communication & Scholarship”

Some additional details of my discussion w/ reporter John Bohannon for his Science story on Google Scholar

There is a story in today’s Science magazine on Google Scholar by John Bohannon.  Entitled “Google Scholar Wins Raves, But Can It Be Trusted” the article discusses some of the pros and cons of using Google Scholar.  The author of the article interviewed me on and off over a few weeks about Google Scholar because I have written multiple blog posts on how I use it.  For example see:

And also a diverse array of posts on Twitter which I will not rehash here.

Anyway – the new article covers some interesting points but is very very short (ahh — the fun with page length restrictions).  So I thought I would post here some of the comments I made about Google Scholar in emails with the reporter.

Bohannon wrote to me on December 1, 2013

Dear Dr. Eisen-

I’m writing a news story for Science about Google Scholar. Have you continued using their article recommendation engine? (I saw your blog post about it from last year.)

(and then he wrote some details about what he was working on which I am not sure he would want me to post here and I have not asked so I am leaving them out)

I’d very much like to hear your thoughts on how Google Scholar has developed, and how well it works as a replacement for traditional library/proprietary/non-open literature databases.

cheers and thanks in advance,
John Bohannon

I wrote back, that same day (unusual for me)

Well, was just hacking around with Google Scholar this AM.

Have you dug into their new function – Scholar Library? I am playing around with it but have not quite figured it out. What I am hoping to do is to figure out how to get recommendations based on lists in the Library. Currently, the recommendation engine has one very very big limitation. It bases recommendations on one’s own publications. And if you are trying to move into a new area – well that is pretty useless.

So – some comments

1. I find the recommendation engine to be very very useful still. One of the best ways to find out about new papers. With the limitation mentioned above.

2. I use automated Google Scholar searches to find all sorts of papers of interest. This helps cover topics more broadly than the recommendation engine.

3. There are other recommendation systems out there – but I have not used them too much.

4. As for “free and open” – I don’t think I would use such terminology here. Yes, Google Scholar is free. But is it open? I don’t think so. For example, I am not sure if they publish / release all their code that works behind the scenes. And I am not sure how open the results are (not saying it is not open – I just don’t know).

5. The citation information is a wonderful tool – and it is great to have this information be freely available. I use it routinely for all sorts of purposes including getting around the massive limitations of IF. One issue however with GS is that it takes citations from ALL sorts of sources including non peer reviewed material and even material that people may not have been aware was even publicly available. So – citation counts are generally higher – sometimes much higher – in GS than with other metrics. I note – I put info for my Citations based on GS on my CV and various other places.

And this allows GS to be seriously gamed in terms of citation counts.

6. GS is still clunky in a few ways and I hope that Google puts more effort into it. It could become THE tool for academic scholarship searches and tracking but it has some bugs and minor annoyances still.

Just some quick thoughts. Let me know if you have any other questions.

Jonathan Eisen

 

He then wrote back with some questions.  Again, I won’t share the whole email here at this time but I will share the specific questions he asked.  And I then wrote back with some answers

Question:

** Care to share a recent example? Something nicely illustrates its usefulness.

Answer:

 

Question regarding my comment on other systems

Me: There are other recommendation systems out there –  but I have not used them too much.

Bohannon: ** Are there?  For example?  I should take a gander.

Me: Mendeley has one – called recommended or something like that

Question regarding my comment on free vs. open
Bohannon: Right, good point. Free but not really open. Do you feel like this is a worrying limitation? Is it realistic to make it open?  Or at least, more open?
Me: Well, I am certainly not one to tell others how to run a business.  So I am thankful for any free or open material released / produced by for profits.  But there is a major worry here which is – if we do not know how the system works – we do not know if it is biased or how it can be gamed, etc.  And if it is not fully open then if we invest in making use of it, Google could simply kill it at any time and we would have no source material to use for other purposes.
Question regarding my comment on Impact Factor
Bohannon: What is an example of a massive limitation of IF that GS solves? And I wonder why GS can’t tell the difference between peer-reviewed and non-peer-reviewed sources. I’ll add that to questions below.
Me: GS helps with article level metrics as opposed to journal level metrics like IF.  I can look up citations to all my papers quickly.  And I can track metrics like H index and I10 and others.  IF – being a journal level metric – is not really that informative in my opinion (and that of many others)

Question on other things GS could do:

Bohannon: What do you see as the most important things that need to be fixed before it could really take over?

Me:

1. Transparancy in how it works and it what they are planning
2. Make it open source
3. Ability to create reference collations easily (e.g., like Mendeley or Zotero or CiteULike or Endnote).

Questions for Google

Bohannon: * Any other questions you’d like Google to answer, or features to request?

Me:

Well many features to request.

1. Better hot linking of all authors of a paper. Right now they only seem to link the 1st author to their google scholar page.

2. Better handling of long author lists.

3. Better ability to upload collections exported from other tools.

And many more …

After this Bohannon then wrote me another email with additional questions which I answered, briefly

Bohannon: Questions I should have asked but forgot:
Bohannon: When was the last time you exclusively used paper journals to find articles? (Ever?)
Answer from me:
never – or a very very very long time ago
since I started using email / the web 20+ years ago I have tried to use electronic / digital searches to find papers
Bohannon: What electronic services did you use before GS?
Answer from me:
Pubmed searches of course
And pubmed had some automated email alerts that I used to use
I also used lots of searches of journal web sites
And some journals offered automated searches too
The best thing out there was probably “related articles” in Pubmed
Bohannon: Have you switched entirely to GS?
Answer from me:
No – I still use pubmed searches quite a bit – partly because they are less cluttered than GS searches
I still occasionally search journal web sites but that is rare
And I actually use straight google searches a lot

And then another question

Bohannon: Oh, and when did you start using GS?  And when did it become your main search service?

(Sorry for the shotgun interrogation!)

Answer 1 from me:

heh – I have no idea …

Answer 2 from me:

I note – I think right now Twitter is the best source of information about new papers — far better than GS …

And I looked through my email and found that for a few years I used F1000 automated searches

And then in response to a follow up comment about Twitter from Bohannon I said

Yes, did not say Twitter was fun … but if you follow the right people, they can crowdsource for you all of the right literature better than GS

 

And then on Dec 6 Bohannon wrote me about visualizations

Hi Jonathan-

We’re sorting out some art for the article about Google Scholar.  I’m thinking that the best option would be a cropped screen shot of one of your Google Scholar automatic article recommendations.  Would you be OK with that?
Do they come as an email or appear in browser? If email, you can just fwd it.  If browser, could you take a screen shot and send it?
The idea is to show people what GS article recommendations look like, using your own experience for a visual.

 

So I sent a few things including these

 

Screen Shot 2013-12-06 at 12.09.09 PM

 

Screen Shot 2013-12-06 at 12.05.40 PM

We had a few other discussions about related topics but I thought it might be useful to see that general gist of the discussion and some of my thoughts on Google Scholar in more detail.  I note, at our meeting in February on the Future of Scholarly Publishing and Careers here at UC Davis Anurag Acharya from Google Scholar will be talking and will be on a discussion panel that will focus broadly on “metrics” for scholars so should be interesting.