Publishing: The peer-review scam

RE: http://www.nature.com/news/publishing-the-peer-review-scam-1.16400

When a handful of authors were caught reviewing their own papers, it exposed weaknesses in modern publishing systems. Editors are trying to plug the holes.

26 November 2014

Illustration by Dale Edwin Murray

Most journal editors know how much effort it takes to persuade busy researchers to review a paper. That is why the editor of The Journal of Enzyme Inhibition and Medicinal Chemistry was puzzled by the reviews for manuscripts by one author — Hyung-In Moon, a medicinal-plant researcher then at Dongguk University in Gyeongju, South Korea.

The reviews themselves were not remarkable: mostly favourable, with some suggestions about how to improve the papers. What was unusual was how quickly they were completed — often within 24 hours. The turnaround was a little too fast, and Claudiu Supuran, the journal’s editor-in-chief, started to become suspicious.

In 2012, he confronted Moon, who readily admitted that the reviews had come in so quickly because he had written many of them himself. The deception had not been hard to set up. Supuran’s journal and several others published by Informa Healthcare in London invite authors to suggest potential reviewers for their papers. So Moon provided names, sometimes of real scientists and sometimes pseudonyms, often with bogus e-mail addresses that would go directly to him or his colleagues. His confession led to the retraction of 28 papers by several Informa journals, and the resignation of an editor.

Moon’s was not an isolated case. In the past 2 years, journals have been forced to retract more than 110 papers in at least 6 instances of peer-review rigging. What all these cases had in common was that researchers exploited vulnerabilities in the publishers’ computerized systems to dupe editors into accepting manuscripts, often by doing their own reviews. The cases involved publishing behemoths Elsevier, Springer, Taylor & Francis, SAGE and Wiley, as well as Informa, and they exploited security flaws that — in at least one of the systems — could make researchers vulnerable to even more serious identity theft. “For a piece of software that’s used by hundreds of thousands of academics worldwide, it really is appalling,” says Mark Dingemanse, a linguist at the Max Planck Institute for Psycholinguistics in Nijmegen, the Netherlands, who has used some of these programs to publish and review papers.

But even the most secure software could be compromised. That is why some observers argue for changes to the way that editors assign papers to reviewers, particularly to end the use of reviewers suggested by a manuscript’s authors. Even Moon, who accepts the sole blame for nominating himself and his friends to review his papers, argues that editors should police the system against people like him. “Of course authors will ask for their friends,” he said in August 2012, “but editors are supposed to check they are not from the same institution or co-authors on previous papers.”

Peer-review ring

Moon’s case is by no means the most spectacular instance of peer-review rigging in recent years. That honour goes to a case that came to light in May 2013, when Ali Nayfeh, then editor-in-chief of the Journal of Vibration and Control, received some troubling news. An author who had submitted a paper to the journal told Nayfeh that he had received e-mails about it from two people claiming to be reviewers. Reviewers do not normally have direct contact with authors, and — strangely — the e-mails came from generic-looking Gmail accounts rather than from the professional institutional accounts that many academics use (see ‘Red flags in review‘).

Red flags in review

Signs that an author might be trying to game the system

A handful of researchers have exploited loopholes in peer-review systems to ensure that they review their own papers. Here are a few signs that should raise suspicions.

  • The author asks to exclude some reviewers, then provides a list of almost every scientist in the field.
  • The author recommends reviewers who are strangely difficult to find online.
  • The author provides Gmail, Yahoo or other free e-mail addresses to contact suggested reviewers, rather than e-mail addresses from an academic institution.
  • Within hours of being requested, the reviews come back. They are glowing.
  • Even reviewer number three likes the paper.

Nayfeh alerted SAGE, the company in Thousand Oaks, California, that publishes the journal. The editors there e-mailed both the Gmail addresses provided by the tipster, and the institutional addresses of the authors whose names had been used, asking for proof of identity and a list of their publications. One scientist responded — to say that not only had he not sent the e-mail, but he did not even work in the field.

This sparked a 14-month investigation that came to involve about 20 people from SAGE’s editorial, legal and production departments. It showed that the Gmail addresses were each linked to accounts with Thomson Reuters’ ScholarOne, a publication-management system used by SAGE and several other publishers, including Informa. Editors were able to track every paper that the person or people behind these accounts had allegedly written or reviewed, says SAGE spokesperson Camille Gamboa. They also checked the wording of reviews, the details of author-nominated reviewers, reference lists and the turnaround time for reviews (in some cases, only a few minutes). This helped the investigators to ferret out further suspicious-looking accounts; they eventually found 130.

As they worked through the list, SAGE investigators realized that authors were both reviewing and citing each other at an anomalous rate. Eventually, 60 articles were found to have evidence of peer-review tampering, involvement in the citation ring or both. “Due to the serious nature of the findings, we wanted to ensure we had researched all avenues as carefully as possible before contacting any of the authors and reviewers,” says Gamboa.

When the dust had settled, it turned out that there was one author in the centre of the ring: Peter Chen, an engineer then at the National Pingtung University of Education (NPUE) in Taiwan, who was a co-author on practically all of the papers in question. After “a series of unsatisfactory responses” from Chen, says Gamboa, SAGE contacted the NPUE, which joined the investigation into Chen’s work. Chen resigned from his post in February 2014.

In May, Nayfeh resigned over the scandal at his journal, and SAGE contacted the authors of all 60 affected articles to let them know that the papers would be retracted. Chen could not be reached for comment for this story, but Taiwan’s state-run news agency said in July that he had issued a statement taking sole responsibility for the peer-review and citation ring, and admitting to the “indiscreet practice” of adding Taiwan’s education minister as a co-author on five of the papers without his knowledge. That minister, Chiang Wei-ling, denies any involvement, but nevertheless resigned “to uphold his own reputation and avoid unnecessary disturbance of the work of the education ministry”, according to a public statement.

The collateral damage did not stop there. A couple of authors have asked SAGE to reconsider and reinstate their papers, Gamboa says, but the publisher’s decision is final — even if the authors in question knew nothing of Chen or the peer-review ring.

Password loophole

Moon and Chen both exploited a feature of ScholarOne’s automated processes. When a reviewer is invited to read a paper, he or she is sent an e-mail with login information. If that communication goes to a fake e-mail account, the recipient can sign into the system under whatever name was initially submitted, with no additional identity verification. Jasper Simons, vice-president of product and market strategy for Thomson Reuters in Charlottesville, Virginia, says that ScholarOne is a respected peer-review system and that it is the responsibility of journals and their editorial teams to invite properly qualified reviewers for their papers.

Nature Publishing Group (NPG) owns a few journals that use ScholarOne, but Nature itself andNature-branded journals use different software, developed by eJournalPress of Rockville, Maryland. Véronique Kiermer, Nature‘s executive editor and director of author and reviewer services for NPG in New York City, says that NPG does not seem to have been the victim of any such peer-review-rigging schemes.

Free Podcast

An interview with Ivan Oransky from Retraction Watch.

00:00

But ScholarOne is not the only publishing system with vulnerabilities. Editorial Manager, built by Aries Systems in North Andover, Massachusetts, is used by many societies and publishers, including Springer and PLOS. The American Association for the Advancement of Science in Washington DC uses a system developed in-house for its journals Science, Science Translational Medicine and Science Signaling, but its open-access offering, Science Advances, uses Editorial Manager. Elsevier, based in Amsterdam, uses a branded version of the same product, called the Elsevier Editorial System.

Editorial Manager’s main issue is the way it manages passwords. When users forget their password, the system sends it to them by e-mail, in plain text. For PLOS ONE, it actually sends out a password, without prompting, whenever it asks a user to sign in, for example to review a new manuscript. Most modern web services, such as Google, hide passwords under layers of encryption to prevent them from being intercepted. That is why they require users to reset a password if they forget it, often coupled with checking identity in other ways.

Security loopholes can do more than compromise peer review. Because people often use the same or similar passwords for many of their online activities — including banking and shopping — e-mailing out the password presents an opportunity for hackers to do more than damage the research record. Dingemanse, who has published in a number of journals that use Editorial Manager, including PLOS ONE, says: “It’s quite amazing that they haven’t got around to implementing a safe system.” Neither Aries nor PLOS ONE responded to several requests for comment.

Safety measures

Lax password protection has resulted in breaches. In 2012, the Elsevier journal Optics & Laser Technology retracted 11 papers after an unknown party gained access to an editor’s account and assigned papers to fake reviewer accounts. The authors of the retracted papers were not implicated in the hack, and were offered the chance to resubmit.

Elsevier has since taken steps to prevent reviewer fraud, including implementing a pilot programme to consolidate accounts across 100 of its journals. The rationale is that reducing the number of accounts in its system might help to reveal those that are fraudulent, says Tom Reller, a spokesperson for Elsevier. If it is successful, consolidation will roll out to all journals in early 2015. Furthermore, passwords are no longer included in most e-mails from the editorial system. And to verify reviewers’ identities, the system now integrates the Open Researcher and Contributor ID (ORCID) at various points. ORCID identifiers, unique numbers assigned to individual researchers, are designed to track researchers through all of their publications, even if they move institutions.

ScholarOne also allows ORCID integration, but it is up to each journal to decide how to use it. Gamboa says that not enough scientists have adopted the system to make it possible to require an ORCID for each reviewer. And there is another problem: “Unfortunately, like any online verification system, ORCID is also open to the risk of unethical manipulation,” says Gamboa — for example, through hacking.

That is a common refrain. “As you make the system more technical and more automated, there are more ways to game it,” says Bruce Schneier, a computer-security expert at Harvard Law School’s Berkman Center for Internet and Society in Cambridge, Massachusetts. “There are almost never technical solutions to social problems.”

“As you make the system more technical and more automated, there are more ways to game it.”

It ultimately falls to editors and publishers to be on the alert, particularly when contacting potential reviewers. Carefully checking e-mail addresses is one way to ferret out fakes: a non-institutional e-mail address such as a free account from Gmail is a red flag, say sources. But at the same time, it could also be a perfectly legitimate address.

Jigisha Patel, associate editorial director of BioMed Central in London, says that it is definitely possible to catch cheaters by being on the alert for dubious e-mail addresses. “We’ve had some cases where we’ve caught them tweaking the e-mail addresses to try to steal someone’s identity,” she says. But such screening is imperfect. In September, the publisher retracted a paper in BMC Systems Biology, stating that it believed that “the peer-review process was compromised and inappropriately influenced by the authors”.

Some scientists and publishers say that journals should not allow authors to recommend reviewers in the first place. John Loadsman, an editor of Anaesthesia and Intensive Care, which is published by the Australian Society of Anaesthetists in Sydney, calls the practice “bizarre” and “completely nuts”, and says that his journal does not permit it.

It is unclear exactly what proportion of journals allows the practice, but as fields become more specialized it provides an easy way for busy editors to find relevant expertise. Jennifer Nyborg, a biochemist at Colorado State University in Fort Collins, says that most of the journals to which she submits articles request at least five potential reviewers.

For most of the 60 articles retracted by SAGE, the original peer review had used only author-nominated reviewers. Despite this experience, the Journal of Vibration and Control still allows authors to suggest peer reviewers (and provide their contact e-mails) when they submit a manuscript — although more safeguards are now in place, says Gamboa.

The Committee on Publication Ethics (COPE), which serves as a kind of moral compass for scientific publishing (but has no authority to enforce its advice) has no guidance on the practice, but urges journals to vet reviewers adequately. Good practice is always to check the names, addresses and e-mail contacts of reviewers, says Natalie Ridgeway, operations manager for COPE in London. “Editors should never use only the preferred reviewer.”

NPG journals do allow authors to suggest independent reviewers. “But these suggestions are not necessarily followed,” says Kiermer. “The editors select reviewers and the selection includes checking for the absence of conflict of interests.” On the flip side, authors can ask an editor to exclude reviewers who they believe to have unmanageable conflicts, such as competing research. The publisher usually honours such requests, as long as authors do not ask to exclude more than three people or labs, Kiermer says.

Sometimes, recommending reviewers can backfire. Robert Lindsay, one of two editors-in-chief of the Springer-published journal Osteoporosis International, says that his publication allows authors to recommend up to two reviewers — but that he often uses this information to rule those reviewers out. This is based on past experience, in which he has seen authors recommend their own contacts, or worse: “We have had family members, folks in the same department, postgraduate students being supervised by an author,” he says. The journal generally uses suggested reviewers — who have passed screening — only if it runs into trouble finding other scientists to perform the task.

But screening can be difficult. Usually, editors in the United States and Europe know the scientific community in those regions well enough to catch potential conflicts of interest between authors and reviewers. But Lindsay says that Western editors can find this harder with authors from Asia — “where often none of us knows the suggested reviewers”. In these cases, the journal insists on at least one independent reviewer, identified and invited by the editors.

In what Lindsay calls the worst case that he has seen, an author suggested a reviewer who shared her first name but not her surname. Some investigation revealed that the surname was the author’s maiden name — she was recommending that she review her own paper. “I don’t think she is going to submit anything to us again,” says Lindsay.

Nature
515,
480–482
(27 November 2014)
doi:10.1038/515480a

Author information

Affiliations

  1. Cat Ferguson, Adam Marcus and Ivan Oransky are the staff writer and two co-founders, respectively, of Retraction Watch in New York City.

RE: http://www.nature.com/news/publishing-the-peer-review-scam-1.16400

 

The Old City: Leviathan Review: In the Dust of this Planet

Title: The Old City: Leviathan
Price: $15
System: PC
Release Date: Dec 1, 2014
Publisher (Developer): Postmod Softworks (Postmod Softworks)
ESRB Rating: N/A

“You and I have shared a grand nothingness.”

—Solomon

“When meaning is deconstructed, the only driving force for progress is the acceptance of uncertainty.”

—Jonah

Half walking simulator, half existential novel, The Old City Leviathan is a deliberate, sober examination of both the human mind and its ability to cope with the traumas and contradictions intrinsic to life. It confronts the intellectual roots of human suffering head-on, pouring through conflicting ideologies and the ultimate cynicism they finally engender, making a serious effort to develop a new state of mind that will free people from endless cycles of destructive thought. It’s a rigorous, unflinching examination some of the worst aspects of being human, and Postmod Softworks was incredibly brave to make it.

The Old City: Leviathan probes the bleakest corners of life in search of hope, embracing insanity, violence and despair in a genuine search for some way to live freely and meaningfully amidst the ruins. Despite all this, it is not a dark game. It contains elements of horror, madness, and death, but there is beauty in the fallen world it depicts, and the game never stops encouraging you to move forward, even if you don’t know where or why. The Old City: Leviathan features a character named Jonah, and like his biblical namesake Jonah’s story is one of isolation and struggle, yet he emerges from it renewed. It makes for a remarkably deep experience, despite only lasting about four hours.

The Old City Leviathan Review (4)

After the Fall

Jonah inhabits an island that became a refuge after civilization collapsed—or at least it was, before the everyone split into different factions and began killing each other. Referred to as the Fall, the collapse of society is described as a gradual, unremarkable affair that was the inevitable result of the erosion of the ideals holding civilization together. Fleeing the Fall, people retreated to the island and revived old ideologies in different forms to serve as the foundations of their new society. Called Dwellers, the island’s inhabitants settled in the underground tunnels and buildings on the surface, all residing outside the walls of what they called the Old City. They only drank water that had been filtered on an industrial level, since Unclean Water exposed them to a state of mind called Void that causes hallucinations, severe migraines, and thoughts of suicide. Jonah admits to having ingested Unclean Water, but claims an entity called Leviathan saved him.

Without going into too much detail, the factions in the game are the Guild of the Greater Eye, the Unknown, the Order of the Cosmos, and the Minotaurs, representing spirituality, agnosticism, materialism, and nihilism, respectively. Inevitably, war broke out between the various schools of thought as each concluded that theirs was the one true belief, leading to the destruction of their home.

The Old City Leviathan Review (10)When the game begins the internecine wars are over, and Jonah is the last living soul on the island. Save for when Jonah speaks to Leviathan, the entirety of The Old City: Leviathan‘s story is told through the written word. Typed journals and letters are plastered on walls and left on desks. Graffiti is scrawled on the walls in forms that range from obscene puns to furious outbursts of despair to ornate graphs representing entire cosmologies of belief. Obscure maxims are carved in stone.

The game doesn’t have a save feature, rather you pick a level and you always start at the beginning. Each level was brief and self-contained enough that it never really became an issue for me. You can get through each of The Old City: Leviathan‘s 11 chapters in just a few minutes if you so desire, but if you want to understand how the island reached its present state and the ideas that led it to ruin, you’ll have to venture away from the direct route and explore. The messages you encounter both reveal the world Jonah occupies and obscure it, showing human beings in diverse states of conflict, faith, introspection, and despair as they desperately cling to their beliefs, then watch them slip away. It’s a mystery why Jonah has survived while everyone else perished, though the entity he calls Leviathan may have something to do with it. The war having burned itself out, we travel with him across the areas of the island surrounding the Old City, experiencing his reactions to the destruction around him, his philosophical ruminations, and his hallucinatory madness. It’s a meditative experience that encourages many conflicting interpretations, and I have no doubt that some people will find it utterly unbearable.

The Old City Leviathan Review (7)

Getting philosophical

If you like first-person exploration games but existential philosophy makes you gag, you should skip The Old City: Leviathan. Even if the Unreal Engine Postmod Softworks used is starting to show it’s age andP.T. spoiled us all for looking closely at stuff in games, The Old City: Leviathan‘s graphics are good. However, all the game’s environments are reflections of inner conflicts, and if the player can’t invest in the interplay between the two, the game will have no impact. The Old City: Leviathan owes much toDear Esther in that way, as well as a great many other ways.

Dear Esther and The Old City: Leviathan both feature men exploring a bleak but attractive landscape devoid of human life. The protagonists in both games are probably insane, experiencing hallucinations and spending the game talking to someone who isn’t there. The men in both games are on a kind of pilgrimage for something intangible: in Dear Esther it’s the search for closure, in The Old City: Leviathanit’s the search for a replacement for certainty. The Old City: Leviathan even has you leap down into darkness to transition between levels like Dear Esther does. Both games consist entirely of wandering around in a first-person perspective and looking at stuff. The list goes on.

The Old City Leviathan Review (1)
Despite all this, the similarities never bothered me, nor did I ever think it showed a lack of creativity on Postmod Softworks’ part. My opinion would probably be different if there were a dozen games coming out with the same formula, but as it stands now the medium is still novel enough to adapt different ideas without getting stale, and Dear Esther and The Old City: Leviathan are very different in this regard. Dear Esther is about one man’s grief. The Old City: Leviathan is about finding a way to live in a world that defies hope. Aside from an an affinity for caves and bending reality, the two games have little in common besides their formula.

The Old City: Leviathan is much more hallucinatory than Dear Esther, and much more violent. The consequences of the Fall are all around Jonah, the grim reminders of people killing each other in pursuit of a belief, or falling prey to Void and killing themselves. Jonah’s mental state causes hallucinations to appear in the real world, but he also leaves it completely in favor of what he calls Dreams. Entire levels are set in a strange fantasy world of Jonah’s creation, a world where palaces and sculpture gardens float on islands in a heavenly sky, and great bestial titans slumber peacefully like fantastic ideas yet to be realized. In this way Jonah is able to split his time between an exterior world of conflict and an interior world where ideas are free and don’t hurt anyone. The Old City: Leviathan is the story of the island’s destruction, but it’s also about the world inside Jonah, as well as a tragic, self-negating character named Solomon.

The Old City Leviathan Review (3)

One man’s experience.

Throughout the game Jonah can find collectibles called Solomon’s Notes, each of which are first-person journals kept by a passionately miserable man who lived before The Old City: Leviathan begins. Totaling seven in all, Solomon’s Notes form a picture of a brilliant mind furiously torturing itself. Solomon can’t help deconstructing every thought, impulse, feeling, or action he encounters in himself or anyone else, a cycle of destructive over-reflection that’s led him to loathe people, the world, and himself. Solomon and Jonah are both mad, but while Jonah is steeped in fantasy, Solomon suffers from a debilitating excess of rationality. Solomon’s Notes resides in the rich tradition of existential novels narrated by angst-ridden characters such as Notes from the Underground, The Fall, and Steppenwolf. In short, if you were miserable in high school, Solomon’s Notes will be immediately familiar.

The Old City Leviathan Review (2)Solomon’s Notes aren’t “notes” anymore than the Underground Man’s fevered journals in Notes From The Underground are notes. Taken together, the seven entries are long enough to form a 30,000 word novella unto themselves. They’re far too long to read immediately after you find them without becoming completely removed from the game, which is entirely contrary to the spirit of a game that strives so hard to establish a sense of place. Though I know they don’t have the resources for it, I wish Postmod Softworks could’ve published the entirety Solomon’s Notes in a supplementary book. As it is, Solomon’s Notes are best accessed from the game’s main menu and read at your leisure. The first entry is too long and meandering, but they get shorter and more interesting as you go on. They ultimately prove tragic, horrific and moving—in other words, well worth your time.

The Old City Leviathan Review (11)There’s a chapter in Solomon’s Notes where he encounters Jonah, prompting an exchange where we learn more about Jonah and his beliefs than we do in the game itself.  The Old City: Leviathan ends abruptly, leaving us to ponder the questions it’s posed and the character we’ve inhabited. Who is Jonah? Who was he before he went insane? What is the nature of the entity called Leviathan he spends the game talking to? Is it related to the biblical Leviathan, or the societal Leviathan of Thomas Hobbes’ book? Why did people shun the Old City?  We don’t know. There is no resolution in The Old City: Leviathan, in fact, the ending will likely be greeted with a great, collective “That’s it?!” Like the characters in the game, we’re forced to draw our own conclusions.

The Old City Leviathan Review (5)

A thoughtful experience.

The Old City: Leviathan doesn’t indulge in what is pejoratively referred to as “dorm room philosophizing”. It isn’t Heidegger, but it goes deep, and expects you to follow. Along the way are environments that range from the industrial to the abstract, accompanied by a fitting score of music that echoes, fades, soars, and rumbles, moving seamlessly between genres that range from ambient music to Gregorian chant.  It’s not an easy journey or even a very pleasurable one, but I feel like it moved me.  When the Leviathan finally spits us out we emerge optimistic, or at least determined to pioneer a world devoid of ideology or certainty.

Postmod Softworks says on their website that The Old City: Leviathan is the first part of a larger story. They’ve successfully drawn me in to the point where I dearly want to see what they do next.

 

REF: http://webcache.googleusercontent.com/search?q=cache%3Ahttp://www.technologytell.com/gaming/138750/old-city-leviathan-dust-planet/