First and foremost and in case it’s not clear I find this forum a great idea.
I have a question wrt the public discussions here and the double blind peer review process.
On the one hand we are a small community posting publicly with our real names. On the other, our questions and data and problems are very specific: it won’t be too difficult to make the link between a post here and an anonymous paper I might have to review in the next months, which is not especially true in other technical scientific boards.
For example, while a lot of people work on/with BERT and questions on stackoverflow/reddit/wherever can remain anonymous or drowned in the sheer amount of similar questions, not many people have to spelling normalise a 500k-letters corpus of Dutch that is geographically ranging from Groningen to Ghent (or study nation building in Swedish-language newspapers from Finland, or sexism in Dutch ads using CV, etc.). Obviously in normal times we know what others work on and might have ideas/theories as to who wrote this or that paper, and if we email someone for advice/help/opinion we mark them as reviewing conflict when submitting. But this forum is public, probably crawled and indexed, etc., so it’s different.
I don’t think making the forum members-only would be a solution either, as I guess ideally this community would be as inclusive as possible.
My questions are then:
Are we not shooting ourselves in the foot? If this community is to bring like-minded researchers and those same researchers can’t review in good faith, then it’s problematic.
That is an interesting and difficult question, which, in my opinion, is a bit related to part of the culture of humanities research. In comparison to other sciences, humanists often keep their data and methods to themselves. Even after publication, it is often difficult to get the data that comes with a paper. In a sense, the same also applies to research ideas. In other sciences, it is much more common to publish preprints, which obviously makes it even easier for a reviewer to track the author of an article. Practice has taught us that this should not be a problem, and I don’t have much trouble with that myself. In fact, I think it contributes to the transparency of research. And a forum like this can also contribute to that transparency. In addition, not all questions and topics need to be about concrete research ideas or research questions. And hopefully you can also get good advice here if you don’t want to give up your new super cool idea already.
Sure, but on the other hand they also don’t do double blind review, sometimes not even single. I recall having a conversation with a more senior historian who told me that “peer review is whether you get booed out of the room or not when you give your paper” (paraphrased).
It has? This is anecdotal and solely based on my twitter feed, but I recall big names in NLP complaining that their papers were rejected after xACL started implementing the anonymity period ~2 years back. Somehow, now that reviewers didn’t know who wrote the papers, the papers started getting rejected.
Yes, sure. Hopefully we can discuss more “meta” things like methods (like Melvin’s recent posts for example), but there will inevitably be talks about very specific data.
Sorry if I didn’t make myself clear – my point is not about “idea stealing”/flag planting, it’s about breaking anonymity/double blind peer review. I doubt anyone really cares enough about what I’m currently working on right now to be heavily copying it. Also, hopefully, CHR is about interesting outputs and not leaderboard chasing. I would love to discuss some things here – I’m actually currently working on new methods I am not 100% sure I master and could use some feedback.
What I am afraid of is that if I describe it here and ask for help, other forum goers (who would be the target group for reviewers) would have to declare themselves as conflicts of interest since they would know it is me. In the past I have emailed people to ask questions and when I submitted papers I listed them as conflicts (and added them in the acknowledgments of the camera ready ), but the publicity of this forum changes the game.
I now understand beter what you mean. Thanks for clarifying. It’s a fair concern, and I don’t immediately have a concrete solution. The model for this forum is the forum used for the statistical programming language Stan https://discourse.mc-stan.org/. Now of course, for Stan it is easier to ask questions about more practical matters, but here too, more general questions are asked about how best to tackle a particular modeling problem. I don’t know whether that can lead to or has led to problems with peer review. I guess we may need to learn how to ask questions on our forum without compromising the peer-review process. Given the benefits that a forum brings, I really hope that’s possible.
We could also consider using pseudonyms, but one of the goals of the forum is community building. Especially with the workshop taking place virtually, I think this place could help foster this. Nonetheless, these are valid concerns and we should keep discussing them.
Personally, I think it’s possible to review/discuss work by colleagues and friends, while staying critical and supportive at the same time.
I agree with Melvin. In small communities it’s pretty much impossible to not know or be able to make a good guess who (some of) the authors are of some of the papers you review. That in itself is not a reason to declare a conflict of interest. What I learned growing up as a researcher in Information Retrieval is that COIs are for when you’ve recently co-authored with someone who’s submitted a paper for review, or when you’re from the same institute/department.
Within small research communities, you’re more likely to know those who have similar expertise as you have, and who are therefore more likely to be able to review your work. We rely on each other’s willingness to fairly and critically discuss each other’s work. Good guidelines for reviewing can definitely help with this. An alternative would be open reviewing where the reviewer’s name is attached to the review, but that also has its limitations.
True, but I think @marijn.koolen made a fair point that it’s difficult to go for complete double-blind review in a small community. In an ideal world, you could aim for this, but still, you can often infer who the author is. In my post, I should’ve been more precise. I meant reviewing colleagues who I know, but have not co-authored with. w/r/t friends, most often we discuss research before submission. And it’s ok to raise these issues, it’s easy to develop blind spots or to become to laissez-faire concerning bias when reviewing.
), but the publicity of this forum changes the game.