12:08
(upbeat music) [JULES VAN BINSBERGEN] Welcome to the Lauder Institute at the University of Pennsylvania. I'm Jules van Binsbergen, director of the Institute and a finance professor at the Wharton School. [JONATHAN BERK] And I'm Jonathan Berk, a finance professor at the Graduate School of Business at Stanford University. This is the All Else Equal podcast. (upbeat music continues) [JULES VAN BINSBERGEN] Welcome back everybody to the last episode of the season for All Else Equal.
Jonathan, I think it was another fantastic year. We had so many interesting guests and topics, and we heard back from so many of our listeners that they absolutely loved the topics that we've addressed. And so I'm very happy, and I'm also very happy with the topic that we're doing today, which is on informed discourse, an initiative that you have recently started with a number of faculty members, and I've gladly signed on to that project. Let's tell the listeners what the main question is that we want to address today Which is how should academic discourse take place? And is the journal system that we've had for the last, what, 150, 200 years,
really still the best system to have academic debate in? Or maybe it is time that we reevaluate that question and come up with, given the new technology that we have, better alternatives to the journal system? [JONATHAN BERK] Well, Jules, certainly the popular press and certainly amongst non-academics, people feel as if the system is in crisis. I think we'll have a slightly different take today. I think we both agree the system is broken, but I'm not sure for the reasons that the popular press do. So, why don't we begin with what I perceive as the source of what I consider to be the main problem
with the system today. Fifty years ago, the only technology for distributing research was on paper. In an academic journal. Okay, we didn't have photocopying machines. We didn't have any way to reproduce the papers ourselves. So you'd spend a lot of effort. Obviously, it was very costly to produce a journal. And so there, a lot of effort went into producing the papers, and they were distributed. After all that effort went into place. And most importantly, because of the shortage of space
and the enormous resources that go into putting together a paper journal and distributing it, there was a process for deciding which papers should be published and which papers were not published. And back then, and different fields differed, but in economics, the way it worked was there were a few elite journals that everybody agreed about, and academics competed pretty rigorously to get papers published in those elite journals. There weren't that many journals. And so what it meant was, as a consumer of research, it was well within your intellectual ability. In other words, say you were economists, you know, the top journals were the American Economic Review,
the Journal of Political Economy, and Econometrica, say. So it was three journals. You could add a few more for field journals, like applied journals and theory journals. So let's add, because you're in a field, say six journals. Each journal had, say, six papers. So we're talking 36 papers, and the journals came out four times a year. So we're talking about between 100 and 200 papers a year that you would have to look at. Yeah. Not read, just look at and read the abstract. So it's entirely within the number of hours in the day to process that and decide what to read. And that was the world 50, 60, 70 years ago. And then technology arrived. And fundamentally upended this world.
In essentially, first the photocopying machines that people then started distributing the papers themselves by mail, also costly and somewhat inefficient. But then the internet came around and it became free then to distribute your papers electronically. [JULES VAN BINSBERGEN] We need to take one quick step back here, which is the question: those between 100 and 200 papers that were published at the time. There was already the question: way more papers were written, but only those papers between 100 and 200 were published, and there was a system in place to decide which 100 to 200 papers would be published and which ones wouldn't.
[JONATHAN BERK] Absolutely. That's the peer review system. And let's be clear about something. The peer review system has never been perfect. I'm sure it was about as imperfect then as it is now, meaning that many bad papers and papers that are probably wrong got published And more costly, good papers did not. And so, you know, we have a colleague, Mark Rubenstein, who sadly passed away, who used to say that if a major academic idea was named after somebody, it was almost certainly true that they didn't discover it. So basically, the peer review system, and it's true today too, is biased against truly
new ideas, and often those ideas get rediscovered because the papers aren't. So certainly that was a problem. The first problem of bad papers being published, people were less concerned about because their feeling was They die anyway. People don't pay any attention to them, and so that's okay. And the old system was pretty good at that because you only saw 100 or 200 papers a year, so that people, when they saw the curated papers, could actually keep curation going. And a bad paper, they would just ignore, and the good papers, they would reference, everybody would reference it. And so I think that system Worked as well as one could expect. But then what happened was distribution costs went down,
people started distributing their own papers. And so now the question was there was no longer this curation. And different fields in academia are different in this respect. And I would say the two fields that I think are on the extremes, there's our field, economics, which is one extreme, which is The review process takes an enormous amount of time, maybe up to four years. I know that sounds ridiculous to our listeners, but it's not. Or the alternative is medicine, where the lag is very, very short.
It's a matter of months between a paper submitted and when it comes out. But the problem is in medicine, the review process is greatly flawed. As far as I can tell, many, many, many papers that are basically claiming stuff that isn't right get published. Much fewer in economics, although again, we're not perfect. But there's this inherent trade-off. [JULES VAN BINSBERGEN] So one thing that I do want to discuss, I think I may agree, although I'm not sure with your statement that the peer review process is as flawed today as it was in the past. But there is one thing, and we talked about this in the show in general many times, which is that when you evaluate a system, you do need to think about what are the incentives of
each of the individual players that are in this equilibrium. And so, for example, the question, what is worse for an editor? Not publishing a pathbreaking paper because it was too new and you couldn't judge it and you didn't want to take the risk, or publishing a paper that ended up being wrong. And that might be different between different fields, how those incentives work for the editor. And then of course, there's the very big piece which is, what's the incentive for the referee?
You're being asked to referee a paper. We can all agree that the compensation, the financial compensation that any person gets for refereeing a paper is nowhere near what the hourly rate would be if somebody would truly have to spend the multiple hours really evaluating the paper. And so that does raise the question, both in the old system and the new system, what is the incentive for a referee to write a good report? [JONATHAN BERK] Okay, Jules, you're right. Incentives are a key feature. So let's focus on economics, because that's the area we know best about. So let's just describe for our listeners how it works.
I'm not sure all of them understand this, but there'd be a high prestige journal. So Jules and I are financial economists, so the highest prestige journal is the Journal of Finance The Journal of Finance has a board of editors that makes the decision, the ultimate decision of whether it's published or not. But that depends on asking regular academics like Jules and I to evaluate the paper and write what is called a referee's report. There's essentially no compensation for writing those referees' reports. And if you do a very good job, it is at least a day's work, maybe two days' work. So it's enormously costly. To do a referee's report with very little compensation.
And so the question now is why do people do it? Most importantly, it's anonymous. The author never finds out who the referee is. So given that anonymity, you'd think there'd be even less incentives for anybody to either do a report or do a good job on the report. [JULES VAN BINSBERGEN] For sure. [JONATHAN BERK] In truth, the only incentive is the editor. Because most academics would like to get published in the Journal of Finance. And so what they do is they feel as if they do a good job on the referee report, the editor will treat their papers with a more favorable light and in particular, choose higher quality referees for their papers. And I think that effect is probably there.
[JULES VAN BINSBERGEN] Just to be clear also for our listeners, there's absolutely no formal quid pro quo system in this that you just described. This is all implicit. And so as a referee, you hope that if you write good reports, that the reputation that being a good referee gives you will give you some benefits. But it is in no way - your referee report is never scored. In some journals it is, but in many journals it's not, and you definitely won't hear back about what the score for your referee report was, and therefore, there's also no formal expectation that you can have that you will
be treated differently if you wrote a good report. But a lot of people do think implicitly that they will get rewarded for it. [JONATHAN BERK] Well, Jules, I think the following is almost certainly true, which is if you're a junior faculty member and you're not very well known, and you write a referee report that is deep and insightful, the editor is likely to remember that you're not a fool And if you're not a fool and you submit a paper, and he's thinking about which referee resources to use, he's gonna realize, "Well, look, there's some chance that this paper actually is interesting."
And so I should use the refereeing resources that I have on this paper. So purely from a selfish perspective, not from the editor doing the author a favor. Now, there's also the fact that he may wanna do him a favor, but I think that is what most people think of the incentives. I think we should also say something that's unique to economics is the acceptance rate. Economics has unusually low acceptance rates. As far as I know, the Journal of Finance acceptance rate is around eight percent.
Yeah. So the vast majority of papers are rejected from the Journal of Finance. And so it's an extremely big signal to get accepted, and therefore, a lot is hanging on this process for individual faculty members, and particularly true for junior faculty members. [JULES VAN BINSBERGEN] No, for sure. And then just to give our listeners some context, there are many universities in the world where one Journal of Finance publication will give you tenure. [JONATHAN BERK] Yeah. I mean, obviously not a place like Stanford or Wharton, but yes, it's a big deal to get published. And you might think with all this oversight that the quality would be very high.
But I think the biggest problem we have is the quality isn't that high, that there's many, many mistakes made. And so then the question is why? And it gets down to the question of incentives. How incentivized are referees to do a good job? So their incentive is not to do a good job, but to impress the editor. Let's assume that's true. I think most people wouldn't disagree with me on that. And so what a referee does is find ways to impress the editor without putting in as much effort as he would otherwise have to do.
What that invariably means is the referee finds low-hanging fruit in the paper, right? Small little things that are easy to find that he can ask the authors to do and that somehow convince the editor that he's doing a good job. But the really deep stuff he doesn't have to spend. And that has happened in referee reports. So what referee reports have become today are these lists of things that authors are expected to do and As I said, the acceptance rate of the Journal of Finance is 8%. The way it works is you don't send a paper in and it gets accepted.
You send the paper in, it goes for referee. The referee has a bunch of comments, and the good news is the editor says, "Please revise this paper along the referee's comments and resubmit it," and then sends it back to the referee. If the referee doesn't say, "Okay, yes, he's addressed my concerns." You get a rejection. And we just said how important an acceptance is. So you're forced to do whatever the referee wants, regardless of whether it makes any sense at all, regardless if there's a trade-off. Oftentimes it may make a little bit of sense, but it takes an enormous amount of effort, and nobody would ever make that trade-off, but you're forced to. And then what happens is it goes back and the referee wants
to impress the editor again, so they find a new problem. So often you can go two or three rounds with the referee, each time involving more work, which essentially means that there's a huge amount of wasted effort. So junior faculty especially spend an enormous amount of time, instead of doing new research, revising papers in ways that don't really improve the paper. And that's really a problem with our system. [JULES VAN BINSBERGEN] And I think that one question that you could ask, I don't know whether somebody's done that, is that if you do a survey of all of the published papers and you asked authors the question,
and of course they're biased, but it still gives you some sense, how many authors would admit that due to the refereeing process their paper got better, and how many of them would convincingly argue that due to the refereeing process their paper got strictly worse? And obviously, given the fact that so many resources go into the process itself, you would hope that there is some visible sign of the papers becoming actually better from going through this process. And I don't know how much evidence we have for that, that if you would run An experiment where, with 50% probability, you accept a decision on the spot, and with the other 50% decision, you would go through the whole process of the three rounds of revisions,
and then 10 years later, you would somehow evaluate which one of the papers had more influence or which ones were perceived to be better. I wonder whether you would see in that controlled experiment a difference. I'm worried about the answer to that question. [JONATHAN BERK] Yeah, Jules, probably there's a small increase in quality, but there's a huge amount of effort spent. So on the effort-to-quality trade-off, it's clearly negative. But even worse than that, there are externalities.
So what's happened in economics in the last 40 years is the length of papers has grown astronomically to the point where they're very difficult to read. [JULES VAN BINSBERGEN] Yes. [JONATHAN BERK] And why is that? It's because referees demand more and more irrelevant stuff to be put in the paper. So I have an interest in evolutionary psychology. I, you know, at one point thought of writing an economics paper And so I needed to read the evolutionary psychology literature, and I was shocked.
I could open one of these papers and completely read it and understand it. There's no way an evolutionary psychologist could open an economics paper and understand it. Our papers are so completely jargon-filled and so, so obtuse. [JULES VAN BINSBERGEN] Convoluted, yes. [JONATHAN BERK] Instead of just getting to the point, we're required to do literature reviews and put the thing in context and do all this stuff so that if you really want to know where is the meat in this paper, it really requires effort to find the meat in these hundred pages. And so there's this huge negative externality.
Now, the last negative externality is the length of time it takes for curation. So papers are distributed by individuals. The distribution occurs as soon as the paper is written. But to curation, the decision about whether the paper's worth reading is four years delayed. Meaning, it's useless for the purposes of deciding whether to read the paper. And now we're in this terrible equilibrium where there's just an enormous number of papers, way too many for any individual to process in any reasonable amount of time.
And so the net result is people do not really read papers. They may read introductions, they may read abstracts. And so there's this enormous benefit to writing a jazzy introduction without necessarily having the academic rigor to support it. I think the overall quality of the papers, because of that, has dropped dramatically. And I think in that sense- The profession is in crisis. And so that's this context that I decided to try to do something to fix the problem. [JULES VAN BINSBERGEN] So yes, before we go there, information technology has made it very simple to distribute a lot of work, and we also need to be cognizant of the fact, I think, that universities also, due to increased demand for teaching, have just grown,
and all of the faculty members in all of these departments had, as, sometimes for good reasons and sometimes for bad- Reasons, perks of doing research. And so the total amount of research output has also just exploded. So if you take a lot of research output, you combine that with a very easy distribution model, you essentially get signal jamming, where being able to separate the wheat from the chaff and what should be read and what shouldn't be read has become even more important. But for some reason, and then for some of the reasons that we discussed actually, the journals are just not that well able to separate the wheat from the chaff.
[JONATHAN BERK] Well, Jules, and there isn't a place to share crucial knowledge in real time. You know, I have in mind a paper that was written and published in one of the best journals in economics, a theory paper. It had a very strange result. And another researcher, who is obviously an expert in this area, was very puzzled, and he figured out the problem with the paper is the initial assumptions were mutually inconsistent. Now, the paper was published after that researcher distributed a note. You know, had there been efficient distribution of knowledge, the editors would have realized, "Wait a minute," I've got a problem on my hands here."
They didn't even know about this note. And so it's this strange situation where you don't have knowledge efficiently distributed. [JULES VAN BINSBERGEN] And so for this reason, essentially what you did was with a number of people, you've started a new initiative in which you let the technology not just work against us in terms of the signal jamming, but you let the technology work in favor so that the distribution of the knowledge, the crucial pieces of information about papers gets distributed faster.
[JONATHAN BERK] Yeah. So the idea is, you know, take this example of the guy who pointed something out about a paper that, you know, pretty fatal flaw in a paper that gets published. And the question is, do the people inside that area Share the knowledge. And in our current system, we don't have any way of doing that. We have no way to get that information to the relevant parties. And so that's really the focus of what I was trying to do. I should give credit where credit is due. I'm doing it with three other researchers: Matt Ringgenberg, Martin Lettau, and Willy Fuchs.
What we've done is we've created a website. On this website, people will be able to review papers and comment on papers. We called it Informed Discourse because it's a place where we envision people to talk about research as experts. So one of the most important things about this website is anybody can read the website, but only people who are true experts can contribute. One of the problems with the internet is there are too many people contributing who don't know what they're talking about. So we will have a system of only allowing experts to contribute. When I say experts, I don't mean only Harvard
University professors. I mean professors at top universities. Well-recognized universities. And where well-recognized is broad. All state universities will, of course, be top universities, but not anybody on the street who happens to think they know about finance. And a crucial part of this website is there's no anonymity. So the peer review system is anonymous, and people have argued for the benefits of anonymity. I personally think that what the web has shown us categorically is that people misbehave when they are anonymous.
We have to get rid of those incentives. So in this system, there's no anonymity, and people have reputations. And our idea would be your reputation is what incentivizes you to do a good job. Yeah. So that's the first thing. And then the second thing is to curate in a way that the group of researchers who are knowledgeable are most likely to get the papers that are relevant to them. And so the idea would be we would create a place that people would go for research. So everybody on the system would have their own personalized homepage. Where we would be curating the stuff that they see relative to their interest. So people would have reviews, papers would be reviewed on
the system, people would have comments on papers, and we would use that to curate. So think about all the information we'll know about any user. Any user, we'll know the papers they're reading, the papers they're putting in their library, the papers they're writing, the papers they're citing, the papers that cite them. We'll also know the papers they're reviewing. And the papers that they're commenting on. So we can use that information to figure out what research is this person likely to be interested in?
Yeah. And so we can curate a place where they can go to see the current research in a way that their brains can process. It's not an overload of research. And the idea would be we create a community that's called Informed Discourse, because the idea would be that scientific discourse and real-time sharing of knowledge to greatly improve the knowledge process and the research process [JULES VAN BINSBERGEN] And of course, if you present people with the papers that are closest to their interests, they're also more likely to
write reviews and comments on those particular papers. So in that sense, you curate the ecosystem. But of course, you could abuse that too, right, Jonathan? I mean, right now you do it in an automated way, which I think is very good. But, you know, I think that part of the problem that we're currently seeing in the social media landscape and in other places is that the information that we're seeing is so curated that it's curated by opinion. That's not the case here. The curation that you do is on topic, not on what is argued in the paper.
[JONATHAN BERK] Exactly. And of course, look, Jules, any system you write down can be abused. So my question is not whether this system is abuse-free, it's not, but whether it is less easy to abuse than the current system. So in the current system, one of the things I really don't like is the power of editors. So an editor has immense power. And many editors abuse that power. What I like about this system is there's no single individual that has any power. You know, the system does the curation, not any individuals. And so I like to think of it as a crowdsourcing mechanism for deciding research.
So it's the crowd that decides, not any individual. One of the things I really don't like about the current system is it's so uniquely determined by who your referee is or who your editor is. One or two people are making a huge decision about what happens to a paper, and I just think that in general, individuals are very idiosyncratic. It's a very idiosyncratic thing. Whereas if you let the crowd do it- It will lead to a much better system of how information is dispersed.
One of the other things I wanted to say is generally reviews in journals, in economics tend to be very negative, right? Even when the paper is on a path to acceptance, the reviews tend to be negative, which I don't think is very helpful. So in this case, the reviews will generally be positive, which I think is not a bad idea. But the real question is then what is a bad review? What is a review that says this paper isn't worth reading? And I think this is a crucial difference between this system and the current system. In this system, a bad review is no review No attention. In the current system, every paper seemingly has
a right to be reviewed. I send it to the Journal of Finance, the editor sends it to a referee, and I basically have a right to be reviewed. I think that's a crazy system. I think part of the paper being influential is it has to generate interest. Not forced interest. And the fact that a paper on the system gets no interest is information that is important and we should use, which we don't use in the current system. [JULES VAN BINSBERGEN] No, I can see that. You know, sometimes in democratic systems, you need to first have a number of petitions before you're allowed to petition something for a law or something like that.
Clearly, in academia, we don't have such a rule. As long as you pay the submission fee, then you can get your paper reviewed. And I think everybody agrees that if you have too many people that submit, increasing the price is not optimal either, because that is the wrong selection. [JONATHAN BERK] What if there are people that have really good ideas, but they just don't have the means to submit? Is that really the way in which you would wanna price discriminate? That doesn't work. But to come back to one other point that you said, you said to let the crowd decide, but let's also be precise, you let a very specifically selected, curated crowd decide.
So it's still the case that power is distributed in a particular way. And, you know, we're about to launch What we really are looking for are enthusiastic first-time initial users. 'cause obviously the more users, the more attractive the system is. So when you first start out, there's a problem because you don't have enough users. So we're really looking for initial people who are willing to help build the system. As you say, we signed up a whole bunch of thought leaders in the profession. One of them is you, obviously. But we have 20 other thought leaders, you know, influential people in financial economics signed up Who have written reviews on the system.
But it really has to be endogenous. People have to want to write reviews, and then we can use that to curate and give you an experience on the homepage. One of the other things we've done is we're gonna launch a conference system. There'll be a list of upcoming conferences, but if you happen to be a reviewer for one of those conferences, it'll be flawless for you to push a button, get the papers you're supposed to review, enter your reviews, and then that'll be passed on to the program chairman, and he will choose papers. And of course, we'll help the program chairman because what we'll do there is help them
Assign papers and reviewers, which for these big conferences can be a tremendously taxing task, right? So absolutely. If you have, you know, typical profits will have 200 submissions and maybe 60 reviewers, and those have to be assigned, and the system could do that. So we're doing that as a service to the profession. By the way, this whole thing is a service to the profession. [JULES VAN BINSBERGEN] Wonderful. Well, so the website is called informeddiscourse.com [JONATHAN BERK] Yeah, very good. [JULES VAN BINSBERGEN] I hope our listeners will go there and take a look at it.
It looks really great. [JONATHAN BERK] Okay, well, that's the end of the season for both of us, so we look forward to starting the new season. [JULES VAN BINSBERGEN] Thanks for listening to the All Else Equal podcast. Please leave us a review at Apple Podcasts. We love to hear from our listeners. Also be sure to catch our next episode by subscribing or following our show wherever you listen to your podcasts. For more information and episodes, visit allelseequalpodcast.com or follow us on LinkedIn.
[JONATHAN BERK] The All Else Equal podcast is a joint production of Stanford University's Graduate School of Business and the Lauder Institute at the University of Pennsylvania. It is produced by University FM.
