Virtual Panelists and Thoughts on Assessing Science

(Copied from a February post on my "home" blog -- its still timely)

I was quoted (correctly) in a commentary in Science entitled “Meeting for Peer Review at a Resort That’s Virtually Free” (http://www.sciencemag.org/content/331/6013/27.full) — nice. The article talks about the advantages of using virtual technology to convene scientific review panels at the National Science Foundation (NSF) and other Federal agencies like NIH instead of flying the panelists cross country for a physical meeting of a day or two. The article focuses on the very cool technology of virtual worlds, like Second Life (http://secondlife.com/), to host such activities, but video conferencing and phone are alternatives. Some might think that a phone is primitive technology; perhaps, but a land line is extremely reliable and not as primitive as an airplane, at least relative to the task of putting talking heads together.

In the Science article the lead reason presented for using virtual participation for NSF (NIH, etc) is that it saves money. I am somewhat conflicted on how to talk about this. On the one hand, the article says that approximately 19,000 reviewers were used by NSF last year; even if all of these had traveled to NSF, but were replaced by 19,000 virtual panelists, saving $1000 per panelist (all VERY optimistic), that would be savings of $19,000,000 (19 million); that may seem a lot, but only a dent in NSF’s 7 Billion dollar yearly budget, which in turn is only a wafer thin (0.0002) slice of the federal budget of 3.64 TRILLION dollars; http://nationalpriorities.org/en/resources/federal-budget-101/budget-briefs/federal-discretionary-and-mandatory-spending/). Nonetheless, the current budget debates suggest that President and Congress are focused on pocket change, so perhaps demonstrating any small budget cut would be the sacrificial lamb needed to buy NSF and science/engineering generally some political goodwill; and when such savings are spread across all such travel across all agencies, it could be significant.

I worry though that if monetary savings are presented as the lead story when talking to researchers/panelists, it will convey the (wrong) message that quality in the review process is being sacrificed to save money, and for the researcher/academic this could easily be a source of disappointment, if not resentment — after all, every two days the national debt grows by over 8 billion dollars (http://www.usdebtclock.org/), comfortably more than the yearly NSF budget! Amazing! Deeply discouraging. If virtual participation on panels picks up, it would be terrible to have scientists and engineers (or heck, anyone who cares about the US’s future) falsely believing that a few bucks are being saved out of the hide of science and engineering research in the US. Thus, I winced a bit that monetary savings took the lead in the article.

In my three years at NSF (2007-2010), fully 3/4 of my over 200 panelists were virtual. While I went in to NSF with an interest in virtual paneling to reduce ecological footprints, what pushed me over the edge (almost immediately) was that I couldn't find available rooms for panels on the days I wanted to hold them. I quickly saw more and more reasons for virtual paneling (as did others), and I recorded my experience on the NSF wiki. On my last active day as a program director, a Saturday (8/21/10), I mass emailed the entirety of NSF on the virtual paneling experience before leaving the office for the last time :-), and even on a weekend, received responses from interested program directors before my email address expired at midnight!

The reasons for using virtual panelists, and particularly in giving panelists a CHOICE on going to Washington DC (or anywhere, and for any organization) or participating virtually are many. The article does NOT address most of these reasons.

(1) Virtual participation reduces travel wear and tear on panelists — West coasters, rural and others underserved by airports, those overseas — this was alluded to in the article; travel is a great inconvenience/impossibility for many, but its a pleasure to many others, and/or an opportunity to network. I like a system in which panelists weigh the costs, benefits and choose for themselves on whether to attend physically or virtually. So, rather than suggesting an artificial dichotomy between all physical and all virtual, lets recognize that there are hybrids that allow both. Behavioral economics suggests that one can influence the proportions of the two kinds of participants by making one form (e.g., virtual) the default, and indicating the other option (e.g., physical) as welcome. If default specification strikes some as “mind control,” I suggest that its preferable to requiring one form (e.g., physical).

(2) Virtual participation broadens participation to many who might not otherwise serve — parents of young children, senior and very busy researchers, those who have to teach a class, and/or those who an agency might not otherwise ask because of monetary cost, such as those overseas, who may well have special expertise in an area that would benefit the US.

(3) Virtual participation reduces wear and tear on government agency staff; this reason probably would be the most under-appreciated by those outside of government agencies; I am talking here about administrative staff primarily — arranging catering, cleanup, travel and reimbursements, and many other misc responsibilities require a lot of time. Many federal admin staff commute an hour or more EACH way and have oodles of other responsibilities to which they must attend. Early in my 3-year NSF career I attended an outreach talk at San Jose State University, in which the NSF/OLPA presenter showed a graph of the number of proposal actions by NSF, and these actions were growing at what was clearly greater than a linear function over the last 25 years. In this same period staffing numbers remained flat. The implications for workload are obvious. I don’t see these trends changing — even when the President and Congress agree to funding increases, it is for new programs that come with additional overhead, and not for increased staff; the research community, given the historically very low funding rates, will continue to push proposal pressure up. Streamlining in operational efficiency is absolutely necessary. Using virtual participants offloads burden from staff, so that they can do other necessary things and maintain quality.

(4) Virtual participation can improve important aspects of the proposal vetting process; giving choice to prospective panelists can only increase acceptance rates among them, increasing the number of first choices among the experts, and reducing workload by those having to research and solicit prospective panelists. Again, these time savings get channeled into other important activities.

(5) Virtual participation does not diminish the quality of information necessary to make funding recommendations. Some nominal skills as a moderator are required to insure that all panel voices are heard, important issues debated, etc, but one could accurately assert the same need for nominal skills for all-physical panel moderators too. My perception is that virtual participants are every bit as well prepared as physical panelists and as attentive, and not harried or worried about catching a flight. I sometimes hear push back that there is “something about physical presence”, and there *is*, and much of its good, but its irrelevant for purposes of making science funding recommendations. Indeed, facial expressions, winks, and hand gestures are relevant to surviving together in the jungle, but if they are really nontrivial factors in panel recommendations, then respectfully, you are begging for over fitting the data.

(6) Virtual participation decreases ecological footprints. In almost all white papers on the environmental impacts of using information technology, the most common *proposed* POSITIVE impact is to offset footprints in other sectors, notably travel. And so why isn’t this actually happening?! This is the low hanging fruit of the promise of information technology, and if technologists don’t start exercising its promise, there is really no point in expecting others to do so. That said, I have a friend since kindergarten who is an airline pilot, a sister-in-law who is a flight attendant, a friend whose spouse is a pilot — facing the financial consequences for many people of changing lifestyles has to be part of what we worry about — but this worry can’t stop us from acting to change unsustainable lifestyles either!

Again, there are things important about physical presence that can’t be beat — at a minimum, the intellectual and social reward to a panelist, as well as the moderator — I loved dinner out with those few panelists that came to NSF for panels that I ran, Networking is important too, where it is possible that hallway talk will lead to an exchange of ideas and perhaps even collaboration. But this doesn’t happen much, certainly not to a degree suggested by theory, with hallway talk getting displaced by checking email and cellphone chatter. But if this networking were really a desired capability, then lets do it deliberately, do it by design, and do it virtually, in small groups and at regular intervals — why leave it to chance, as some accident of flying people across the country.

But in any case, I would hope that allowing choice to attend physically versus participate virtually was part of meeting designs in the future. Heck, if the timing was right, I could set up other meetings, sightsee and otherwise make good use of the trip, I’d go physically — I don’t know of anyone who likes slapping a back as much as I do.