Cancelled Flights, Biodiversity, Infinite Sums, Heart and Soul

This was originally posted June 12, 2009 at 7:48am, by the same name, on another blog platform.

I’m at Ithaca Airport – my flight delayed, but that means some think time. There are times that an airline, and I’m thinking about one in particular, cancels flights and may chalk it up to mechanical problems when the truth is (I’m sure!) that the flight was under-booked. The “canceled” flight may be mine, which affects me directly, or it might be one “upstream”, which affects me indirectly as a flight delay (because they used my aircraft upstream for another flight). What’s odd is that when a flight is cancelled there is no statement about why it’s cancelled, and I don’t see anyone asking why it was canceled, at least not anymore. Sometime after taking the NSF job I actually went to the ticket agent and politely asked why the flight was cancelled and he was visibly surprised that someone asked!!?? After fumbling around, pecking at the computer terminal, the agent answered my query with “mechanical problems.” That’s amazing, because 20 years ago the agent would have announced up front why it was cancelled. Now, I don’t know for sure whether any particular flight is cancelled because of under-booking or perhaps the need for the airplane to serve a more heavily-booked flight which would otherwise have to be delayed, or the like, but thinking of it this way, be it true or not, has helped my serenity, because the canceled flight won’t produce Greenhouse Gas (GHG)! Carbon calculators will often reflect a savings for YOU if you fly less, but the only way that savings happen is if a flight is cancelled, and as people fly increasingly less, flights are eliminated entirely.

I’m now at Philadelphia Airport, my delayed flight having resulted in a missed connection, and now another wait. I’m serene. 

I had expressed concerns previously that biodiversity wasn’t on the map of the to-be-sustained from my experience at other venues, but MOST of the research talks at the Computational Sustainability Conference at Cornell reported on using computational and mathematical formalisms to reason about characterizing, monitoring, and protecting biodiversity !! Examples included the various approaches to addressing White Nose disease in black cave bats or fighting an atrocious disease mangling and killing Tasmanian Devils (which despite their name are awfully cute), estimating current and future distributions of a large variety of species, determining “optimal” policies for acquiring habitat, through purchases of private land, for the Grizzly bear, Canadian geese, Black Ducks, and other species (i.e., so-called ‘reserve design’), using computer vision to identify whale individuals, and GPS tracking to monitor seagull activity. A few hours ago, just before leaving, I listened to a coral biologist describe a differential equation model of disease transmission among Caribbean coral and their symbionts (new word for me) – the biologist is seeing rapid and pervasive evolution in coral (and pathogens!) through host-pathogen interactions, and this is but one example of moving targets in ecological research that makes computational and mathematical tools for analysis so important for handling complexity. Besides the ‘moving-target’ (aka dynamical) nature of ecologies, uncertainty in observations (how many geese were really parked at that lake?), and competing interests (the bear’s, the rancher’s, the tourist’s, the native American’s, the salmon’s, ….) as expressed by multi-objective functions, all make biodiversity questions complex. It’s a safe bet that the computer scientists and mathematicians are working on these questions because they ARE tough, BUT NOT TOO TOUGH, as they are problems that scientist’s can make a start at formally expressing. Nonetheless, this group isn’t satisfied with stopping at their current level of complexity, and a few economics talks are signaling anticipated scale-up to full-blown (human) societal questions – more later.

I know that there are a fair number people that would not put biodiversity on their list of priorities, and frankly, if push came to shove, I don’t know where I would put it, because I’m myopic. There was one session yesterday dedicated to the social science of sustainability – the human factor -- with one sociologist saying that it was almost impossible to change someone’s ‘values’, but you had a better shot at changing their ‘beliefs’ (of facts?) – changing beliefs was a better avenue towards changing actions/behavior than changing values. You can perhaps show how values are connected – maybe, just maybe, there is a transitivity property of values.

Speaking personally, its good to know that there are those investing their intellectual and physical efforts in preserving other species – that some egghead scientist spends her or his life worrying about some slug that lives in only one lake on Earth, for example, may be a way that Providence implements a caring for that species – I’m glad for those people, even if I’m not one of them – I respect them – I want to help them. Perhaps that’s how a caring for a species piggybacks on my caring for a person??

It’s also been interesting to see how concern with sustainability is implemented in mathematical and computer models. Let’s say that we want a species to persist into the future – into the very long-term future. Ecologists told us that one way they represent this long term assumption was that they summed the *discounted* value of having that species around to infinity UNDER DIFFERENT CONDITIONS, and then they (e.g., USGS) would be biased to take actions that would lead to conditions that maximized the estimated sum of values (or rewards). What is meant by “discounted” is that today brings greater reward than tomorrow – for example, MY being above ground today is most important; my being above ground tomorrow is important too, but slightly less important that my being here today – if I’m not here today, then I can’t possibly be here tomorrow. Likewise, it’s important (to me!) that I be here day after tomorrow too, but its value is slightly less than tomorrow’s, which again is slightly less than today’s. Now replace “MY being above ground” with “Species X being EXTANT” and instead of a timescale of days, think of a time scale of years.

You might think that summing over an infinity of positive values, regardless of what they were, would equal infinity regardless of circumstance, thus there would be no way to distinguish the value of different scenarios (except those for which a value term for a year goes to ZERO aka EXTINCT). But so long as the discounting over the years causes future values to asymptote to zero (approaching zero, but never quite reaching it), the infinite sum will equal a finite value (and there are probably other conditions in addition to approaching zero that are required to achieve a definite finite value for an infinite sum). This was all intuitive to me, but still I had reservations with future reward approaching ZERO … could we have a mathematical conceptualization in which having a species around was NOT infinitesimally close to zero? And so on the last day I was affirmed (which is always good), when an economist got up and talked about non-discounted reward valuations that were possible (presumably, not based on infinite sums?)– the distant future did NOT approach zero value -- apparently well known stuff to some!! This was a great illustration of why this conference was such a good idea. 

Discussion following the economist’s talk got into issues of valuing the current generation (YOUR/OUR experience), the previous generation (PARENTS), two prior (Grandparents), the subsequent generation (Kids), and two forward (grandkids), and its interesting to consider whether mathematical models that value a window of five generations (your grandparents through your grandkids) would lead to policies with long-term benefit (because this window of five would slide down the generational stream so that while our generation doesn’t explicitly worry about our grandkids’ grandkids experience, eventually someone will). The whole discussion reminded me of the Great Law of the Iroquois that states, "In our every deliberation, we must consider the impact of our decisions on the next seven generations." (and on which the company “Seventh Generation” takes its name), and on a quote I heard at the OECD conference a few weeks back in Copenhagen: “we don’t inherit planet from our parents, we borrow it from our kids”, imploring us to leave a better world to our children. In talking to one of the other participants, ML, we were also reminded of the social scientist’s talk of the day before about the near impossibility of changing values – but there might be critical points, corresponding to births (and deaths) of those we love where people are particularly open to value shifts – ideally, for the benefit of long-term sustainability of the planet we know and love.

Complexity, Computing, Citizenship

This was originally posted on June 7, 2009 at 9:33 pm by the same name on another blog platform.

Here is a quote from an article (http://www.news.cornell.edu/stories/June09/CompSustainConference.ws.html) on the conference I'm attending tomorrow:

"Conservationists will report on using computers to find the best balance of many competing factors in deciding, for example, which tracts of land to purchase to preserve a species, taking into account the budget, cost of parcels, economic impact on nearby communities and the best interests of the species. ... In projects like these, computers offer a way to try out thousands or even millions of possible scenarios, ..."

The scope of the conference is much broader than this one example, but all or most of the projects to be discussed will be looking at complex environmental and societal problems with multiple competing interests -- interdependent interests that have to be (or should be) balanced.

As my plane was touching down in Ithaca -- I mean just as the wheels hit the ground, I was reading this from "Soul of a Citizen" (Paul Loeb), quoting Wendell Berry:

"A bad solution is bad, because it acts destructively upon the larger patterns in which it is contained ... because it is formed in ignorance or disregard of them. A bad solution solves for a single purpose or goal, such as increased production. And it is typical of such solutions that they achieve stupendous increases in production at exorbitant biological and social costs."

and after more acknowledgement of Berry, and Loeb's inferences that "good solutions ... are part of a larger whole", Loeb goes on to say "We can contribute to the well-being of our society, the body politic, by applying a similarly holistic ethic of interdependence, and by listening to those whose voices are too often excluded from public discussion." (p. 131, Soul of a Citizen).

This reading was striking because acknowledging and balancing interdependent factors, concerns, and the like, is largely what this conference on Computational Sustainability is about, and using computer power to converge on and manage complex balancing acts which long-term sustainability will require.

I'm going into this conference, having just finished the conference of policy makers in Copenhagen on Information Technology and the Environment, with several "big" questions. One is what are the different meanings of "sustainability", if in fact anyone has a good definition? Exactly what are people expecting/wanting to be sustained? Of late, I've been disappointed that biodiversity doesn't seem to be something that is an explicit part of discussion -- it seems that as the environment worsens, values that we thought we wanted to be sustained fall away and we are "happy" with increasingly impoverished outcomes. This all fits the addiction model pretty well.

One point that I take from Loeb is that human decision makers ignore or deliberately remain ignorant of factors, sometimes, because they just don't know how to deal with them -- they'll even lie -- blatantly lie -- so that they can avoid the overwhelm caused by complexity. I'm hopeful that computing that enables us to deal with great complexity by processing millions of scenarios will also let us hear and consider all voices, be it the grizzly bears', the ranchers', the tourists', or in another setting, the homeless or the uninsured -- this is, perhaps, a humanistic possibility of computing.

Copenhagen OECD Conference 2009

This was originally posted on May 30, 2009 at 11:31 am on another blog platform as "Copenhagen 2009".

I just got back from Copenhagen, where I participated in the Organization for Economic Cooperation and Development (OECD) conference on Information and Communication Technology (ICT), Climate Change and the Environment. Last year I had been the only US government representative, and that had been a little scary, somewhat disappointing, and very cool – when they called on me for questions and comments they called on “United States” and before opening my mouth I reflected on what I was about to say :-). This year there were representatives from the State and Commerce Departments, and my (big) boss and I from NSF. I’m not revealing any secrets here – last year’s and this year’s are on the Web (http://www.oecd.org/sti/ieconomy/ictstheenvironmentandclimatechange.htm) -- more on this later and elsewhere.

I took the redeye from Dulles to Copenhagen late Monday afternoon, arriving Tuesday morning Copenhagen time. I was on Scandinavian Airlines (SAS) and I like it – every seat its own little screen with on demand video and music, a salmon dinner with not-quite-spaetzle pasta and lima beans (!?) to die for, power outlets for laptops, and stuff I’m forgetting. I watched two movies, "The Curious Case of Benjamin Button" (on the way out) and "The Reader" on the way back, both of which I'd balked at when they were in theaters, but I liked them each, particularly Benjamin Button, which was especially moving.

I took the train from Copenhagen airport to downtown, stowed my big backpack in a locker, only keeping my government laptop around my shoulder for safekeeping. Last year I happened upon the National Gallery of Art just before having to return, and wanted to see it especially this trip, but in no rush, and in fact exhausted but with really no option but to stay awake for many more hours, so I walked. 

<snip>

After a full morning, I took the train down to Helsingor, a beach town where the conference was held, did some brief walking around, talked to the tourist center, then a public bus to my hotel, a spartan place a couple of miles inland. I really wanted to sleep, but there was a conference reception that I thought I should attend for professional reasons, I wanted to stay up as long as I could for a better adjustment to the time zone, and frankly I felt isolated and knew connecting would help, and it did help.

The conference, in a nutshell, was focused on using information and communication technology (ICT) to reduce greenhouse gas (GHG) emissions due to humankind, and most notably what governments should do about promoting and supporting these efforts. There is great optimism that much can be done through virtualization (e.g., telework and conferencing with Web-based computer tools instead of the barbaric practice of flying cross country – yes, I recognize the irony :-), smart embedded systems (e.g., smart cars, smart buildings), which save energy through a large variety of computer-controlled mechanisms, and intelligent decision-making and planning systems that combine climate, economic and other social models. There is a lot of pushback though in implementing so much of this – for example, there are innumerable organizations that require that you keep your computer running 24/7 for reasons of “pushing” software and security upgrades, but my gosh, fixes to this should be 20 years old by now and the stupidity of designing such an energy-inefficient system is stunning -- it offends the engineer in me. I could go on and on with other forms of pushback and lack of awareness, and frankly, among other reactions, our failure to solve even simple, gross technological inefficiencies like the 24/7 computer-on policy has given me a certain sympathy for the disfunction that seems apparent in those trying to solve the planet's truly hard problems. I expect to be writing more on the topic of the conference. Generally, there is a lot that computer scientists and engineers can offer, both in terms of the products that they produce and in the ways that they think – my (big) boss Jeannette Wing has termed the latter “computational thinking”. 

It would have been fun having Pat with me to share the simple excitements and anticipations of a plane ride, train ride, castles, gardens, museums, and I hope that for next year as we are a good travel team, but next year I might be participating over the Web! So many decisions coming up, and even the “easy” ones can give me angst.

NSF's Broader Impacts Criteria

I attended a panel of colleagues, including a former member of the National Science Board (NSB) and three recent NSF review panelists, who discussed the revised broader impacts directives from NSF. It was an excellent panel, with great ideas, important insights, and pointers to helpful campus resources. In the latter half of this post, I offer some qualifications to some of the more nuanced comments, stemming from my experience at NSF (http://www.cccblog.org/2011/08/24/first-person-life-as-a-nsf-program-director/), but on the whole I was really struck by how much I resonated with what was said. I'll start by elaborating on these broad points of agreement, first talking about broader impacts as societal implications of the research, then broader impacts through formal and informal educational mechanisms.

 

Broader Impacts as Societal Implications

In my mind, the most important change to NSF's guidelines (http://www.nsf.gov/pubs/policydocs/pappguide/nsf13001/gpg_3.jsp#IIIA2 ) is that broader impacts (BIs) are to be evaluated by the same factors as intellectual merit (IM):

1. What is the potential for the proposed activity to:
    a. Advance knowledge and understanding within its own field or across different fields (Intellectual Merit); and
    b. Benefit society or advance desired societal outcomes (Broader Impacts)?
2. To what extent do the proposed activities suggest and explore creative, original, or potentially transformative concepts?
3. Is the plan for carrying out the proposed activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success?
4. How well qualified is the individual, team, or organization to conduct the proposed activities?
5. Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed activities?

There are additional requirements for proposal content to ensure that sound assessments along all 5 factors of BI can be made, and of course, some of these 5 factors are newly applied explicitly to broader impacts.

The importance of "institutionalizing broader impacts" was made early by the panel moderator. A university that has advancing broader impacts in its bones encourages everyone to leverage and grow institutional resources, and creates a collective intelligence that isn't myopic about societal implications of science and engineering, even if individual scientists are often myopic.

When I was at NSF, most proposals didn't elaborate much on broader impacts. I think most PIs take on faith that their research will have broader societal significance, and don't feel the need or ability to elaborate beyond a phrase to a paragraph. Often I sympathize. For example, the PI working on a new computer programming language might feel that the broader impacts of that work are coextensive with all that is touched by computer programming! I am guessing that mathematicians and theoretical physicists are of the same mind -- that the broader impacts are so pervasive and sufficiently distant that its almost impossible to reason about and express. But particularly in the computing and engineering disciplines, someone should be thinking about the societal implications, because they won't all be positive.

Here are several more thoughts.

1) When a program director (PD) and/or a panel sees a proposal that elaborates intelligently on broader impacts, it really makes a proposal stand out from the rest. Occasionally, I've heard comments like "I have never weighted broader impacts so highly" from a panelist. A PD hears that and it makes a difference in the PD's recommendations for funding. For example, research in novel variations on mathematical and computational optimization that will be applied to ecological problems (e.g., design of wildlife reserves) or health problems (e.g., kidney exchange arrangements) are examples that would stand out, could be verified, and would be vehicles for describing the science and its motivation to the public, to include Congress -- a big plus, and one that I believe in.

2) One thing that I have never seen is an NSF proposal that considers the possibility of negative societal impact (together, we hope, with societal benefits too) -- for example, that increasing energy efficiency of a class of devices will cause those devices to be used more, and therefore the collective energy footprint of those devices worldwide will increase. If I ever did see such a proposal, coupled with some plan to guard against it or just test for it, I'd really be impressed, and I think it might impress (some) panelists too. As a PD, and more recently as a panelist on interdisciplinary proposals (e.g., Science, Engineering, and Education for Sustainability), I've seen good reasons to bring in social and behavioral scientists to an otherwise technical proposal because there are implications, often negative, to how humans interact with a technology.

3) In the case of multidisciplinary proposals, what is BI to one field might be IM to another, and vice versa. For example, a computer scientist working with an ecologist might propose to create a new sensor (IM to computing) that would enable better environmental data collection and analysis (BI for computing, and IM for ecology), and therefore better management of resources (e.g., water) for communities (BI for both computing and ecological science -- 1st-order BI for ecology, 2nd-order BI for computing). I find that this observation about the discipline-specific nature of IM and BI is generally new to PIs, and helpful in their starting to think about the IM and BI of a multidisciplinary proposal.
     Interdisciplinary teams generally can mitigate myopia (e.g., in the example above, consider how the "2nd-order" BI for computing can be traced through work with ecology, and these higher order BI effects can be negative as well as positive). Good for NSF for encouraging such proposals through funding programs! I think universities can do a better job of mitigating scientific and engineering myopia through interdisciplinary teaming, and this is NOT usually part of what many universities mean by "institutionalizing broader impacts".

4) Different divisions and programs of NSF view BI differently. The foundational areas (e.g., computer programming languages, computer hardware) are (almost by definition) farther from the broader societal impacts of the research -- after all, they are at the "foundation"! The PDs in the CISE Division of Computing and Communications Foundations will tell the PDs in the CISE Division of Information and Intelligent Systems "You ARE our broader impacts!!!" In the foundational divisions generally, dissemination mechanisms (e.g., workshops, published papers, etc) and education initiatives may dominate the discussion on broader impacts. This came out in the BI panel. I think that these differences will continue, at least I hope that they do, though I also hope we find mechanisms that allow scientists and public alike to appreciate the implications (i.e., 1st-order, 2nd order, higher order still) of foundational research to societal impact. This generally happens through anecdotal stories (e.g., the creation of the Internet, fertilizer that enables feeding the world, …), which is good, but many at NSF would like better longitudinal tools for visualizing impact of NSF's investments, through citation tracking and technology transfer, for example.

In my experience, measuring societal impact generally is not the focus of attempts to institutionalize broader impacts through "evaluation shops" and the like, except at the Center level -- but it can be.

Education, Outreach, and Diversity

 

Good mechanisms for broadening impact is through formal and informal education, where I would call much of what we call informal education to be "outreach". With respect to education components of BI, whether elaborated or not, most proposals I saw at NSF didn't aspire to broader impacts that went beyond the funding period. These proposals essentially proposed to do something worthy, but local, both regionally and temporally. Again, when you see ambition to institutionalize educational innovations so that they persist beyond the funding period and beyond the PI's immediate network, it really stands out. Here is where much of the emphasis on "institutionalizing broader impacts" (Google it!) can be found (Vanderbilt, OSU, Missouri, Stanford, etc). At Vanderbilt, the Center for Science Outreach (VCSO: http://www.scienceoutreach.org/) is giving PIs mechanisms for broadening the impact of their science through formal and informal education mechanisms. I expect that the Vanderbilt Institute for Digital Learning (http://www.vanderbilt.edu/vidl/) will work with VCSO and other groups for non-STEM, to further institutionalize broader impacts, insuring that positive BIs persist and grow.

The former NSB member highlighted the importance of evaluating BIs, just as PIs are expected to evaluate IM (see factor 3 above). This is fantastic -- I can't remember seeing a scientific evaluation plan for BI activities in proposals, except for large Centers where NSF required that an "independent" evaluation team for the BI aspects to be appointed. While NSF has been pushing on BIs for a long time, making BIs "first class" along with IM, is overdue.

I came back from NSF believing in the importance of institutionalizing broader impacts; there should be dedicated funds for BI (see http://www.vanderbilt.edu/provost/cms/files/Broader-Impacts-2-0.pdf) and particularly for medium and large proposals, there should be a co-PI who is explicitly named as the BI lead (my opinion); and some funds set aside to support communicating science and technology to the public too, because I haven't seen this latter activity explicitly called out. Apropos this last point, I spent late nights rewriting a fair number of award abstracts so that there was some chance that the research and the motivations for the research would be understood at some meaningful level by a larger public, including congressional staffers. While there were some notable exceptions, most PIs seem to think that they could let the proposal project summary serve as the award abstract -- sheesh! That summary might be a good starting point, but iteration is necessary to make it publicly accessible.

When I returned from NSF I learned about Vanderbilt's  Communication of Science & Technology major (http://www.vanderbilt.edu/cst/major.html); Vanderbilt must be (close to) unique in the nation in having such a major (good for Vanderbilt!), and it can be the basis for institutionalizing these kinds of broader impacts. Also, there can and should be a better connection made between the communications' teams at universities and schools with NSF, other agencies, and foundations. When I was at NSF, I can't remember ever getting award highlights from the professional science news writers who I know are writing for universities and schools -- why not?! Rather, again I had to iterate with PIs to get research award highlights that were informative and accessible to the public. In most cases, getting such highlights from PIs was like pulling teeth -- ugh!  Some probably don't value highlights much, while others would like to contribute, but they are busy too.  These highlights will be read by congressional staffers, and they need to be good, rather than some annoyance.

Related to the education components of BI, are diversity concerns, ranging from diversity of the research team, particularly on Center-level proposals, to diversity in future generations of scientists and engineers. Again, on Center level proposals there will be special accommodations to ensure that diversity and change in diversity over time is evaluated. But as with (other) education components, there was often little ambition and creativity in attention to diversity. Its not that broadening participation isn't an intellectually interesting area of study (e.g., see http://www.nsf.gov/pubs/2012/nsf12037/nsf12037.jsp), its that few PIs are thinking about it in those terms and so you read silly things, almost disrespectful in my mind, like listing the race and gender of selected members of the research team as the sole attention to broadening participation. In some cases you get the impression that the PI has put about 10 minutes of creative thought into broadening participation, and broader impacts more generally. Again, what are the ambitions for initiatives that move beyond the PI's institution and that will persist and grow after the funding period ends? Institutionalizing broadening participation concerns is germane here too.

Behind the Scenes

There was talk about "why don't PDs do this or that", "NSF should do this". There were some things that were said on the BI panel that aren't wrong per se, but some important factors don't seem to be appreciated. 

One of the most important things I learned at NSF was that there is substantive noise, and different sources of noise, in the process of vetting proposals; I don't mean that the noise is debilitating and that it compromises the validity of peer review as implemented at NSF, but its easy I think to "overfit" your experience on a panel and think you can prescribe simple fixes. Here are some observations.

1) BIs are historically weighted less than IM. In my experience, panels will judge a proposal worthy of funding or not based on IM, and break ties based on BI. The new guidelines won't guarantee equal weighting of IM and BI (see 4c of http://www.vanderbilt.edu/provost/cms/files/Broader-Impacts-2-0.pdf), and I don't think that they should, but I think that the new guidelines will insure that BI is more than a tiebreaker. In some cases, BI might be more heavily weighted than IM, and in a diversified NSF grant portfolio, I think that is perfectly fine. But again, recognize that IM and BI are discipline specific. As an aside, good grantsmanship would suggest that if you are getting declined for an education-heavy proposal in CISE (or MPS or ENG ...) then recast it and submit it to EHR!    

2) Review Panels are usually great at telling a PD which proposals are worthy of funding and which are not worthy of funding. This is already a big win for a PD who has to make decisions on what to recommend. In my experience problems arise when a PD PUSHES a review panel to do what it is not equipped to do. I do not think, for example, that a review panel is in a position to make hard recommendations (e.g., highly competitive versus competitive) based on projected funding levels. That's because the panel does NOT have all the facts in front of it to make such fine-grained recommendations.
         Funding levels are often much less than the percentage of proposals worthy of funding. This can lead a panel to "overfit" the proposals, with great angst over those last few proposals that are being placed in highly competitive versus competitive, and competitive versus not recommended. It's not that overfit will lead to "wrong" decisions or even "wronger" decisions (because most experts will focus on one valid set of characteristics over another valid set), but it can lead to great angst, and it can lead to odd factors for making the final hard calls (like who needs to get to the airport, an advocate or a detractor, of the proposal in question?).

3) One BI panelist said that on an interdisciplinary NSF panel that he/she had served on, 3/4 of the proposals were quickly decided because of IM weaknesses, and the remaining IM-strong proposals were placed in final categories based on BI factors. That sounds consistent with my experience, seems perfectly fine to me, but may seem less than ideal (aka overfitting) to some NSF panelists. Some additional points:

• (i) The new NSF guidelines may make proposal assessment more holistic (IM AND BI) throughout the paneling process, rather than IM assessment followed by BI assessment. Such a change may lengthen panel time.

• (ii) The weighting of BI is INCREASED in interdisciplinary settings. What would we otherwise expect an interdisciplinary panel to do??? Paneling interdisciplinary PRE-proposals relies even more heavily on BI factors. Its interesting to me that scientists agree with Congress on the importance of BI, when its not a proposal in their area.

• (iii) I once suggested to a PI who was not getting a proposal funded through the core program to recast it and submit to an inter-disciplinary, cross-directorate program, specifically to take advantage of the BI bias on interdisciplinary panels. Some might view this as exploiting noise (yes!) and some might say its one mechanism for getting out-of-the box research funded (yes!). The PI's proposal was recommended and funded under the interdisciplinary program; it had also been well regarded by previous core area panels, Competitive or Not recommended for Funding (yes, it can still be a good proposal in this latter case).

4) Not Recommended for Funding is not the same as not worthy of funding or not ready for funding. Again, we invite a panel to increasingly overfit the more we ask them to make finer-grained distinctions. Making finer-grained distinctions is more likely to tweak personal, professional, and scientific biases and constraints. I mean, why should charisma be a factor in making scientific recommendations? More importantly, why is NSF shooting itself in the foot by misrepresenting to the public and to Congress that some large percentage of proposals are NOT "recommended" because many will view this as NOT worthy, but this is NOT the case. At least some proposals that are not recommended for funding are, in the opinion of the panel, worthy of funding! Thereby we misrepresent the under-funding of science -- "but the expert panel said this stuff wasn't worth funding, so why increase funding!"
 
5) It's often the case that there is no consensus on the final, close call recommendations by a panel. This difference of opinion can and should be represented in a Panel Summary. If one or more panelists believe that a proposal should be rated more highly (and in any case), make sure that opinion and the reasons for it are expressed in the Panel Summary and that the PD has heard the argument during discussion (because of what I will say in the next bullet point about PD discretion). In fact a recommendation (HC, C, NRF) by the panel is NOT required (what's the PD going to do ? -- "make you" do something??!! -- no chance, only in your head). In one or two situations I had a panel split down the middle, and no one would budge on an HC vs C (for example), so they described the deadlock, and left the recommendation box unchecked. I had heard what I needed to hear to make a recommendation.

6) In my experience, NSF PDs are relatively quiet during review panels -- and I think that's a good thing. An NSF PD is not a DARPA PD, thank goodness, nor vice versa, thank goodness. NSF PDs have visions for their fields, but their actions are highly modulated by the research community, at least within their core discipline areas (PDs often branch out more when creating and implementing interdisciplinary initiatives that will influence their fields).
     A PD needs information for making recommendations, and while the panel recommendations are the single most important factor in a PD's recommendation, they are far from the only factor. Portfolio balance (where balance does not imply equal cardinality), institutional balance (ditto), PI balances (ditto), balances within the larger programmatic unit (e.g., robotics versus natural language processing versus …), …, AND WHAT THE PD HEARD DURING THE PANEL DISCUSSION. A good PD is a good listener. A good PD will likely speak up from time to time, but not too much. When I have seen what I regard as a PD stepping over the line and being too prescriptive, its been a rotator.
     In some sense it doesn't matter too much if a review panel "overfits" in its recommendations, because while a PD is very influenced by a panel, the PD is NOT tied to it. In fact, arguably the PD is there to compensate for panel overfitting, scientific conservatism, and bias. Its no small thing to decline a Highly Competitive proposal because you think a Competitive proposal should be funded instead (and there are not the funds to do both), and all this needs to be justified IN WRITING, so there is nothing flippant about all this. On rare occasions a Not-Recommended-for-Funding proposal may be funded (because that's not the same as Not Worthy) but that takes considerable justification.
    Thus, you might see a PD remain silent during the panel itself, because the panel is there for the PD to collect information, not about making final decisions. Should a reader advocate, in contrast, that a PD take a "leadership role" on the panel, for example on the importance of BI, recognize that that is a slippery slope. When I opened my mouth, it was most often to ask or answer a question, but yes, I would have to insure that the panel addressed BI to my satisfaction, that they wrote a respectful and informative panel summary, etc.
    That said, I think its a wonderful thing to set aside a session before the panel begins to talk to a panel about issues of intrinsic bias, broader impacts, etc, but once the panel starts, don't start (trying to) direct them TOO MUCH, else you won't know where they will go on their own, informed by the factors that they are in a position to assess, and thus a PD will confound her or his decision making process with the panel's decision making process.
    Would I advocate that we don't push panels to make the fine grained distinctions among those last close calls on the borders of categories (e.g., HC, C, NRF)? Sometimes perhaps, but suffice it to say that having a panel make the fine grained distinctions gets them to talk through the issues thoroughly, and its one mechanism for getting the issues on the table and heard by the PD, even if a PD might come down differently on the close calls than the panel does.
     But alas, there is another reason that PDs and their superordinates may push panels to make those final hard calls! Those final placements into HC, C, NRF are heavy lifting, and if the panel doesn't do it, the PD must. Its not that I think that the PD will do a better or worse job in those final placements (but might use different tie breakers than a panel) -- its that the PD often just doesn't have time. Exercising discretion, when you are (thankfully) obligated to justify it, takes a lot of time, which a PD often doesn't have.

Time, Time, and Time

Lots more I could say here, but let the following general points anticipate suggestions that "NSF" (as if NSF was monolithic) do this or that. Most NSF staff are working very long hours, and this includes a lot of in-the-trenches work. In the CISE (Computing) Directorate, I would sometimes think that if work weeks of more than 50 hours were made illegal, with stiff penalties for violators, there would be a year of extraordinary angst and pain for NSF and academia, followed by consistency, and organizational and programmatic sanity. Its only because of extraordinary hard work by many NSF staff that the whole system doesn't fall apart, but institutional performance is degrading, albeit gracefully. Any increases in funding to NSF generally go to new scientific funding programs, each of which increases overhead, and not towards increases in staffing. After getting back to Vanderbilt, I recall the excitement caused by the Robotics Initiative!!! And it was exciting. But you can bet that the overhead associated with it came out of the hides of NSF staff.

I've heard that NSF talks out of both sides of its mouth on broader impacts or on other issues, or that it drops the ball on this or that. Consistency requires training and that requires time. Going against a panel recommendation (supporting a Competitive proposal because of BI over a Highly competitive proposal) requires justification in writing, which requires time. Reading and pushing PIs for BI updates as well as IM updates requires time. Getting the "best" panelists to peer review proposals requires time, because in CISE at least, PDs will often get an acceptance rate from panel invitations of 20%-30%; I had high rates -- about 60-70% as I recall, because I allowed panelists to "phone in" (http://science-and-government.blogspot.com/2011/08/virtual-panelists-and-thoughts-on.html), but still, designing and recruiting and running a balanced panel takes time. And of course big thinking takes time, be it on designing funding programs along societal dimensions such as sustainability, health, and education; or tech/science dimensions such as robotics, computational game theory, etc.

A major constraint on NSF, or I should say staff within NSF, in responding to suggestions for "this or that" is time, time, and time. In addition to writing NSF, write Congress about funding of Science, and funding of the staff who create, implement, and run the  programs.

Rotating Program Directors at NSF

I recently commented on a blog post by Jeffrey Mervis on the AAAS Science blog at http://news.sciencemag.org/policy/2013/10/special-report-can-nsf-put-right-spin-rotators-part-1 , which acknowledged the pros of using faculty members from academic institutions as "temporary" or "rotating" program directors at the National Science Foundation (NSF), side by side with permanent Federal staff, but Mr. Mervis' article also points out that monetary savings might be achieved over the present implementation of NSF's rotator program.

I served at NSF as a rotating program director in the Computer & Information Science & Engineering (CISE) Directorate from 2007-2010 and have thoughts on the NSF rotator program. I repeat my comments to Mr. Mervis's article here, but I emphasized in these comments what savings might be most productive and doable; in addition, I think that some of the other recommendations of monetary savings in the Inspector General (IG) report cited in Mr. Mervis' original post seem less achievable or even less desirable -- maybe I will elaborate another day. I also argue that NSF should broaden its perspective on the possible benefits of rotating program directors.

-----

Your post (part I) and the IG’s report paint an accurate, though brief, picture of the IPA program: IPAs (and other staff) work hard and very competently, benefiting science and engineering research and education in the United States, but cost savings are possible. Of the suggested savings, reducing IPA travel back and forth between home institution and NSF would probably be (the most) productive. Frequent (e.g., weekly) travel by an IPA is costly, and it can also disrupt operations in NSF’s team-oriented environment. For IPAs who commit to a life predominantly in the DC area, I hope that NSF continues to pay for their relocation. However, for those who would prefer life predominantly at their home institution, let them telework, probably after an onsite orientation period that is designed to protect NSF esprit de corps. In either case, limit travel back and forth to some sensible number of trips, because 50 IRD trips a year is ridiculous, even if 50 days of IRD is not. This might also put NSF in a better position to negotiate for partial IPA compensation by the institutions of those rotators who stay at home (because the idea that NSF should expect home institutions to partially compensate IPAs who are working extraordinary hours for the government, and that's particularly true of anyone onsite at NSF, seems misplaced). Importantly, these arrangements are easier said than done, at least while preserving the benefits of the IPA program.

While I limited trips to my home institution of Vanderbilt University, I nonetheless ran two “virtual” review panels from my Vanderbilt office, supporting the IG’s contention (and many in NSF’s operational divisions too!) that much can be done through remote communication technology. And now we are getting into a largely underutilized advantage of the IPA program – that IPAs can benefit NSF operations as well as the scientific mission. IPAs are smart, usually very dedicated people who are watching and innovating the operations of NSF. For example, fully 3/4 of the review panelists that I recruited were virtual panelists – they participated by phone or video conferencing, and saved NSF substantial travel costs. My supervisors in the organization, including two IPAs, supported this activity. Other IPAs innovated in similar ways, as well as some members of the permanent staff. If NSF made a commitment to supporting IPAs who had a desire to telework from their home institutions, with protections in place to protect high-quality communication, responsiveness, and NSF esprit de corps, it would go a long way to building a culture in which much larger monetary savings could be realized through the use of virtual panelists (http://www.sciencemag.org/content/331/6013/27.full ), as well as reaping other substantial advantages of virtual panelists (http://science-and-government.blogspot.com/2011/08/virtual-panelists-and-thoughts-on.html )

Apropos the possibility of operational benefits of IPAs, exit interviews of IPAs seemed spotty and certainly not universal when I was there. It strikes me as a terrific lost opportunity if NSF is bringing in talented faculty members, almost all of who have the luxury of speaking their mind because of job security that stems from tenure, and not exit interviewing them and then acting on those interviews!

The IG report also suggests the desirability of a person or office dedicated to evaluating the IPA program on a continuing basis – that is a terrific idea. I have no doubt that ongoing evaluation would affirm the scientific advantages of the IPA program and improve IPA management. In particular, John Conway’s article alludes to the “cultural” differences that often exist between academia and the team-oriented environment of NSF. An IPA-oversight officer who respected and appreciated the IPA mission would presumably help define best practices of IPA orientation, training, and management to effect the transition to the NSF environment, as well as evaluate the program.

Finally, part 2 (http://news.sciencemag.org/people-events/2013/10/special-report-can-nsf-put-right-spin-rotators-part-2 ) of your article highlights a case where an IPA may have been powerless and dismissed summarily. I do not know this case, but five comments seem relevant and responsible: (1) I was proud of NSF’s policies and practices regarding conflicts of interest (COI), and I wish they were standards practiced throughout our Federal government; (2) my experience was that the professional ethics officials at NSF were honorable, highly competent, and responsive to requests for clarification and other help on COI issues; (3) the COI standards are high (thus my pride), but I would regard a case like that outlined as forgivable and correctable in a gentler and more constructive fashion than that described -- I can imagine circumstances in which I might have missed real or perceived COIs too; (4) if there were an officer responsible for assessing the IPA program at NSF, then presumably they would have looked carefully at the actions of all IPAs involved, including supervisors, and made corrective recommendations on IPA training and management at all levels; and (5) the individuals within NSF best placed to speak out on any injustice might well be IPAs, again because of the job security that stems from tenure at their home institutions. That’s not to say that rotators should be watchdogs, but more thought should go into how to use IPAs effectively to inform operations and management, as well as science.

"Inside Job", COIs, and Academic Economics

<originally posted August 2011 on earlier blog>

I've been watching the "50 Documentaries you must see before you die" or some such title. "Inside Job" (http://www.insidejob.com/) was one one them -- as the title suggests, its a story of the economic collapse and it includes a very disturbing portrayal of economics in academia (and elsewhere). It's portrayal of economics motivated me to write the Association of American Universities (AAU: http://www.aau.edu/). That organization's "Scientific Enquirer" is a response to the drubbing that science has taken in some quarters and I think its an informative publication, though very new (http://www.aau.edu/research/Science_Enq.aspx?id=12370).

***

Dear AAU,

I recently watched the Academy Award winning (2010) documentary “Inside Job.” In the eleven minutes starting at 1:22:30 in that film, academic economics was portrayed very negatively. The film’s claim that (a) some in academic economics are seriously conflicted with the financial industry and that (b) this conflict contributed to current, severe worldwide economic troubles through long-term academic advocacy of deregulation, is the most disturbing portrayal of academia that I recall seeing. While the film highlights only a few institutions and a few faculty members and university administrators, there is a suggestion that the problem of conflict of interest is systemic in academic economics.

Your Press Release of 02/28/2008 on Conflicts of Interests states:

“According to the report, institutional conflicts of interest are becoming a growing concern as academic institutions assume more complex roles and expand their relationships with industry. Conflicts of interest policies are critical to assure that these essential interactions remain principled and are conducted within a rigorous, transparent, and credible framework.” (p. 2, first link of http://www.aau.edu/policy/COI_policies.aspx?id=10096)

While your recent documentation appears to focus on health care, if there is any truth to the film’s claims, in whole or part, the statement would apply to (segments of) academic economics, arguably with even broader societal significance than healthcare.

I hope that AAU is participating in, if not leading, development of and/or increasing awareness of conflicts policy and training in economics, if in fact you find this to be appropriate. Additionally, if academic economics was significantly misrepresented in the film, even while it showed some academic economists in a positive light as commentators, then this misrepresentation (again, in whole or part) could be appropriate subject matter for commentary in the Scientific Enquirer.

Coburn Report on NSF

(Copied from a July post on my "home" blog)

Last week I responded to my Congressional delegation on Senator Tom Coburn’s report entitled ““The National Science Foundation: Under the Microscope” (http://coburn.senate.gov/public//index.cfm?a=Files.Serve&File_id=2dccf06d-65fe-4087-b58d-b43ff68987fa). I was aware that some scientists whose projects were represented in the report had already responded through blogs and other public forums. Having worked at NSF, however, there was an aspect of the report to which I could respond specifically, and I decided to stick pretty closely to observations that this unique perspective offered. My letter to Senator Lamar Alexander (R-TN) is below; the same letter was also sent to Senator Bob Corker (R-TN) and to Congressman Jim Cooper (D-TN) representing Tennessee’s 5th District.

*******

July 21, 2011
Senator Lamar Alexander
3322 West End Avenue, #120
Nashville, TN 37203

Dear Senator Alexander:

I recently read the entirety of Senator Coburn’s report “The National Science Foundation: Under the Microscope.” You are one of my senators and I am thus writing to you to express significant concerns with the report, focusing on those for which I have something of a unique perspective. From July 2007 through August 2010 I was on leave from Vanderbilt University, serving as a Program Director at the National Science Foundation (NSF) in the Directorate of Computer & Information Science & Engineering (CISE). It was a pleasure and honor to serve at NSF as a member of a hardworking, dedicated team. I regard my time at NSF as service to my country of which I am very proud.

Foremost, I worry that there seems to be an attitude of disrespect for NSF staff permeating the report. For example, the report includes a section “NSF Flying High with First-Class Junkets” (p. 14), with no indication in the section that anyone did this. Indeed, I never flew first class at NSF, or on a junket, nor do I know of colleagues who did. In a second example, when commenting on NSF’s desire to become environmentally friendlier, the report says “Some might find it interesting to note, then, that the NSF currently owns 375 vehicles, including 52 sports utility vehicles” (p. 15). This statement seems flippant and is vague, but is suggestive that NSF staff are hypocritical, not environmentally conscious and/or that the vehicles are not used for science. A third example is the report’s claim that porn surfing was “pervasive” (p. 15), with at least six citations of the same article in The Washington Times. This statement is very wrong. I can believe that such cases, though anomalous, have consumed a large part of the Inspector General’s time in the recent past, since a tiny proportion of a group is often responsible for a large proportion of the angst. We have seen a very recent example in Congress of misbehavior that consumed large amounts of time and energy, but I would not claim that such behavior was rampant in Congress.

My experience at NSF contrasts with the report’s representation of NSF staff activities. My colleagues and I performed many diverse tasks, including the vetting of research proposals with input from other scientific experts, preparing research solicitations, preparing and giving outreach talks to the public, and representing NSF and our country overseas. It is because of administrative and scientific staff dedication that the agency functions as well as it does in spite of very heavy workloads. I worked 60+ hour weeks and this really was pervasive across the Foundation. It appears that even when the President and Congress agree on budget increases for NSF, it is for scientific initiatives that come with yet more overhead and not for the addition of staff to deal with that overhead. I very much encourage you to consider additional funding for staff as well as scientific initiatives.

I believe that the report’s tone will cause many to reject the report entirely, to include points that I think have validity. For example, I generally agree with the report’s statements that a discussion about funding priorities is important. In fact, the very hard discussions I had with my fellow program directors at NSF were on what projects to fund given differing opinions on priorities and our limited budget. Our budget only allowed us to fund about one-third of the projects that had been judged by scientific experts to be most worthy of funding, which in turn was about one-third of all projects submitted; these proportions varied across the agency. The heartbreaking part of my job at NSF was that the majority of projects worthy of funding could not be funded, and my colleagues and I knew the costs for science research in our country and the costs to faculty members and students behind that research. NSF personnel take the job of assessing the scientific qualifications of projects and funding priorities very seriously.

I also agree in general with the report on the importance of metrics and tools to evaluate the payoffs of scientific investments; I believe that NSF staff would welcome such tools with open arms. Data analysis and visualization tools are critically needed to track scientific investments and to evaluate the US funding portfolio within and across agencies. The report highlighted the importance of the STAR Metric initiative, which is aligned with the Science of Science and Innovation Policy (SciSIP) program housed in the Social, Behavioral and Economics (SBE) Directorate of NSF. Thus, I was surprised that the report also recommended eliminating the SBE Directorate. I can’t emphasize enough how wrong I think this would be. I am a computer scientist and daily I witness how computing technology is transforming the ways that humans interact, perceive, decide and learn. The last thing we need is to cut research on understanding human behavior in a time of transformative technology. We should understand, for example, what video games and social networks are doing to our children and all citizens, so that we can design technology to enhance learning and decision-making, not diminish it.

Finally, I fear the report’s tone because it comes across as demeaning public servants whom I know to be dedicated, talented and industrious. I am not sure where degrading stereotypes of ‘government bureaucrats’ originated, but I for one, a lifelong academic, gained an incredible respect for the hard work and brains of federal staff, not just at NSF, but in agencies and departments across government.

Government colleagues can and should offer constructive criticism to one another, but I believe Senator Coburn’s report doesn’t paint an accurate picture of those at NSF who loyally and diligently serve their country. In particular, I wanted to convey my experience to you that NSF is an institution that Americans can be proud of, respected and emulated the world over, with staff who do their best in making difficult decisions on matters of national and scientific importance, despite a limited budget and a heavy workload.

Thank you for your attention and for your service to our country.

Respectfully,
Douglas H. Fisher

**** End letter ****

Generally, what significant empathy I have for the challenges currently faced by Congress and the President is due to my NSF service. It also makes me sick that so many members of our government appear to be so disrespectful of each other.

Despite my fears about the Coburn report, I think that some good things could come of it and the responses that are following. Notably, I hope that scientists, after reading the report, see the vital importance of communicating science to the public, to include Congress and scientists in disciplines other than their own. I’d like to see every research team be associated with those skilled in communicating scientific findings and their national and international relevance to the public. Universities have individuals skilled in communicating science to the public, but they are probably too few and far between – communication can be integral to research projects, and researchers can ask for the funds necessary to support that activity. Rather than blog posts erupting after reports such as this to explain and justify scientific research, maybe we’ll see more proactive outreach.

Generally, my experience suggests that scientists and engineers can more embrace their role as citizens, and all citizens should recognize that science and engineering are integral to citizenship. I could be wrong, because I might be working from a biased sample, but my sense is that not a lot of scientists write their elected officials, advocating more funding for science research and the like. I hope that scientific professional organizations not only respond themselves, which is happening, but that they encourage civic engagement by their individual members. Its probable that the climate scientists have internalized this message over the last few decades, and the social scientists may be getting it as well.

In any case, while my 3+ years at NSF reenergized the teacher in me and the researcher in me, it may have reenergized the citizen in me most of all. I hope so.