GAO and US Advisory Commission on Public Diplomacy discuss evaluation tools

~ Matt Armstrong

The subject of public diplomacy evaluation tools and methodologies has been front and center this week. Debating the difference between “measures of effectiveness” (or MOE), “measures of performance” (or MOP), and throwing spaghetti at a wall can seem like arcane stuff, understanding the value of engagement, and the ability to communicate that value, is extremely important. Measures are fundamental to discussions on what to do and why.
Of course in order to measure, one must not only know the audience (primary, secondary, tertiary as they must be categorized… or do they?), where they are (as they are less likely to be within neat geographic coordinates), and how they communicate, but also the effect, intentional and unintentional, of the activities of allies, adversaries, and neutrals on the audience. The world cannot be put into a laboratory.

As none of us, including (and sometimes especially) our adversaries, are not endowed with unlimited resources, we must pick and chose where and how to engage for the greatest effect, if we even know what the effect is or when that effect is to be achieved. Part of the communication of effectiveness is creating the appreciation that objective, black and white, measurements may not be available (are they ever?) and that subjective measurements must take place. Failing to understand the totality of engagement and perceptions leads not only to misguided questions like “Why do they hate us?” and “can we deliver a public diplomacy message in 140 characters?”

The two discussions on public diplomacy evaluation this week were the public meeting of the US Advisory Commission on Public Diplomacy and a report by the GAO. The topic of the Advisory Commission’s meeting was pertinent to their soon to be completed report on the subject. Morgan Roach of The Heritage Foundation and Tijana Milosevic from the Alliance for International Educational and Cultural Exchange both summarized the discussion.

Morgan, indirectly referring to the Commission’s role as an oversight body, described the meeting as a discussion on “State Department’s capacity to evaluate whether it was having an effective impact on international audiences.”

Tijana provides more details. For example, Cherreka Montgomery, Director of the Evaluation and Measurement Unit, described the reduction in MOP from 898 in 2005 to 21 today. The Cherreka noted the current evaluation project, the “Public Diplomacy Impact Project” (or PDI) combines both qualitative and quantitative approaches.

It is worth noting that while Under Secretary Judith McHale did publish the “strategic framework” for public diplomacy, and regardless of the timing of the report, views of the report, and blanks yet to be filled in or communicated, it is really the State Department’s framework. The GAO gets it right. The difference is the enterprise (or bureaucracy) signed off and buy into the document.

The GAO report, Assessment of Public Diplomacy Platforms Could Help Improve State Department Plans to Expand Engagement (PDF, 5mb), was a broader review of State’s engagements, both virtual and physical, with audiences beyond our borders (don’t say “foreigners” as the engagement is based on geography not nationality; there are 15 million immigrants and visitors in the US, +/- 5 million) in response to a request from Representative Howard Berman (D-CA), Chairman of the House Foreign Affairs Committee.

Guiding the GAO analysis were the following questions:

  1. Describe the outreach platforms State uses overseas;
  2. Examine the challenges and opportunities related to these platforms;
  3. Review State’s plans for these platforms; and
  4. Assess the extent to which State has evaluated these platforms.

The GAO thus reviewed far more aspects of State’s engagement than what is controlled, or even guided by, the Office of the Under Secretary for Public Diplomacy and Public Affairs.

The bottom line assessment of the GAO, as stated up front in their report, is efforts to expand and improve outreach “platforms”, which may be virtual or physical, must be preceded or accompanied by an “assessment of the relative effectiveness” of each platform. There was no waiving in State’s reception of the GAO report: it endorsed the GAO’s findings and conclusions as well as concurred with the recommendations, going so far as to ask the GAO to consult and provide input on the assessment efforts.

The report provides a useful overview of department activities, including American Presence Posts, American Centers, Binational Centers, American Corners, and Virtual Presence Posts. Many of these were covered last year by comprehensive report “US Public Diplomacy – Time to Get Back in the Game” by Senator Richard Lugar’s (R-IN) staff.

The GAO report noted the State Department’s online activities to engage “foreign” audiences:

  • 230 Facebook accounts (including about 80 embassy and consulate accounts),
  • 80 Twitter feeds (including over 50 by overseas posts),
  • 55 YouTube channels,
  • 40 Flickr sites, and
  • 25 active blogs.

It is unlikely they counted State.gov which receives about 30% of its visitors from outside the US, but they probably counted America.gov’s main Twitter account (@AmericaGov), which had the opposite audience make-up: 30% of its subscribers came from outside the US (according to research conducted by MountainRunner in July 2009).

In the end, GAO’s summary gets it right on an issue that vexes the US Government in general. Substitute “State” for “Defense” and you’ll still have problems (or absences) of baseline analysis, comparative and comprehensive reviews.

State lacks comprehensive information on the relative effectiveness of its platforms, such as how each platform has expanded U.S. engagement with foreign audiences. Without such information, it is difficult for policy makers to make an accurate assessment of the relative benefits of each type of outreach platform and effectively allocate scarce resources.

For what it’s worth, it would be nice if somebody in Congress asked the GAO to examine the engagement (call it “public diplomacy” if you want) efforts of different countries. The last such analysis at such a level, which was done by the GAO (then the Government Accounting Office) was in 1979: The Public Diplomacy of Other Counties: Implications for the United States. This time around, we should explore the “public diplomacy” (or communication and engagement) of non-state actors as well as countries.

See also:

Published by Matt Armstrong