Transparency is Bunk
Adapted from an impromptu rant I gave to some people interested in funding government transparency projects.
I’ve spent the past year and change working on a site, watchdog.net, that publishes government information online. In doing that, I’ve learned a lot: I’ve looked at everything from pollution records to voter registration databases and I’ve figured out a number of bureacratic tricks to get information out of the government. But I’ve also become increasingly skeptical of the transparency project in general, at least as it’s carried out in the US.
The way a typical US transparency project works is pretty simple. You find a government database, work hard to get or parse a copy, and then put it online with some nice visualizations.
The problem is that reality doesn’t live in the databases. Instead, the databases that are made available, even if grudgingly, form a kind of official cover story, a veil of lies over the real workings of government. If you visit a site like GovTrack, which publishes information on what Congresspeople are up to, you find that all of Congress’s votes are on inane items like declaring holidays and naming post offices. The real action is buried in obscure subchapters of innocuous-sounding bills and voted on under emergency provisions that let everything happen without public disclosure.
So government transparency sites end up having three possible effects. The vast majority of them simply promote these official cover stories, misleading the public about what’s really going on. The unusually cutting ones simply make plain the mindnumbing universality of waste and corruption, and thus promote apathy. And on very rare occasions you have a “success”: an extreme case is located through your work, brought to justice, and then everyone goes home thinking the problem has been solved, as the real corruption continues on as before.
In short, the generous impulses behind transparency sites end up doing more harm than good.
But this is nothing new. The whole history of the “good government” movement in the US is of “reformers” who, intentionally or otherwise, weakened the cause of democracy. They too were primarily supported by large foundations, mostly Ford and Rockefeller. They replaced democratically-elected mayors with professional city managers, which required a supermajority to overrule. They insisted on nonpartisan elections, making it difficult to organize people into political blocs. Arguing it would reduce corruption, they insisted city politicians serve without paying, ensuring the jobs were only open to the wealthy.
I worry that transparency groups may be making the same “mistake”.
These are some dark thoughts, so I want to add a helpful alternative: journalism. Investigative journalism lives up to the promise that transparency sites make. Let me give three examples: Silverstein, Taibbi, Caro.
Ken Silverstein regularly writes brilliant pieces about the influence of money in politics. And he uses these sorts of databases to do so. But the databases are always a small part of a larger picture, supplemented with interviews, documents, and even undercover investigation — he recently did a piece where he posted as a representative of the government of Turkmenistan and described how he was wined and dined by lobbyists eager to build support for that noxious regime. The story, and much more, is told in his book Turkmeniscam. (His book Washington Babylon is similarly indispensible.)
Matt Taibbi, in his book The Great Derangement, describes how Congress really works. He goes to the capitol and lays out the whole scene: the Congressmen naming post offices on the House floor, the journalists typing in the press releases they’re handed, the key actions going on behind the scenes and out of the public eye, the continual use of emergency procedures to evade disclosure laws.
And Robert Caro, in his incredible book The Power Broker (one of the very best books ever published, I’m convinced) takes on this fundamental political question of “Who’s actually responsible for what my government is doing?” For forty years, everyone in New York thought they knew the answer: power was held by the city council, the mayor, the state legislature, and the governor. After all, they run the government, right?
And for forty years, they were all wrong. Power was held — held, for the most part, absolutely, without any checks or outside influence — by one man: Parks Commissioner Robert Moses. All that time, everyone (especially the press) treated Robert Moses as merely the Parks Commissioner, a mere public servant serving his elected officials. In reality, he pulled the strings of all those elected officials.
These journalists tackled all the major questions supposedly addressed by US transparency sites — who’s buying influence? what is Congress doing? who’s in power in my neighborhood? — and not only tell a richer, more informative story, but come to strikingly different answers to the questions. In this era where investigative reporting budgets have been cut to the bone and newspapers are folding left and right, it’s fallen to nonprofits like ProPublica and the Center for Independent Media and, from a previous era, the Center for Public Integrity, to pick up the slack. They’ve been using the Internet in innovative ways to supplement good old-fashioned narrative journalism, where transparency sites are a supplement, rather than an end-in-themselves.
For too long we’ve been funding transparency projects on the model of if-we-build-it-they-will-come: that we don’t know what transparency will be useful for, but once it’s done it will lead to all sorts of exciting possibilities. Well, we’ve built it. And they haven’t come. The only success story its proponents can point to is that transparency projects have bred even more transparency projects. I’m done working on watchdog.net; I’m done hurting America. It’s time to give old-fashioned narrative journalism a try.
Previously: Disinfecting the Sunlight Foundation [November 2006]
You should follow me on twitter here.
April 23, 2009
Now, that’s a timely rant.
Everybody and their brother seem to be freeing data nowadays, but - for some odd reason - the actual uses (and perhaps even professional evaluation) are then put too far aside.
How does your rant add up to what Watchdog is up to currently? Are you looking for journalists, or…
posted by Tommi
on April 23, 2009 #
I’ve always thought that getting the raw data and turning it into a format that’s useful was really just the first step. Open government data is a tool to use to solve the problem, but not the solution itself.
Once investigative journalists and researchers really figure out how to use those tools effectively, the benefits of opening up the data will spill out.
posted by Luigi Montanez
on April 23, 2009 #
What you don’t give due attention to here is that improving existing government transparency is part of a larger information ecosystem. Digitizing previously paper-bound and siloed databases makes possible a new kind of “connect-the-dots” work that used to take investigative journalists months to do—or they simply didn’t even try. Your own project, Watchdog.net, has pioneered a new way of finding links, or “handshakes” as you called it, between campaign contributors and congressional earmarks, and the demographic data you’ve cataloged is going to help power new tools for investigative types to answer such questions as “How many members of Congress who represent districts where X percent of their households have experienced a foreclosure last year voted for the banking bailout?”
We’re just at the beginning of a new wave of data- and citizen-empowered watchdogging of government, and it would be a huge mistake to discount how valuable these government transparency projects are in opening up the process in fundamentally new ways. Yes, it’s true that government still wants to hide its dirt and the data it discloses is often only part of the picture. But the more we open access to the existing data and involve people in connecting the dots, the greater the appetite and demand for even more transparency, of the sort even you might find useful.
posted by Micah Sifry and Clay Johnson
on April 23, 2009 #
the databases are always a small part of a larger picture, supplemented with interviews, documents, and even undercover investigation
This doesn’t refute the need for transparency websites.
I’m not sure if we should ever have expected the general population to visit watchdog websites to help them vote… I think the point was to allow investigative journalists to “outsource” the menial parts of data collection/analysis. That way, those who already have the persistence to do investigative journalism get more of it done, and we lower the barriers to entry to the profession for hobbyists.
posted by Andrey Fedorov
on April 23, 2009 #
Good rant. I think what Aaron is saying isn’t that these sites are not helpful, but that “the generous impulses behind transparency sites end up doing more harm than good.”
So, it’s the exuberance behind “transparency-for-its-own-sake” that he takes issue with. I’m sure he agrees with all the comments saying transparency is fine as part of a larger movement; I think he just wants to make sure people have the correct expectations of what transparency alone is (and isn’t) capable of.
posted by Alex
on April 23, 2009 #
Micah and Clay, Andrey: Then they should be directed by investigative journalists. But I called up the best and they just weren’t interested in any of the things the sites were doing.
posted by
on April 23, 2009 #
Micah and Clay further argue that transparency leads to more transparency and, doggone it, some of it must be useful.
If you believe that, then I have a bridge to sell you. As soon as you disclose anything, you create the incentive to falsify it. More disclosure will only lead to more complicated lies.
posted by Aaron Swartz
on April 23, 2009 #
“More disclosure will only lead to more complicated lies.”
If you believe that, I have some nihilistic hopelessness to sell you.
posted by Eric Mill
on April 23, 2009 #
It’s a fair point. If we look at corrupt states and not-so-corrupt states, what are the differences? I’m willing to bet it’s not the presence or absence of watchdog websites. It’s probably more to do with the structure of how politicians relate to and are bound by their constituents.
That doesn’t mean watchdog sites are useless. They might still be interesting because they could enable different forms of activism. They’re just not going to solve corruption on their own.
posted by Neil Kandalgaonkar
on April 23, 2009 #
By the way, this recent interview of David Simon, creator of the TV show “The Wire”, echoes your concerns in one segment. Statistics are always gamed, whether in No Child Left Behind or drug war arrests.
However, Simon is a former investigative journalist and he believes that journalism is the mode of discourse which is currently outdated, which has lost its power to outrage or change things.
http://www.pbs.org/moyers/journal/04172009/watch.html
posted by Neil Kandalgaonkar
on April 23, 2009 #
But transparency is often one part of a larger concept of fostering broader participation in budgetary decision making. This is for the most part still mainly by interest groups that themselves may not be truly representative, but it does create greater focus not just on corruption and awste but also on how budgets reflect the reality of policy deciasions on the ground e.g. you say you support education for the poor - show me where it is in the budget.
posted by
on April 24, 2009 #
Hooray for Aaron Swartz. To his observations we can add two more:
1. Power corrupts open government activists no less than anyone else.
With the arrival of the Obama administration the game changed for the “big names” in open government: suddenly, they had friends on the “inside” and a leadership (executive and legislative) that took up the rhetoric of open government. Suddenly they are “working closely with” a handful of government IT specialists and some legislative offices.
This is widely regard as a huge step forward for the cause yet it is anything but that. It’s a disaster for the movement. It takes the pace and substance of the movement out of their hands and gives political ownership of the movements to the very politicians who should be under scrutiny.
At the same time the incoming administration and the Congressional leadership “took over” the project of even *defining* government transparency, the open government leadership was tossed some bones. They get to be “expert consultants” (“some of the brightest minds in industry”, no doubt). They get some first-mover advantage setting up or extending existing their own transparency web properties.
The movement is at extreme risk of turning into something mostly about itself.
2. Journalism is only a small part of the answer. Organizing is needed, as well.
Aaron suggests investigative journalism as a way to cut through the politicization and agenda hoarding that threaten the open government movement. Certainly, it has a role.
What is palpably lacking in the open government movement is a broad base of support in the form of informed demand for transparency. As a society, we don’t have much use for this data.
In the hands of demagogues (sometimes calling themselves journalists) isolated snippets of the data - nearly meaningless in their isolation - can be good for evoking and directing outrage at this or that.
What we lack, though, is any systematic habit of people self-organizing to examine the facts and then to act upon them in ways that meaningfully impact government.
In contrast, around here at least, people know darn well how to organize around the school system or the public transportation system. Citizen committees spring up like wild-flowers, around these things. Important hearings and elected official meetings are frequently well attended. What remains of the local press gives voice to these groups.
On federal issues? Or even state issues? Not so much. There is no impressive amount of local organizing at those levels.
Creating new “transparency web properties” is unlikely to change that. Such sites draw heavily on the technology architectures and business models of commercial web properties and in so doing adopt the form, hence the function, of an asymmetric power arrangement of powerful site owner / operators and a mostly passive consumer audience who, if their input is used for anything at all, it is to be aggregated to further enhance the power advantage of the site owner / operators.
It is as if, by shear professional habit, the open government movement made the two successive mistakes of (a) treating government transparency as a category of entertainment; (b) then not even being especially entertaining compared to, say, demagogue journalists.
A telling case in point is surely the recent “teabagging” protests. Boy, if ever there was a teaching opportunity to spread the power of transparency tools to grass roots organizers: that was it - yet the open government movement was nowhere apparent in the stories.
——————
My conclusion from these two problems is that while the open government movement surely must remain engaged with their friends in government that the course there is more or less set for the next few years and little will change it. Meanwhile, much progress could be made by changing the emphasis from building web site properties to building freely shared tools (IT and social) for meaningful citizen organizing and participation.
-t
posted by Thomas Lord
on April 24, 2009 #
I posted your interesting and refreshing rant at the GoldTalk.com Forum with added links to the book titles and small editorial annotations … hope you don’t mind: http://www.goldtalk.com/forum/report.php?p=237190
posted by FastEddy
on April 27, 2009 #
Your article reminds me of Malcolm Gladwell’s Enron article from a few years ago:
http://www.gladwell.com/pdf/opensecrets.pdf
Gladwell distinguishes between situations we can’t figure out because we don’t have enough information, and situations we can’t figure out because, even though we have all the right information, we can’t distinguish the wheat from the chaff. He argues that it was possible to deduce Enron’s actual condition from public financial records, but that few tried or were able to, and most believed the “cover story” that ran along the surface of what the company disclosed.
I wouldn’t call Malcolm Gladwell an investigative reporter, but he might as well have been talking about exactly what you’re saying here.
posted by Nat Friedman
on April 28, 2009 #
The direction I’ve always envisioned for OpenCongress (http://opencongress.org) is one where constituents monitor and direct the activities of their representatives.
Data always seemed like an obvious first step, because it’s easy and hard to argue with (though this post does a good job).
But the goal is creating communities with enough weight to push government around on anything important (including steps to undermine structural corruption).
Maybe “controlcongress.org” would be a better name (it’s taken).
posted by Holmes Wilson
on April 28, 2009 #
You can also send comments by email.
Comments
Now, that’s a timely rant.
Everybody and their brother seem to be freeing data nowadays, but - for some odd reason - the actual uses (and perhaps even professional evaluation) are then put too far aside.
How does your rant add up to what Watchdog is up to currently? Are you looking for journalists, or…
posted by Tommi on April 23, 2009 #
I’ve always thought that getting the raw data and turning it into a format that’s useful was really just the first step. Open government data is a tool to use to solve the problem, but not the solution itself.
Once investigative journalists and researchers really figure out how to use those tools effectively, the benefits of opening up the data will spill out.
posted by Luigi Montanez on April 23, 2009 #
What you don’t give due attention to here is that improving existing government transparency is part of a larger information ecosystem. Digitizing previously paper-bound and siloed databases makes possible a new kind of “connect-the-dots” work that used to take investigative journalists months to do—or they simply didn’t even try. Your own project, Watchdog.net, has pioneered a new way of finding links, or “handshakes” as you called it, between campaign contributors and congressional earmarks, and the demographic data you’ve cataloged is going to help power new tools for investigative types to answer such questions as “How many members of Congress who represent districts where X percent of their households have experienced a foreclosure last year voted for the banking bailout?”
We’re just at the beginning of a new wave of data- and citizen-empowered watchdogging of government, and it would be a huge mistake to discount how valuable these government transparency projects are in opening up the process in fundamentally new ways. Yes, it’s true that government still wants to hide its dirt and the data it discloses is often only part of the picture. But the more we open access to the existing data and involve people in connecting the dots, the greater the appetite and demand for even more transparency, of the sort even you might find useful.
posted by Micah Sifry and Clay Johnson on April 23, 2009 #
the databases are always a small part of a larger picture, supplemented with interviews, documents, and even undercover investigation
This doesn’t refute the need for transparency websites.
I’m not sure if we should ever have expected the general population to visit watchdog websites to help them vote… I think the point was to allow investigative journalists to “outsource” the menial parts of data collection/analysis. That way, those who already have the persistence to do investigative journalism get more of it done, and we lower the barriers to entry to the profession for hobbyists.
posted by Andrey Fedorov on April 23, 2009 #
Good rant. I think what Aaron is saying isn’t that these sites are not helpful, but that “the generous impulses behind transparency sites end up doing more harm than good.”
So, it’s the exuberance behind “transparency-for-its-own-sake” that he takes issue with. I’m sure he agrees with all the comments saying transparency is fine as part of a larger movement; I think he just wants to make sure people have the correct expectations of what transparency alone is (and isn’t) capable of.
posted by Alex on April 23, 2009 #
Micah and Clay, Andrey: Then they should be directed by investigative journalists. But I called up the best and they just weren’t interested in any of the things the sites were doing.
posted by on April 23, 2009 #
Micah and Clay further argue that transparency leads to more transparency and, doggone it, some of it must be useful.
If you believe that, then I have a bridge to sell you. As soon as you disclose anything, you create the incentive to falsify it. More disclosure will only lead to more complicated lies.
posted by Aaron Swartz on April 23, 2009 #
“More disclosure will only lead to more complicated lies.”
If you believe that, I have some nihilistic hopelessness to sell you.
posted by Eric Mill on April 23, 2009 #
It’s a fair point. If we look at corrupt states and not-so-corrupt states, what are the differences? I’m willing to bet it’s not the presence or absence of watchdog websites. It’s probably more to do with the structure of how politicians relate to and are bound by their constituents.
That doesn’t mean watchdog sites are useless. They might still be interesting because they could enable different forms of activism. They’re just not going to solve corruption on their own.
posted by Neil Kandalgaonkar on April 23, 2009 #
By the way, this recent interview of David Simon, creator of the TV show “The Wire”, echoes your concerns in one segment. Statistics are always gamed, whether in No Child Left Behind or drug war arrests.
However, Simon is a former investigative journalist and he believes that journalism is the mode of discourse which is currently outdated, which has lost its power to outrage or change things.
http://www.pbs.org/moyers/journal/04172009/watch.html
posted by Neil Kandalgaonkar on April 23, 2009 #
But transparency is often one part of a larger concept of fostering broader participation in budgetary decision making. This is for the most part still mainly by interest groups that themselves may not be truly representative, but it does create greater focus not just on corruption and awste but also on how budgets reflect the reality of policy deciasions on the ground e.g. you say you support education for the poor - show me where it is in the budget.
posted by on April 24, 2009 #
Hooray for Aaron Swartz. To his observations we can add two more:
1. Power corrupts open government activists no less than anyone else.
With the arrival of the Obama administration the game changed for the “big names” in open government: suddenly, they had friends on the “inside” and a leadership (executive and legislative) that took up the rhetoric of open government. Suddenly they are “working closely with” a handful of government IT specialists and some legislative offices.
This is widely regard as a huge step forward for the cause yet it is anything but that. It’s a disaster for the movement. It takes the pace and substance of the movement out of their hands and gives political ownership of the movements to the very politicians who should be under scrutiny.
At the same time the incoming administration and the Congressional leadership “took over” the project of even *defining* government transparency, the open government leadership was tossed some bones. They get to be “expert consultants” (“some of the brightest minds in industry”, no doubt). They get some first-mover advantage setting up or extending existing their own transparency web properties.
The movement is at extreme risk of turning into something mostly about itself.
2. Journalism is only a small part of the answer. Organizing is needed, as well.
Aaron suggests investigative journalism as a way to cut through the politicization and agenda hoarding that threaten the open government movement. Certainly, it has a role.
What is palpably lacking in the open government movement is a broad base of support in the form of informed demand for transparency. As a society, we don’t have much use for this data.
In the hands of demagogues (sometimes calling themselves journalists) isolated snippets of the data - nearly meaningless in their isolation - can be good for evoking and directing outrage at this or that.
What we lack, though, is any systematic habit of people self-organizing to examine the facts and then to act upon them in ways that meaningfully impact government.
In contrast, around here at least, people know darn well how to organize around the school system or the public transportation system. Citizen committees spring up like wild-flowers, around these things. Important hearings and elected official meetings are frequently well attended. What remains of the local press gives voice to these groups.
On federal issues? Or even state issues? Not so much. There is no impressive amount of local organizing at those levels.
Creating new “transparency web properties” is unlikely to change that. Such sites draw heavily on the technology architectures and business models of commercial web properties and in so doing adopt the form, hence the function, of an asymmetric power arrangement of powerful site owner / operators and a mostly passive consumer audience who, if their input is used for anything at all, it is to be aggregated to further enhance the power advantage of the site owner / operators.
It is as if, by shear professional habit, the open government movement made the two successive mistakes of (a) treating government transparency as a category of entertainment; (b) then not even being especially entertaining compared to, say, demagogue journalists.
A telling case in point is surely the recent “teabagging” protests. Boy, if ever there was a teaching opportunity to spread the power of transparency tools to grass roots organizers: that was it - yet the open government movement was nowhere apparent in the stories.
——————
My conclusion from these two problems is that while the open government movement surely must remain engaged with their friends in government that the course there is more or less set for the next few years and little will change it. Meanwhile, much progress could be made by changing the emphasis from building web site properties to building freely shared tools (IT and social) for meaningful citizen organizing and participation.
-t
posted by Thomas Lord on April 24, 2009 #
I posted your interesting and refreshing rant at the GoldTalk.com Forum with added links to the book titles and small editorial annotations … hope you don’t mind: http://www.goldtalk.com/forum/report.php?p=237190
posted by FastEddy on April 27, 2009 #
Your article reminds me of Malcolm Gladwell’s Enron article from a few years ago:
http://www.gladwell.com/pdf/opensecrets.pdf
Gladwell distinguishes between situations we can’t figure out because we don’t have enough information, and situations we can’t figure out because, even though we have all the right information, we can’t distinguish the wheat from the chaff. He argues that it was possible to deduce Enron’s actual condition from public financial records, but that few tried or were able to, and most believed the “cover story” that ran along the surface of what the company disclosed.
I wouldn’t call Malcolm Gladwell an investigative reporter, but he might as well have been talking about exactly what you’re saying here.
posted by Nat Friedman on April 28, 2009 #
The direction I’ve always envisioned for OpenCongress (http://opencongress.org) is one where constituents monitor and direct the activities of their representatives.
Data always seemed like an obvious first step, because it’s easy and hard to argue with (though this post does a good job).
But the goal is creating communities with enough weight to push government around on anything important (including steps to undermine structural corruption).
Maybe “controlcongress.org” would be a better name (it’s taken).
posted by Holmes Wilson on April 28, 2009 #
You can also send comments by email.