Next Community Hangout 12 June

We’ll be running another Community call via Google Hangout Wednesday this week.

When: 12 June, 19:00 CET / 18:00 GMT / 13:00 EDT – your time zone

Where: On Google Hangout. Sign up in the form below.


– “How to crunch numbers from the European Investment Bank (EIB)?”. Introduction by Anne Roggenbuck from Bankwatch about how they analyse loan data from the EIB.
– News from around the community: Who’s got something to share – who needs help moving forward?
– Procurement data on OpenSpending: French and Senegalese procurement are among the most recent contributions to OpenSpending. Should this lead to any changes in the way the upload module is set up?
– Planning the Global Spending Weekend July 20-21

  • Updating budget and spending data in the Open Data Census. Is your country up to date?

  • New Features in May 2013

    Image by Mosman Library (cc-by 2.0)

    This is perhaps not a very attention grabbing title for this post, but some cool new features have landed in OpenSpending this month. We’re really proud of them so we thought we’d share them with you.


    We’ve got badges! OpenSpending administrators can now award badges to outstanding datasets. To begin with there are only a few badges but we foresee collaboration with organisations that want to give their approval to datasets. These organisations can do quality assurance on datasets originating from them but have been modified to fit better into OpenSpending (the badge would then say: “Yes this is still the data we published”). These organisations could also give badges to datasets that help their cause (the badge would then say: “This dataset helps us reach our goal”).

    If you’re representing an organisation and want to be able to give out badges, please get in touch.

    If you’re managing a dataset, find out which badges you can get and start collecting!

    Show Analysis Results

    After loading a source users can’t really see if the load was successful. It may appear to be successful but that might be just because OpenSpending was able to download something.

    To provide some kind of feedback OpenSpending now shows the results of the analysis it does on the data. In particular it shows the columns that it found. If there aren’t any columns or the column names are weird, users might now catch it before something goes wrong.

    EU Cookie Compliance

    This may not be a big thing for you and it might even be slightly irritating but for us this is really important. We want to follow the law and OpenSpending now does so by implementing the EU Cookie Directive (if you want to know about cookies being placed on your computer this is important to you too).

    Users are now presented with a small banner that tells them about the cookies and offers them link to a page where they can read more about our cookie policy.

    OpenSpending Contacts Map

    There are many projects out in the world that work in some way with spending data. We want to be able to connect those initiatives together. They can attract new contributors, learn from other projects and spark new interesting projects (even in other countries).

    To help you establish these connections or to find projects you are interested in we’ve put up an OpenSpending app on There you can see the world and find all of the projects that relate to OpenSpending in some way.

    If your project isn’t there, don’t panic! Thera are lot of projects so we might have missed some of you. Please let us know about your project and we’ll add it to the list right away. When we’re sure we have most of the projects on the list we’ll make this map more prominent on OpenSpending’s main site.

    Satellite Template

    Before May users had to fork Where Does My Money Go? or some other site in order to create their own satellite site (a site that provides context to the data and analysis in OpenSpending).

    In May we created a satellite site template so you can easily recreate sites like Where Does My Money Go? with a simple config file without having to remove a lot of context specific information.

    Just fork our satellite template repository and start configuring your own satellite site.

    Other Changes

    There are loads of other smaller changes we did this month. We went through all of the issues in our issue tracker that were labelled as bug and fixed them. We tried to make some instructions clearer. We made small headway in getting better IE7 compatibility and we now show the type of a dimension that has been created with the model editor.

    Want to contribute? We need code reviewers!

    We’re really proud of what we’ve achieved this month. There are a lot of upcoming changes in the pipelines (in the form of pull requests). Quality assurance is really important to OpenSpending so we don’t add anything new to the platform unless it’s been looked at by at least one other developer (this is called code review).

    If you’re interested in contributing to OpenSpending, code reviews are a good place to start. You get familiar with the code base and you can (and should) raise all kinds of questions (so if in doubt about anything, just ask the developers).

    There’s even a law in open source software (Linus’ Law): “Given enough eyeballs, all bugs are shallow”. That means the more people do code review, the better OpenSpending will get. That’s why we’d appreciate your help in getting some more quality code into OpenSpending. Look at the pull requests in our repositories on GitHub and comment, both if you see something fishy and if you don’t!

    Interested in being a code reviewer? Get in touch!

    Data Clinics and Community Hangouts – going forward

    A lot of great spending projects in the making are in the need of community support for tips on how to clean and upload the data.

    For this reason we’ll therefore begin to run bi-weekly Data Clinics – a community drop in for all kinds of technical troubleshooting you need in order to get working on OpenSpending.
    All you need to bring is your messy data, and you’ll be able to ask the community on tips on how to tackle your spending data.

    The first Data Clinic will take place today Wednesday 5 June at 19:00 CET / 18:00 BST / 13:00 EDT as chat via skype (ping: anpehej) or IRC #openspending. Sign up here!

    With the introduction of the Data Clinic we’ll now offering community activities every Wednesday – either as Data Clinic or Community GoogleHangout. We hope that you’ll join some of the activities!

    Current community projects

    Here are some of the projects currently in the works. Many of them could need a helping hand:
    – Felix is working on Andalusian procurement data. Data quality seems good, and would be interesting to compare with the data available in OpenTED.
    – Cecilia and the investigative journalism team at IRPI has claimed the Italian structural fund data and will be working to add it to OpenSpending. They need help cleaning a few of the data fields so please get in touch if you can help!
    – Tarek would be interested looking at spending data from Egypt
    – Pierre has helped scrape, clean and upload procurement data from Senegal to OpenSpending. He’ll post on the OpenSpending blog soon.
    – Benjamin is working on the EU procurement data
    – Transactional spending data from the city of Washington DC was added during theweekend as part of #Hackforchange.

    Global Weekend on city spending data July 20-21

    The past months has seen strong growth in local spending projects with 86 cities mapped. As we want to support and grow these initiatives we’re asking if you’ll join us to organise a global spending weekend to open city spending on July 20-21. The spending weekend will offer online trainings and support to your local group to help you get your project off the ground.
    Hiroichi (OKF Japan), Mor (Hasadna) and Adam (Open Budget Oakland) have already expressed interest in the idea, and we’d love to hear if you want to be involved.

    Have a look and add your self to the planning document.


    Thanks to Nigel Babu, Lucy Chambers, Martin Keegan, Anders Pedersen, Rufus Pollock, Stephen Russett, and Stefan Wehrmeyer for their contributions this month (there are probably a lot more who’ve contributed somehow to this month’s features so don’t be sad if we forgot you, let us know and we’ll add you).

    Image of badges used for this blog posts is by Mosman Library on Flickr, released under Creative Commons Attribution, version 2.0.

    Hack for Change helps publish data from Washington DC

    This weekend thousands of people gathered across United States for a national day of civic hacking. OpenSpending joined organisers at Code for DC in Washington to work on transactional spending data published by the city for 2010 to 2013. The data now allows you to browse spending on charter schools and contractors in the corrections industry.

    During Hack for Change we met with journalists and citizens to hear more about what kind of questions that could be answered with the data available.

    If you’re interested to learn how you can get spending data from your city onto OpenSpending, get in touch via our mailing list or Twiiter.

    Making sense of French press aid

    This is a guest post by Samuel Goëta, OKF France.

    On May 22nd members of the OKFN local group France met at la Cantine, the digital hub of Paris, for visualising press aid data. This workshop held prior to a conference organised by the Social Media Club France on the efficiency on press aid for innovation in the media industry. The results of the 3-hour intensive workshop were presented right after to the audience and the speakers of the conference.

    The press has received support in France since Second World War as it is considered critical for ensuring all opinions can circulate in society. These aids are direct subsidies as well as indirect in the form of tax breaks, reduced post fares and subsidised distribution. More recently, in April 2012, the press industry was granted the Strategic Found for the Development of the Press (fonds stratégique pour le développement de la presse) accounting for 60 millions euros of aid. This found was associated with a commitment to transparency, data on the 9 million euros spending, was published in April 2013 on the French open data portal.

    photo (3)

    Following the publication of data on press aid, 5 members of OKF France got hands on the data available. Every participants had no previous experience with data visualization and did not even prepare this workshop. The challenge was to prove that, using open resources and with perseverance, one can visualise a complex system of subsidies in a couple of hours. Data cleansing was obviously the hardest part of the work as participants had little experience of handling data.

    The workshop was divided in two groups, one using Open Spending to visualise data on an interactive treemap.

    Another group used D3 Javascript library to map data on an interactive bubble chart.

    The presentation of the visualizations between the two roundtables of the conference raised great interests, as most speakers were speaking about data they could not visualize nor source. The results were directly published on checkthis. The poster spread quite quickly as it reached 400 views in less than 24 hours. This workshop shows that mediation is crucial if we want to develop data literacy. Open data publishers should encourage and support such events as they can also benefit from simple visualisation everyone can use and understand. It also shows the limits of transparency on press aid: although aid is distributed for the development of innovation projects, the details of the funded projects are not mentioned. Officials state it could not be published based for trade secrets reasons. Also we were able to visualize only the tip of the iceberg as data is published for 30 millions € of press aid while total of direct and indirect aids accounts for 1,2 billion € in 2012. This workshop was a small, probably incomplete, effort to understand a complex and opaque funding system for sustaining an industry in the midst of massive changes.

    Opening the EU procurement database

    OpenTed datastore

    On 2 May civic coders and journalists from across Europe from Norway to Slovenia met up in Brussels for a Hack Day supported by Knight Mozilla Open News to dig into the EU procurement register, Tender Electronic Daily. Here are a few of the highlights from the day and where we’ve been able to take the TED data since the Hack Day.

    Scraping and parsing of TED into OpenTED

    Several people got together to build and improve the parsing of data already scraped. The past years has seen several attempts to scrape and parse TED, but past scraping projects have been missing the granular parsing needed in order to make the data useful for investigations.

    You can find the latest updates here – we’d love your help:

    User stories – ideas for data journalism on procurement data

    During the Hack Day a handful of data journalists sat down to line up questions, which should be investigated by journalists. “If the TED data could speak, what should we ask?”

    Key questions to ask the TED data:

    • Where in the EU are highways most expensive to build?
    • What are the best ways to expose cartel formations
    • What companies are running food services at public school (and what’s on the menu)?
    • Who is the biggest weapons salesman in the EU?
    • Who are the biggest public transport providers – Arriva vs. Connexion?
    • What’s the relationship between procurement process (time) and what companies gets the contract?
    • How many contracts go outside the EU?
    • Do the governments purchase from big or small suppliers?
    • How can we predict when a project will go over budget?
    • What sector is the least competitive (CPV-code vs. average bid)?
    • Are there companies within certain sectors that get a suspicous large amount of contracts?

    Using other data with procurement data from TED

    • Reconcile with company data from Opencorporates
    • Match contracts to actual spending data eg. UK government spending
    • Ask Freedom of Information requests to the procuring authority for entire contract (see examples of such successful FOI requests here and here
    • Get to understand what the contracts are actually about with the official taxonomy of the CPV-code

    A first look at the TED data

    The full TED data set is now available, thanks to the hack team and in particular Friedrich Lindenberg.
    You can access the TED data as CSV files EU wide or on a country by country basis. You can also get to know the data format without downloading it, as we’ve uploaded the data for Czech Republic (2013) as a sample.

    We hope that many of you will dig into it, and examine the contracts.

    Reviewing the data quality

    At a community call on May 15, community members decided to begin a quality review of the data available in TED.
    We’re asking you to help contributing to this review. If you’d like to help, please get in touch! In particular the review will examine missing amounts (non mandatory fields), missing contract fields and inconsistent data formats. The quality review can be really useful in order to get a clear picture of the state of the TED-data and help assess how useful it will be for journalists and researchers. We’d be particular interested to hear from those of you who are already working on procurement data in EU countries.
    Thanks to Julia Keseru from Sunlight Foundation and Ronny Patz Transparency International, Brussels for offering input on this.

    Should OpenSpending include procurement data?

    Procurement data is increasingly becoming available in more countries such as Senegal, Ukraine and Australia, where transactional spending data remains unavailble. By definition procurements are clearly different from transactional spending data. While “contract awards” should indicate a final price, they might not be final, as ammendments regularly occur in contracts and did anyone ever hear of an ICT-project, which went over budget? In such a case chances are that you as a journalist or watchdog would prefer to have access to the actual spending data, but if this is not available procurement data can still help you get closer to your story. Contracts might also lack clarity as to when payments are to actually occur. Finally procurement data will most often only include contracts above certain thresholds, which makes significant shares of actual contractor spending unaccessible.

    I could be relevant for OpenSpending to add procurement data next to bugdet and spending data, because several countries with available procurement data, do not provide access to the ideal option of transactional spending data. This is for instance the case in most EU member states where procurements from OpenTED today is likely offering the most granular data on public spending.

    Do you think OpenSpending should begin adding more procurement data? Let us know!

    Introducing the Progress page

    For the past months we’ve seen many new users come to the OpenSpending community. The community have now added budget data from more than 80 cities and most recently universities have also found their way onto OpenSpending.

    While we’re thrilled by the uptake in use of openspending for visualising budgets, we’ve however also noticed that datasets from transactional spending are added less often than we’d hope for. We would therefore want to make it easier for the community to track what other people are working on and offer their help if needed in sutiations like these:
    – “I’m working on a dataset, but got stuck cleaning the spreadsheet – who can help?”
    – “I’ve found this excellent dataset on Italian structural Fund payments and would like to add it to OS, but need help translating the columns before uploading.”
    – “I wonder if someone is already working to add the latest EU budget to OpenSpending?”

    Enter – the Progress page

    With the Progress page we wish to make it easy to show what datasets people are working on.

    What can you do there?

    • Claim a dataset you’d like help to clean and upload to OpenSpending – and ask the list for help if needed
    • Check what datasets other people are working on – and offer your help if you can
    • Request a dataset you would like to see on OpenSpending

    At this early stage we’re really interested to hear feedback from you, so let us know how we can make the Progress page useful to track how datasets are moving forward. Share your thoughts on our mailing list.

    Do you feel like contributing to the code of the Progress page, check our issues on Github.

    Opening Israel’s Budget

    Public Knowledge Workshop

    This is a guest post by Aviv Sharon and Adam Kariv from The Public Knowledge Workshop based in Israel.

    “Show me the money!” If you ask the developers of “Taktziv Patuakh” (Hebrew for “Open Budget”), one of the projects of Israeli non-profit “The Public Knowledge Workshop”, that’s more than just a line from a film. The government’s true priorities are reflected in the budget, not in its statements. That’s what makes budgets interesting. And that’s why looking at budgets is one of the most important things that the public should be demanding from the government. As the negotiations for Israel’s 2013 budget draw near, informed public discourse on budgetary matters becomes an even more pressing need.

    There are a few important decision-making points in the budgeting process that are most important for the Israeli public to keep track of. First, each law and ordinance gets an overall, macro budget. Then, that budget is broken down into smaller items, on which the state can spend money for purposes like paying salaries, ordering equipment, supporting various non-profits and more. If we, as the public, don’t keep track of the whole process, the state can deceive us in several ways: The state might decide on a policy but not budget it, it might budget it but not spend the cash, and it might spend it in ways worth watching closely, like paying suppliers that were chosen without conducting a tender. In short, there are too many leaky holes in this pipe, and it’s hard for the public to follow the money.

    The flow of money through Israel’s coffers became a concern for the developers of “Open Budget” following a large forest fire on Mount Carmel in December 2010, the deadliest and most widespread fire Israel had ever seen. After the fact, Israel’s ministries of Finance and Interior blamed each other for having neglected the fire brigades. On the one hand, the Ministry of Interior claimed that its requests to increase the budget for the brigades were only partially funded by the Ministry of Finance, and very late at that. On the other hand, the Ministry of Finance claimed that it had passed along all the sums stipulated in the relevant governmental resolutions. The Ministry of Finance also added that these resolutions had even greatly increased the funds for aerial firefighting in 2010 compared with previous years.

    Open Budget 1.0

    To clear up the argument, activists from Israeli non-profit The Public Knowledge Workshop asked Michael Eitan, then Minister of Improvement of Government Services, to help retrieve data they could use to put together a comprehensive picture of Israel’s state budget. Unfortunately, the Israeli Ministry of Finance did just the bare minimum and released several files in various formats, such as Excel, PDF and html. The files were rife with human errors, like parts of the Hebrew text mistakenly written left-to-right. (Hebrew is written from right to left.)

    First, the “Open Budget” developers at the Public Knowledge Workshop took the data and organized it in a uniform database. Secondly they built a web-based interface allowing any user to explore the budget. Eventually the database of the Public Knowledge Workshop reached a stage where this was the most accurate and comprehensive database of Israel’s budget. Some ministries have even asked to export data from the Open Budget for them, because the state systems were much more cumbersome.

    The first version of Open Budget 1.0 was launched in March 2011 and is still available on Israel’s governmental services portal. It allows the public to track budget changes over time, search items for particular keywords and compare the budgeted amounts with the sums spent on each item.

    Open Budget 2.0

    Following the first release, the “Open Budget”-developers realized that many of their basic assumptions for analyzing the data were wrong, and that several important features were missing in version 1.0.

    For instance, they knew that each item of the budget had an ID number, like item 15 for the Ministry of Defense budget, and assumed that such ID numbers would be consistant over time. This turned out however to be wrong. The Ministry of Finance changed the meanings of many items and sub-items from year to year, and therefore tracking spending on a given item over time based on the ID number seemed meaningless. While most high-level numbers usually kept similar meanings over time, the Ministry of Education (Item 20), for example, had changed responsibilities over time: Namely, it was in charge of sports and culture in some years, but not in others. This made it hard to track the education budget properly over time. Regrettably, the IDs on lower level budgetary items were detected to be even less stable in meaning, and little or no annotations were made available from the government to help observers discern the semantic changes.

    Moreover, to track spending on issues such as for instance firefighting, one would need to put together the sums spent on various different items, paying attention to the differences between salaries, procurement and other items.
    Last but not least, while the budget may contain all the data, one can’t easily determine if the sums are too much, too little, or just right. Nor can that be determined objectively. This is a value judgment, and different people will analyze the budget in different, and sometimes contradicting, ways, and use different data to frame and support their arguments. For example, one could argue spending in the Ministry of Education was appropriate 30 years ago, but has not kept up with the number of students and inflation. Hence, a good system for viewing the budget would allow one to easily examine spending versus those data. What’s more, a good system would then allow experts and civic activists in Israel to publicly annotate budgetary items with their respective opinions, to enrich the budgetary debate with accessible, evidence-based arguments.

    These needs are addressed in Open Budget 2.0, which is due for release from the Public Knowledge Workshop within the next few weeks. Namely, the next version of the system will allow users to:
    – unify related budgetary items,
    – analyze them together,
    – compare spending between different items, or between budgetary items and other data sets, which can help put the budget in a larger context, and
    – publicly annotate budgetary items.

    Looking Ahead

    Our dream at the Public Knowledge Workshop is to make any Israeli governmental expense as transparent as reasonably possible, in real time.

    Predictably, one of the biggest roadblocks for progress on budgetary transparency comes from the government itself. Currently we are trying to make the most of the data, which have already been released, but the Ministry of Finance and other branches of government still hold a lot of data sets which they reject to share. Open Budget developers suspect that until they are forced to release them, it will stay that way. Some examples of data sets yet not released are:
    – state-provided evaluations of the effect of each budget item on the economy,
    – budgets on the ministerial level,
    – budgets of governmental agencies,
    – details of individual procurements, and
    – much greater detail on outsourced activities.

    Recent changes

    Presumably as part of the changes made by the new Israeli Minister of Finance, Yair Lapid, the 2012 budget was made fully available to the public on April 12, 2013. Amazingly, a representative of the Ministry personally contacted the Public Knowledge Workshop to verify that activists at the Workshop had seen the new release. Hopefully, this is a sign of more openness to come.

    We encourage new volunteers to join the effort. There are many important causes to contribute to, but extracting knowledge from a bunch of numbers, that is a real challenge.

    Aviv Sharon is a volunteer at the Israeli non-profit “The Open Knowledge Workshop”. He writes materials for the public and plans educational projects.
    Adam Kariv is a board member of “The Open Knowledge Workshop” and project leader of “Taktziv Patuakh” (Hebrew for “Open Budget”).