This story will probably ring bells with those who’ve taken the policy development path.

Ever had an idea for how to tackle a policy issue only to be told in no uncertain terms by ‘older, wiser, more experienced’ folks that it’s been tried and it didn’t work? That is intended to put a stop to further discussion of that option. And heck, sometimes it does. But let me suggest that it shouldn’t necessarily be the end of that conversation. A follow up question about the reasons it failed and any investigation or analysis of the failure might prove enlightening. Too often there may not have been any evaluation of the policy. It was simply parked in the graveyard of unsuccessful ideas.

This is just one reason why evaluation of policies is important. What if the reason it failed was an issue in delivery or design not in the policy itself? Because lots of policy is made on the run, without the option to run pilots or trials (themselves also often subject to derision with claims you can’t do them because people expect that they will always develop into full implementation), the ability to test and fine tune policy delivery and design is simply not even up for consideration. So that a policy developed in a short period of time, with little evidence and no opportunity to test ideas, is then condemned to the pit of total failure if it doesn’t deliver as hoped.

Leaving aside the important matters of evidence and pilots as excellent policy bases, evaluating policies can mean there is an opportunity to improve design and delivery and even if a policy seems to be achieving its objectives, improvements may be possible.

In some cases, for example in new or innovative areas, areas of significance and policies with large price tags, building in an evaluation framework makes lots of sense.

National Competition Policy unearthed a raft of legislative restrictions on competition that had not been reviewed for decades. Some of the legislation was from the 1800s. Maybe it was time for a review of legislation requiring innkeepers to have a certain number of stables for horses. Yes really. so having a review and evaluation requirement can save years of legacy programs and policies lingering long aftr their useful lives.

While legislation can often be unexamined for ages, regulations usually have a sunset clause, meaning they have to be remade after ten years. They have an inbuilt review mechanism. Legislation might include a statutory requirement to review some aspects at given periods, but many do not or the review focus is narrow. But what about policy that is not linked to legislation or regulation?

Again there are sometimes reviews included in the policy, but what would make this even more powerful would be consideration of what a review might look like and building in the collection of data during the life of the policy to assist in understanding how well the policy is achieving its objectives – surely the test for the success or failure of a policy. It’s generally easier to have good data if you collect it contemporaneously rather than try to discover it looking back, or use proxies or other substitutes.

To evaluate the success of a policy you need to start with the question …is it achieving its objectives? Sounds simple enough. But first you need to know what the objectives were. This is not always easy. Objectives are not always spelled out and after a certain time can be impossible to know. So while you can try to infer objectives and work on that basis, of course, it is preferable to be certain of the outcomes that were being sought in the first place.

Then you need to consider do those objectives still hold? Or have other objectives become more important? Did the policy have a finite end goal that’s been achieved? Has policy been designed and delivered in the best way possible? Have there been unintended consequences – both positive and negative? Are there lessons to be learnt? Are there improvements in design and delivery that could be made? And of course, the answers to these questions may well trigger other avenues of inquiry.

Too often policies and programs linger well after their best by date, or with a multitude of faults and collateral damage and they suck out money and resources that could be better used for other things.

One last word – for the moment – as I am sure I will revisit this topic.

I listened to a podcast yesterday that, in part, offered a view from academia on this. While I admit I am not always a fan of academic policy commentary, this woman really had a keen understanding of the mechanics of policy and government – albeit a US perspective – and I would recommend it to you if you have the time and/or the inclination. Econtalk: Jennifer Doleac – On Crime 

It is also interesting for the more general discussion on economics and the law.

Also I have a small caveat. The presenter is a pretty libertarian chap with whom I often have issues, not least of which is his constant reference to Adam Smith, but in this episode, he refrained.