Who’s talking? Social media audits and finding the conversations

As regular readers will be aware….I have a bee in my bonnet about the need for someone to start building civic spaces online – spaces which are designed to support civic and political discourse rather than designed to sell us stuff. However it’s all very well having the idea – you then have to figure out how to build it.

This post provides an overview of the social media audit – a piece of research that is carried out before you set up a civic space in order to gain an objective view of who you should be including in the conversation. I use ‘we’ a lot in this post as though I had the bright idea of doing the audit and put some structure in place its my team at Public-i who have done most of the detailed development of the process.  We’ll be blogging more about this over at the Public-i blog but here is the first draft of the overview that will end up in the thesis.

In essence what we are trying to do is to find the conversations which are already taking place in the local online space.  More importantly we are trying to find the active individuals in order to create a network response to civic interactions – civic spaces are going to be defined by the networks that share them as much as by the content.

A bit of background

When I started looking at this I thought about this idea in terms of government building these spaces. I was influenced by Stephen Coleman’s thinking around ‘A Civic Commons in Cyberspace’ and also Castells’ work that shows the insidious power of media conglomerates and negative impact that gas on objectivity in the press that this brings to the fourth estate (Castells, “Communication Power”). This, combined with the fact that I have been immersed in working with Local Government for almost the last 10 years led me design the “Virtual Town Hall” pilot which you can read about here. The name really gives it away – I was imagining a civic space built by government – echoing real world civic architecture – and then used by the public.

I persisted with this idea for a while and blamed the fact that we were being slow to implement the technology for the fact that the pilot sites were not taking flight. There is no doubt that we were being slow with the technology implementation but I now believe that the reasons for the pilot sites not getting off the ground were more complex than just that and that there were a number of issues with the way I had originally designed the virtual town hall solution, the main one being that the original project design didn’t have the right role for the community. We envisaged using unmoderated community content and then using community moderators or champions to widen involvement but this was really a compromise en route to what has become the inclusion of the affordance of co-production in the final pilot sites. We have to accept that we can only work effectively with the public online if we don’t try and control the conversation that the community moderators were in some way an attempt to manage risk from the point of view of the Council without truly considering their wishes in this.

However once it became clear that these spaces, even if facilitated by government, needed to be equally owned by all stakeholders another issue arose; who do we include in the conversation? The community that you contact to create the civic space is going to integral to how it behaves and even though we would expect participation to shift throughout the life of a civic space that initial group is significant in terms of how likely you are to get an independent conversation started and also in terms of what tone is set for the space from the outset.

Its turns out that picking this group was causing project paralysis – no-one could get started until they knew who to include in the process. I’m going to do some follow up interviews on this point but I think that the issue here was a mix of risk and representativeness. The first was a concern about making the ‘wrong’ choice because we weren’t aware of the full picture. The second is more complex – but I think highlights the real democratic tension here which is the fact that the people who are active online are not representative of the general population and this is both a good and a bad thing. Good in that they are more likely to be civic and active offline as well (OXIS, Coleman) but bad in that they are not well…representative. The solution here is fiendishly simple and fiendishly difficult – involve the elected representatives – but that’s for another post.

Social Media Audits – a solution to the problem

The starting point as the fact that a civic space can’t be initiated until you have some idea as to who might be participating.  The social media audit is a response to this problem – its a systematic piece of research that provides a representative snapshot of the local informal civic conversation so that you can make am informed decision about who to include in the initial iteration of the civic space. Not only that – practically speaking – it gives you the list of people to contact , the conversational lures they are interested and a view of the interactions which are already going on.

We wanted to create an objective view of what was happening so that we provided a starting point for engagement with the local civic content creators. We can’t expect to find everything – and the content will change from week to week – but we were looking for a way to provide a starting point that would then be built on rather than freezing the results in time. Its important that the output of the audit also provides the means to extend and continue to search so that the civic space is created in a state of always being open to new voices.

Objective is a difficult thing to achieve as ultimately this process comes down to making value judgements about which sites should be included in the civic space. What we have therefore done is to create as robust and re-creatable process around the creation of the data set and then been as transparent as possible in terms of qualification of that data set down to something which is manageable for analysis and then for engagement with individuals.

This has deliberately been designed in this way rather than a piece of more quantitative analysis around the number of sites located in a specific area for example as we are trying to uncover individuals with specific intents rather than just to content that they are creating online – we are trying to connect to people as well as places.

What are we looking for?

The audit is designed to find not only an overview of the informal civic participation in the area but specifically to focus in on the significant content creators who will be the most vocal contributors to the civic space. The choice of the word ‘significant’ is deliberate here – we’re not trying to judge influence – just activity.

Significance is a fairly subjective term and so we try and define this with the site hosts to make sure we have a clear idea of what we are looking for. Once a site has been found via the relevant search terms then broadly we are after:

• Persistence – we are looking for sites and individuals that are active over a reasonable period of time – or are linked to a specific campaign – not 2 post blogs that have been set up with the best intention
• Audience – we can’t easily judge audience but we are looking for indications that the creator is aware of an audience and wants to interact with it
• Constructive – we are looking for voices that want to improve their community not just complain

This last one is the most difficult – judging intent from content is extremely tenuous. Another way of looking at this is to say that we looking for content and creators who would satisfy a simple code of conduct test for any community website. Codes of conduct exist to ensure that interactions are respectful and do not insult some basic principles. The point of this filter is to try and rule some of these people out from the start. I see this as largely a pragmatic decision – no council is going to put together a civic space which includes inappropriate content from the start – but its one that needs to be kept under consideration to make sure that the space remains inclusive and open.

Its also worth noting that we usually issue a health warning with respect to language – the language online can be robust but this needs to be included. This issue of language is a cultural one where you need to understand that the social web can use a different tone to that which the more formal world is used to.

What’s the process?

To state the obvious – the internet is huge – and if you try and do this on a rolling basis then you just keep searching forever.  Instead we create a snapshot which we know will not include all the content but will be representative of the local civic space to an acceptable degree.  Here is how we go about creating that snapshot:

• Define a matrix of search terms: This point about language is relevant from the onset of the audit. The process starts with a definition of search terms based on place and topic. We are trying to identify the language that the local residents are using to talk about where they live and about current affairs. We are seeking the stories that are currently active as these are the ones which illuminate activity.
• Create a data set: we then use a combination of advanced use of google and link analysis to create an initial data set. This can be done largely automatically and then gets deduped and cleaned up. This second step may create a data set of over 1000 sites.
• Qualify the data set: Once we have the data set narrowed down to around 3-400 then there is a manual qualification task which is the really time-consuming bit as we check each site against the significance criteria and also categorise it for place, topic, type of site and a few other metrics. We also highlight interesting examples – and also the downright odd stuff that you find online.

At this point we would hope to have a well qualified data set of around 200 sites that give us a good overview of the local informal civic activity. We do not know if these numbers of going to provide a useful benchmark – we’ve run the process a number of times now and they seem consistent but we expect them to keep increasing. However – at the moment – we believe that the 200 sites for a County or urban area is a reasonable benchmark to work against.

And the analysis

This is my favourite bit…

Once we have a coded up spreadsheet then we can do some straightforward statistical analysis and look at the spread of sites and content creators in terms of location and topic. We can see what proportion of activity is on Facebook for example (yes – we even search there), examine interactions on local media sites and see if there are pockets of activity around a specific place. We then use this to identity clusters of sites for a short case study analysis – which is really focused on looking at what is causing the cluster and how it might be used to introduce the group into the civic space.

The other piece of analysis is to use twitter as the starting point of a social network analysis of the space. This is really just a starting point for this and can be considered to be a snowball approach to an open network (Wasserman) rather than a real piece of SNA but what is does show is the potential reach of the civic creators. For my own research purposes I then ask the civic creators we have found to complete a more through social network analysis questionnaire which looks more deeply not just at their online but also their offline networks.

What don’t we do?

In developing the audit process we considered using semantic analysis tools bit in the end concluded that they didn’t offer the sophistication of search combinations that we were after and, more importantly, are designed to find content rather than individuals.

I think we could probably use more of the mainstream analysis tools but to date have not found anything that delivers what we are after – we’ll keep researching this however I will post findings on that when I have time.

It may be possible to have the same result through word of mouth as opposed to this fairly labour intensive research – ie by asking community participants to self report activity. My concern with this approach is that many of the sites that we find are not really describing themselves as civic – they are just people who are doing something that they think is interesting and they don’t feel the need to define it.

And the impact?

Its too early to say what the overall impact will be on the civic space but we have definitely succeeded in overcoming the project paralysis issue and have also been able to shape appropriate approaches and messages in order to involve these content creators in the initial proposition of the shared civic space and I wouldn’t want to try and instigate a site without doing this kind of research as we have not yet failed to turn up content and individuals that the host was not aware of before.

Even without knowing what impact it will have on the civic space its clearly a really effective way of getting a feel for the local activity in order to shape any kind of intervention online.

Its also an excellent way to deal with the people who are still saying that they don’t need to engage online – this is as robust a process as we can make it and is carried out based on search terms that the host defines – excellent and relevant local facts to put in front of anyone who thinks digital engagement is still optional at this point.

We are going to continue to work on the process and also on the automation of the process where possible. We are also trying to build in the idea of ‘discovery’ where we start to set the civic space in listening mode in order to uncover new civic voices but this is still early days – I’ll keep you posted.

As ever comments on this are very welcome.