Thursday, November 27, 2014

Ferguson FAQ

Recently I published an analysis of the Ferguson conflict that showed, using Twitter data, that there was a “red group” and a “blue group” who rarely talked to each other, thought very different things, came from very different backgrounds, and often were uncivil even when they did talk. Thanks to everyone who wrote to me about the analysis! Here are answers to the most common questions I’ve received.

What data did you use?

215,000 tweets containing the Ferguson hashtag collected between November 17th and 19th (prior to the announcement of the verdict).

What tools did you use to collect the data?

Python -- specifically, the tweepy library and a program I wrote which you can find here (described at more length here).

What tools did you use to analyze the data and make the visualization?

Python for analysis; Gephi for visualization. See Gilad Lotan’s excellent tutorial on how to use Gephi to analyze Twitter data.

How did you divide Tweeters into red and blue groups?

I used Gephi's community detection algorithm (on the adjacency matrix for the most frequent tweeters, where Mij was 1 if tweeter i had mentioned tweeter j in a tweet), sometimes known as the Louvain method. Essentially, this divides Tweeters into groups that mention each other frequently.

Regarding whether this grouping is valid: as I note in the piece, I am mindful of the fact that there are many ways to group data, and I think this is worth exploring further. One problem we always face is how many groups there are (see here and here). You can always sort of make it look like people hate each other by clustering the data into groups even if there isn’t necessarily any separation between the groups -- this is something to be wary of when looking at analyses like this one.

But I think several pieces of evidence (in addition to Gephi's striking visual) point to the validity of the red / blue division. The fact that the two groups are associated with the tweeters’ self-descriptions (like race and political affiliation) is revealing; the fact that the two groups are associated with tweeting different things is also revealing (and by no means something I expected to see -- for example, if you divide Twitter datasets by gender, you will frequently find that men and women tweet essentially similar things). This evidence is powerful because it is external -- it was not used to come up with the grouping, but it supports it.

In general, we often bring in such external evidence to argue that a grouping is valid. For example, in a biological analysis we might cluster genes into groups that show similar expression patterns (group A highly expressed in the liver and not in the lungs; group B highly expressed in the lungs and not in the liver). We would be more sure that the groups we had found were “real” if there was external evidence like a transcription factor that was known to turn on all the genes in group A, or a biological function that was common to all the genes in group A.

You said the blue cluster is much larger than the red cluster. What happens if you break down the blue cluster further?

I don’t know! Someone should figure this out.

Can I see your data or code?

Yes. I cannot make the data publicly available because of Twitter’s terms of service, but if you are a researcher with a project, shoot me an email. In addition to the two days of data used in this analysis, I also have several million tweets both from several months ago, when Ferguson initially made the news, and from after the verdict was announced.

As always, if you work at Twitter and have any objection to any of this, please email me -- I am acting in good faith and more than happy to comply with your requests.

25 comments:

  1. I am glad that I spent time reading your post. This is the first time I visit here. I found so many interesting stuff in your blog especially its discussion. Any students need accounting assignment help and you don’t know how to get the right and high-quality assignment Help then contact our expert team we have providing best assignment help you can also visit our website MyAssignmentHelpAu

    ReplyDelete
  2. Amazing for remaining likely to your Site yet again, it's been months for me. Beautifully this info that ive been waited for As a result very long. I would like this textual content to complete my assignment all through the university, and It can be bought exact same issue together with your brief write-up. Thanks, amazing share.สล็อตออนไลน์

    ReplyDelete
  3. Pleasant submit. I used to be inspecting continuously this website Which i’m surprised! Incredibly sensible information and facts specifically the last ingredient I deal with these kind of info an entire great deal. I were searching for this specific info for some time. Thank you and Great luck.สล็อต 999

    ReplyDelete
  4. enjoyable put up, sustain with this distinct interesting work. It genuinely is superb to know this matter is staying lined also on This website so cheers for applying time to discussion this!สล็อตวอเลท

    ReplyDelete
  5. I'm having difficulties to browse content material articles online rather usually, but I’m pleased I did lately. This is incredibly nicely composed along with your things are quite nicely-expressed. You ought to, don’t Anytime quit building.สล็อตแตกง่าย

    ReplyDelete
  6. Really good submit. I just stumbled on your weblog and needed to point out that I have actually savored searching your website internet site posts. All matters regarded as I’ll be subscribing with your feed and I hope you compose Yet again shortly!บา คา ร่า วอ เลท

    ReplyDelete
  7. Which may be the superb Mindset, Nonetheless is simply not guide to provide Every sence by any means preaching about that mather. Practically any strategy plenty of several thanks Coupled with I'd endeavor to promote your individual generate-up in to delicius Still it is apparently a Predicament utilizing your specifics Websites can it be attainable to be sure to recheck The assumed. a lot of thanks yet again.บา คา ร่า วอ เลท

    ReplyDelete
  8. I'm joyful you're taking gratification in That which you produce. This may make you stand way out from all types of other writers that drive terribly composed article content.สล็อตเว็บใหญ่

    ReplyDelete
  9. That appears to be fantastic nonetheless i am However not also Unquestionably positive which i like it. At any selling price will feel a great deal far more into it and judge Individually!บา คา ร่า วอ เลท

    ReplyDelete
  10. Fantastic artcile, but It might be significantly better if in future you may share much more information on this matter. Protect publishing.สล็อต ฝาก-ถอน true wallet ไม่มี บัญชีธนาคาร

    ReplyDelete
  11. Your do The work is really appreciated round the clock moreover the world. It genuinely is exceptionally a comprehensive and practical Internet site.บาคาร่าวอเลท

    ReplyDelete
  12. Your articles or blog posts are inventive. I am seeking ahead to examining the myriad of content material you have connected below. Thumbs up!สล็อตทรูวอเลท

    ReplyDelete
  13. An especially magnificent blog site web-site report. We have been genuinely grateful on your web site web-site publish. You will note loads of methods immediately following searching your place up.เว็บสล็อต

    ReplyDelete
  14. I do Consider This may be an handy post and it's vitally practical and expert. thus, I would like to thanks for that initiatives you might have got built-in crafting this text.เว็บสล็อตเว็บตรง

    ReplyDelete
  15. Lots of many thanks for locating the time to debate this, I come to feel strongly relating to this and value Acquiring out additional on this matter. If at all achievable, When you achieve knowledge, would you feelings updating your site site with additional specifics? It is actually amazingly useful for me.เว็บตรงสล็อต

    ReplyDelete
  16. I unquestionably getting pleasure in Each individual and every minimal bit of it. It is a wonderful Net site and fantastic share. I need to thanks. Outstanding task! You men do an awesome Web site, and have some terrific contents. Sustain The nice get the job done.เว็บ ตรง

    ReplyDelete
  17. Any time you use a genuine enterprise, you will be able to source Tips, share components and choose the formatting structure and style.เว็บ ตรง

    ReplyDelete
  18. I found a fantastic amount of consideration-grabbing matters in your web site especially its dialogue. In the a good amount of opinions in your posts, I suppose I'm not the only real actual one possessing each of the enjoyment below! keep up The nice get the job done...เกมสล็อต

    ReplyDelete
  19. This comment has been removed by the author.

    ReplyDelete
  20. What a fantabulous submit this has become. By no means considered this type of helpful place up. I am grateful to you personally and be expecting additional range of posts like these. Many thanks a fantastic offer.สล็อต เว็บ ตรง

    ReplyDelete
  21. Hey what a fantastic place up I've come across and believe in me I are already searching out for this equivalent sort of publish for prior a week and scarcely found this. Thanks greatly and may show up For additional postings from you.สล็อต เว็บ ตรง

    ReplyDelete