Package data like software and the stories will flow like wine

By • • Media Party in Buenos Aires

Slides

Show the extracted slide text

Slide 1

Package data like software
and stories will flow like wine
Unsolicited advice from yet another loud American

Slide 2

My name is Ben
Sometimes I go by @palewire

Slide 3

I work here
The @latimes in #DTLA

Slide 4

¡Perdón!

Slide 8

bit.ly/packaged-data
All the slides and links in this talk

Slide 9

ACT ONE
The way we code now is barbaric

Slide 19

Remind me why we have
a data journalist.

Slide 24

ACT TWO
We’re getting better, but we’re still coding alone

Slide 36

ACT THREE
To truly open news, we’ve got to get nerdier

Slide 38

$ pip install requests

Slide 39

$ pip install requests
$ python

Slide 40

$ pip install requests
$ python
> import requests

Slide 41

$ pip install requests
$ python
> import requests
> requests.get(‘http://time.gov’)

Slide 42

$ pip install requests
$ python
> import requests
> requests.get(‘http://time.gov’)
<Response [200]>

Slide 44

dotadiw

Slide 52

$ pip install requests

Slide 53

$ pip install ba-election-results

Slide 54

$ pip install blue-dollar

Slide 60

THURSDAY
Media Fair, Noon - 2 p.m.
Workshops, 3 p.m. - 5 p.m.


FRIDAY
The hallways, just grab me


SATURDAY
Hackathon, all day

Slide 61

THE END
bit.ly/packaged-data
bit.ly/code-rush-2

palewi.re/who-is-ben-welsh/
ben.welsh@gmail.com
@palewire

www.californiacivicdata.org
cacivicdata@gmail.com
@CAcivicdata

Recording

Show the timestamped transcript
  1. Ben la charla de hoy la que esta procera ora is bueno a
  2. In pagetar los atos como sifaran software
  3. But our kick us up on Zurgi de a ser facil accessos a todos
  4. Pero por todado también a in workshop que más traciendo que se chamos california code rush que
  5. just a naciendo un
  6. California civic data coalition is un projecto nuevo don de tratan de ser más facil accesso a
  7. Sierto sata sets que pública el estado california
  8. I believe you can the chile a con the chile a de que esta al workshop. He wanna
  9. Enrado como a no datos sabiertos de california
  10. So que tine por lo a pensago a no que tine con sotros y porque alguieniría esa workshop purpose
  11. En presente dar que bueno cuando lo esto quie un poco aben y entendique en era
  12. I think a person in tresante, but baris que el workshop bastar muy bien porque el
  13. Para bessar pienza pienza estos a todos y pienza projecto muy como eso for liberios a todo una parte
  14. Bueno intendar como afunciado un projecto eso for libero
  15. Pero un muy muy tiempo un parce que muchas días de esta projecto que
  16. Bueno estempezando pero a ralmente tambujo por gente como mucha polenda
  17. Parse que hacer muy enteresante en tener que puedemos verde eso y ralmente
  18. chonofu
  19. Intender in bulucras el workshop entender que lo que estan haciendo ocho para barque
  20. Lo que nosotros puedemos trigar para ca. Me parce que esta bueno juar air me terce y me terce bueno
  21. con los a todos de california. Es un trigamuchos tiques para ar asoluar
  22. ¿A cosas grande se piasos de códio, hay documentación. Me parce que esta bueno
  23. Esta las ocosas esta char la el workshop, muy ralacinales, me parce que despose de barla
  24. char la, lejos interesar y ral workshop
  25. All right, thank you. It's hard if I just take this out, walk around like this. Hello. How is everybody?
  26. Hi
  27. My name's Ben. That's me. I'm working on my Argentine beard. You know, it can't be too thick, right?
  28. It's just got to be a little scratchy. That's how I fit in here. Kind of work on that. I
  29. sometimes go by pale wire online and I'm here visiting you from Los Angeles, California where I work at the Los Angeles Times
  30. which is a daily newspaper and 24-hour website and I work on a team there called the Data Desk and
  31. We're a group of nerds that try to take data and turn it into news where what some people now call data
  32. Journalists other people used to call computer-assisted reporters. Some people might call hacks and or hackers. We are it
  33. We are you we are the people trying to do this stuff in a real American newsroom
  34. and I'm gonna talk to you today about how we do things currently and how I think we as a group can maybe do things better and
  35. The criticism that I've made put out is really self criticism as much as that of anyone else
  36. All right, but before I begin I want to start with an apology
  37. I'm sorry. My Spanish is so bad. I don't dare speak it in front of you. All right, I'm married to a Spanish speaker
  38. She tries to encourage me to use it all the time
  39. but really I'm about good at this as this guy and so I'm just going to avoid it and
  40. I'm sorry, right now
  41. I do live in Los Angeles where many people speak Spanish and most of it that I've picked up
  42. I got eating lunch right at places like pinches tacos or no hodas
  43. Cuban kitchen, but my wife has assured me that these terms are not fit for such an august audience as yourself
  44. So I'm just gonna avoid it
  45. Alright anyway, so if I'm talking too fast or I move too quickly and there's things you want to catch up with you can find all
  46. the slides and the links for the things I talk about at this URL and
  47. You can just hit that in so dub, you know, this is HTTP bitly blah blah blah got it
  48. We ready?
  49. Okay, let's go. I don't got much time. All right one. Okay, so how do we do data journalism today in my opinion?
  50. Not really. Well, right. So where does it begin? It begins oftentimes with the government, right?
  51. They have some valuable data about something that they're up to that we want to get at
  52. Understand write about and probably say something nasty about so that they fix it, right?
  53. So we go to them and we say at least in America
  54. We can say I can write a letter and I can say dear government
  55. Please send me every piece of data that you have on this topic and this is a real letter
  56. I wrote and I send it off in the mail or attached to an email which is really kind of a weird thing a letter attached
  57. An email. Why do we do that?
  58. But we do because I don't know and then am I supposed to get the data, but I don't a long period of time goes by
  59. I have 10 or 20 other emails. These are real ones. There's an extended negotiation
  60. We what are we really gonna get? Am I gonna have to pay for it? How's it gonna happen? What's gonna do?
  61. Right then ultimately Wow one day a CD arrives at my desk or in the mail and inside of it is a crazy government database with
  62. No documentation and dozens of tables and so much data that I can barely understand it
  63. And then I then spend days or weeks
  64. Puzzling and trying to figure out what is in this database and how does it work and can I find anything or understand it?
  65. Or what's going on and in that process?
  66. I might meet someone like this the government employee responsible for creating that database who no one has ever asked how it works
  67. And he's so excited to tell me and we have long conversations and I get to learn a little more about it
  68. And I write really bad computer code
  69. This is real
  70. this is by me that just like tries to pull out that data and move it around and try to find what's going on in
  71. It and I'm still lost and more days and weeks go by and I don't know what's happening
  72. And then my editor very polite and then I'm totally confused and just lost and my editor calls me into her office
  73. And she's like, well, you know, why do you work here again?
  74. And what's really happening and shouldn't you be producing something that we can actually publish at some point at which point I then freak out
  75. I get out my machete and I start hacking at the data and like trying to find something out of it and then in that
  76. process I like create something like maybe a graphic and some stories and this thing we're gonna say and then
  77. Whoo, I get across the finish line and we finish it and it's done and I'm just like, oh my gosh
  78. we finally got there and I never would ever think about or talk about the state again and all that computer code and
  79. all that work and
  80. Everything that I learned in that whole process. Where does it go?
  81. Right there
  82. right and I go back to the beginning and I find a new topic and a new government person to pick on and a whole new
  83. Process to begin again and all that stuff that I learned that was really really valuable
  84. Most of it ultimately gets lost and this is really dumb
  85. right and I do this and many of my colleagues do this and I think people in this room do this and it's kind of part of
  86. our like news culture that
  87. In when it works fine when you're just doing a story in one day and you can throw it away and starting in the next
  88. But when you're doing really complex data work on databases that we revisit again and again
  89. And that everyone uses and we're writing software
  90. It starts to stop making sense
  91. And so, you know
  92. I think we need to as a group start being a little bit less like hacks and a little bit more like hackers, right?
  93. If we're going to be writing software
  94. And you know, it's there is some good news people are getting a lot better at it
  95. Largely, I think thanks to a really great website and technology called git and github
  96. Who here is using git and version control when they write their code a?
  97. Lot of people that's good. It should be everybody right?
  98. It should be unacceptable to work on our field without doing it and more and more people are and like that means as you're doing that
  99. Story and writing that code every little change you write gets saved
  100. so that you can walk back and understand everything you did and it all gets recorded permanently and it never gets lost and
  101. 500 straight SQL queries or six Python scripts that got ran and who knows what order at what time and have totally strange names and
  102. Tucked into some bizarre folder with weird names that have like dates in the file names and all that crap, right?
  103. This really helps you get past that a lot of people are then also using it to publish the data that they use as their
  104. Analysis online, which is really great
  105. And then even some people who are total overachievers are then taking the code that helped create their analysis
  106. And they're putting it on github so people can kind of walk back and see what they're up to
  107. And this is great for accuracy. It helps us not make mistakes by being more careful about the code that we write. It's great for
  108. Transparency because it helps us
  109. Be held accountable and for our readers to see what we're doing and build our credibility and maybe avoid some mistakes that way and know
  110. That we're being watched which is also good
  111. And it's great for reproducibility as well in a scientific sense that someone could take our methodology and our systems and be able to recreate
  112. We're doing no good and people are doing this now and more and more doing it and that's a good thing and we should
  113. Be pretty happy with ourselves, right?
  114. This is like something that the science journals and people in other fields feel like they're behind and journalism is doing good
  115. But there's like one problem if you go to any of these repositories
  116. You know not to pick on anyone in particular
  117. But I looked at a whole bunch of them preparing for this talk where people in our field are posting their code and saying oh
  118. I'm so open and this is the open source and let's build the community
  119. What do you see at each one of those when you check them out you see this?
  120. Right, and if you're not a github user, it might not make sense
  121. But the website with your code has some very simple metrics. How many people have contributed to this code?
  122. How many people are interested in it?
  123. How many have made their own copy and have worked on it?
  124. And what I think we find with almost all the open source that our news people are putting out related to stories is you're seeing
  125. very very little
  126. contributions
  127. Right or community building around these topics?
  128. and
  129. There's some reasons for that right which I think we should we should think about and understand because there's no
  130. What is the difference between the ghost town?
  131. That we put online and the trash can that we had before really not much you get some of those scientific benefits of transparency
  132. and accuracy and
  133. reproducibility
  134. But you don't get any kind of benefits of the code
  135. Contributions and the community building of a true open source project
  136. Right and so you end up being Atlas working alone. Just like you were before
  137. So how do we get better we get even nerdier? I think so. Let's like let's look at
  138. Let's just look at a basic open source project and see how it works. Okay, so let's learn from the nerds
  139. So if I go to github and I look at a thing called requests, is anyone used requests?
  140. This is a python library
  141. Right and it does what does it do so first off I can go to my terminal you could do this right now
  142. And I can use a tool called pip which installs package software, right?
  143. So this is a tool on almost all you know, you know computers or can be installed on most computers
  144. Which will go into the cloud and will install software that's pre-packaged for you to use with one line
  145. So it's very easy to get
  146. Right. I don't have to go and read your Python notebook or find some weird thing or whatever
  147. It's there in a centralized repository and with one command. I can install this piece of software
  148. Then I can jump down into my Python terminal and I can now import it and use it. I'm now writing Python. I
  149. Can I can then tell it to to to go get a URL and return it and that's all this library really does is you say
  150. Hey, here's a URL something on the internet. I want you to go get it
  151. It can do more than this, but this is the basics and then it returns a response and that's really all this thing does
  152. That's not as exciting as an investigative story or something more complex
  153. But guess what? It's what people want and it's what people will collaborate on look at this
  154. 38 million downloads of this one simple library, right?
  155. And why is that because it does one thing and does it well one of my favorite nerd acronyms ready does one thing and
  156. Does it well, which is also known as the eunuch eunuch philosophy and it's really a Lego, right?
  157. It's a component of a work when strung together with ten other open source libraries or other things that another developer
  158. Not you part of your story
  159. But maybe future you right can use to build something that they want to do and the code that we're putting out
  160. Oftentimes related to our stories and work is more like this the Cathedral here in Buenos Aires, right?
  161. So instead of being a Lego, it's a cathedral, right? And what do you do cathedrals you go to worship and that's about it
  162. Right, you don't go there to get work done
  163. Alright, or build anything. So if we're gonna try to do that kind of thing in our field
  164. What are our Legos, right?
  165. What are the components of data journalism and the work that we do that might be possible for us to collaborate on?
  166. And I think there's probably a lot of answers to this question. But one that I think is potentially
  167. Interesting is data sources, right?
  168. So there are out there in our world of journalism certain data sources that we return to again
  169. And again and become almost like beats or topics that we're covering right what's copying the data and that can be
  170. Election results public opinion polls. Sorry for the clipart guys
  171. Crime reports, you know what's happening on Wall Street the census school starts
  172. But you know here in Argentina the blue dollar which is one of my favorite phrases I've learned, right?
  173. There's actually a Twitter account right that just tweets the blue dollar at me every day, which I started following just very cool
  174. And each of these are data sources that are consistently updating have some schedule are structured are gigantic and that
  175. Reporters in different ways to pursue different ends to tell different stories are all mining all the time every day
  176. Right and for every one of those data sources
  177. Almost all the reporters have their own little script to do basically this go to the government get the data
  178. Convert it, you know pull it down and save it and do something with it
  179. Load it into a database and repeat right and this process is known in computer software as ETL
  180. Right because they can make anything boring in IT
  181. ETL stands for extract transform and load and it's a basic component of almost all data
  182. journalism that all that many many news outlets are writing for all these same sources and
  183. Everybody's creating their own different competing pipelines to go get that data. There's the Washington Post. There's the New York Times
  184. There's La Nación, whatever you can you know, let's let's twist this metaphor as far as we can when instead I
  185. Think it's possible for us as a community to get together and build big pipelines that are bigger better
  186. Stronger right and then we all collaborate on so that we don't independently all spend all this time doing the exact same thing doing the boring
  187. Part of our work. Why should we compete at downloading and unzipping database tables, right?
  188. We should be competing at finding and telling stories and doing ambitious analysis and making a difference
  189. So this sort of boring component of our work might be the kind of thing we can organize around I think so instead of say PIP
  190. install requests
  191. Why can't I do PIP install BA election results or PIP install blue dollar and very quickly have basic data
  192. Sources and those basic scripts available to me as package software in the same way
  193. I have all these other components and then those can be well-documented maintained and and and you know grow and get better with time
  194. Right. There are existing projects that do this and I think they're all awesome and going in the right direction
  195. And we should do more like them open states Treasury open elections
  196. The data open knowledge foundation are all kind of moving in this direction
  197. And it's also the inspiration for a project that I'm here to talk about this conference, which is called the California Civic Data Coalition
  198. So, you know
  199. It's a team of the LA Times San Francisco Chronicle Stanford and Center for Investigative Reporting and we're coming together
  200. Around campaign finance data the state of California released the bulk database of all the money in California state politics
  201. It's 76 tables
  202. 650 megabytes 35 million records and about about zero
  203. You know clues on how to use it and no one does really good, California
  204. Campaign finance analysis because it's too hard to unpack and work with this database
  205. So why should we all slave separately and fail to work on it when we can come together and write code that will make it easy
  206. For everyone to do it and broaden the pool that can happen
  207. And so that's what our projects about you can see it on github and if you just google California civic data coalition
  208. And I'm here at this conference to promote the project and to try to get people involved
  209. We have an event. We call the California code rush
  210. I'm here with hundreds of tickets related to this repository that can help make us improvements and push towards our
  211. Push towards our next milestones
  212. I have prizes stickers and t-shirts for people who get things done and there's ways anyone can get involved if you're a hacker looking on
  213. Saturday for something of medium-sized to work on come talk to me
  214. We can find something cool if you're someone who's never made a contribution to an open source project or use github before we have lots
  215. A little itty-bitty ones that'll take you 10 or 15 minutes. I'll walk you through it
  216. You'll learn how all this process works and get your feet wet and in real open source project
  217. Okay, so you can find me later today at the media fair the workshops in the hallways anytime and all day at the hackathon
  218. Alright, and that's basically it
  219. You can find again more everything from this presentation here at packaged data
  220. the more about the code rush here and you know, feel free to email me or bug me at any time about anything and
  221. That's it guys

Downloads

Slides PDF · Recording video · Extracted slide text · Timestamped transcript