- about kicking off teed up kicking off I've already mixed a metaphor we haven't even started yet hey Ben I have a question this is Derek again
- um does it mean anything if you don't so when I have to when I run different commands I have to take the the dollar sign out does that matter or not really yeah the dollar sign is just meant to be
- um actually kind of a hint that you're on the terminal it's not something you actually run okay this is one of these ancient computer programming traditions of like the terminal begins with the dollar sign all right okay just make it
- there and there yes I probably shouldn't even put it in the the little code Snippets
- one of the many things we assume sometimes
- car is your hand raised for help
- uh I'm being helped in the chat I'll take my hand great
- foreign
- I actually have a question as well go ahead uh so I did have all the current
- versions of people B and everything and uh and then I did uh pip and V installed brew and for some reason it like ran all the good and everything and then it
- failed and said they erased everything and now when I say it's MV version uh it just says that there's no directory so I think I messed something up Well we'd
- have to get into the details of it I suspect one of our coaches could maybe just uh start message with you and try to sort it out okay thank you I bet it's not as bad as
- you think I bet you're bet you're there it may be um a shadow copy issue so if you read the error message it'll give you a
- command to directly download from the GitHub instead of making a shallow copy
- okay Andrea are we set we are good to go oh I love it thank you
- so much um coaches so go ahead okay oh yeah so again for anybody who's joined uh we are recording this so there
- will be a video if you miss any parts great and if you don't want your little face to show up in any way you might want to hide your video right
- yeah okay great teachers coaches are we ready to start you think I think so
- all right let's do it good morning everyone welcome maybe somebody could help I'm still trying to help Salma a little bit she's
- still stuck well I'm gonna Spiel here for a little while so you got some time all right there's going to be some painful nerdsplaining before we code and I promise I'll keep it short okay
- all right good morning everyone welcome to First python notebook which is a step-by-step guide to analyzing data
- with python for news and this is a open source course or curriculum that has been developed and taught by a team of
- data journalists kind of Across America to help spread skills that we really see as essential to doing our work and then we want to make sure more people know how to do
- and this is a class that was first taught maybe five or six years ago um uh uh as an outgrowth of a of a project called the California Civic data
- Coalition which was an effort between um some people at the LA Times the University of Missouri Stanford and other California news outlets to
- um to get a grip on the state of California's kind of awful campaign Finance database and so the materials both cover how to do basic data analysis
- in Python and also a lot of the particulars of how to conduct an analysis of money in California politics and so over the next I think probably
- three or four hours depending on our Pace we're going to very carefully and very slowly go through step by step everything it takes to install and use
- the python tools for data analysis and and along the way basically do the uh conducted an
- interview of the data to try to develop a story about money in politics in this case we're going to be looking at the money that was spent in November uh up
- running up to the election of November 2016 when there was a ballot measure in California um that asked whether residents would
- like to legalize the recreational use of marijuana spoiler alert they said yes and we're going to try to use these tools to figure out who gave the most money on each side how much was spent
- and try to come up with some story ideas along the way okay I'm joint my name is Ben Welsh I'm the data and graphics editor at the
- Los Angeles Times where we have a team of reporters and computer programmers who do this type of work all day every day to find develop and and tell stories
- using data I'm joined by two fellow teachers today my colleague Iris Lee and Andrew suozo of propublica and over
- the course of this class I'm going to pass the mic to them to teach some other large sections of it after I kick it off here I'm also joined by a number of coaches distinguished coaches from
- across the country Cheryl Phillips at Stanford uh Joe dramuska at the night lab in Chicago or Evanston and uh Derek Willis in in Maryland right Derek
- correct uh do we have other coaches did I overlook anyone
- we have most of the time zones covered we have most of the times that they're very good and um and so they're here to help if you get stuck you know we're going to go through this slowly
- um but it and everything is entirely fully documented we'll look at that in a second but if you have any trouble along the way there's a a feature in Zoom where you're able to raise your hand and
- ask for help and if you do that one of our coaches will uh will spot it and then kind of reach out and try to help you solve whatever issues you have okay
- um the materials that we're going to go through are all available at a website that is called at www.firstpython
- notebook.org this is basically the script for the entire class and we're going to go through it chapter by chapter line by line together everything
- is fully documented you can copy and paste and ought to copy and paste the commands where appropriate into your terminal into your notebook as we go if if we're going too slowly you could you
- could move it you could you know Skip ahead using this if you'd like um it's all there right this is it and it's going to be available for you after class is over too of course
- um so I would ask everybody to open a web browser and go to www.firstnightfirstpythonokebook.org just so that you have it up and you can
- follow along with us as we progress through the class so I'd ask everybody to do that as a first step I'll give you a second foreign
- we can maybe begin with a little bit of
- an overview of all the things we're going to cover right so we're going to there's there's like just some jargon we should probably get out of the way some computer programming terms about the tools we're using that I'd like to demystify if possible before we begin
- the first is python so python is a computer programming language that allows you to do really all kinds of different things it has these very
- fundamental programming tools that let you read and write to files that let you do mathematics that um could let you make music if you use
- it right that can do really anything right and so it is sort of a fundamental tool of programming that you can use to kind of piece together scripts and
- programs to kind of accomplish whatever you want but it is very generic right and and um and using it alone would require writing a lot of code and using a lot of gnarly things and so we're
- going to use Python and some some friends that make it easier for us to install other tools and use them so that we can write less code and things are a
- little more streamlined and easy to do okay so python is the programming language we're going to use and we need again python version 3.6 or higher it's
- this is an open source tool that's constantly evolving and growing and if you have an older version some of the tools we're going to use in class today won't work right and so for everything
- we're going to use to work you need to have at least python version 3.6 or higher right that's buzzword number one buzzword
- number two is pandas so pandas is a um is a is a python tool so there's there's there were some Bros on Wall Street who wanted to write code to analyze the
- stock market and manage their Investments and make charts and have an easier time doing it in Python and so they created an open source tool called pandas which is
- just a huge hunk of python with all these different shortcuts built in to make life easier that allow you to read in spreadsheets or database tables flip
- them around filter them group them sort them aggregate them right and and and and kind of just do stuff with data and so pandas is a python tool that makes
- working with data easier and we're going to use it for that purpose and if you've ever heard of say for instance r or some of these other statistical tools pandas
- is sort of the Python equivalent of that it's Python's version of a sort of data management and analysis tool and Iris is going to guide you through most of the
- instruction of how to use pandas to do stuff all right our third jargony buzzword is the Jupiter notebook which used to be called
- the python notebook the Jupiter notebook is a web browser interface for writing code people primarily use it to write python but you
- can also use it to write r or a language called Julia or several other different languages and what it is is it's just a nice friendly way to write your code
- line by line and document it kind of as you go and save it and share it and remix it and rerun it and it has kind of
- gradually become the standard for reproducible data analysis in a lot of the hard Sciences in Big Data operations
- in Silicon Valley and is increasingly used by journalists as a tool for um for just like writing your code in a way that it's just easier to deal with right
- and it actually grew out of the desire for scientific papers to be more reproducible so if you're going to publish a physics paper with all this
- math in it for instance right how could anyone else redo your math or check on it or make sure it's accurate so the Jupiter tool is a way in which you can
- sort of write something like a scientific paper that has living breathing code in it that can be run and rerun right and so we're gonna we're gonna install the Jupiter notebook and we're going to write our Python and
- pandas inside of it right and along the way we're going to see a lot of the benefits of doing that too right and then our third or our final
- tool that we're going to install is one called Altair Altair is a python tool for making charts so pandas is a python tool that's really good at like working with data
- but it's mostly good for like flipping spreadsheets around and stuff where all tear can take that data and it can then help you make some cool charts with it without a lot of work and
- this is a it's a it's a pretty new tool but it's pretty powerful as well and it's kind of a nice way to get in there and start sketching with your data and drafting charts and kind of seeing
- what's there and at the end of our class today we're going to take the results of our analysis of money in the marijuana ballot measure race and we're going to make some charts of them which actually will be pretty close to stuff that that
- could potentially be published and Andrea's going to teach that part of the class when we get there right um I think with that we've covered all
- the buzzwords and all the all the the nerdy like terminology I think you'll need to know at a high level there'll be a much more nerdy things to come but that's like the big picture
- um before we sort of kick off and just like dive in and start writing code and installing these tools does anybody have any questions about the kind of Concepts
- foreign
- coaches teachers did I overlook anything
- no okay then we're gonna Dive In so uh let's get going so that that's our
- overview we're going to begin with a little bit of a prologue about some of the tools that we're going to install to do our work today right and use today
- so one thing that everyone needs to progress in this class is a command line interface or a terminal you know back in the days before
- um the Revolutionary appearance of Windows 3.1 you know and the user interfaces that we're all now you know comfortable using our computers with uh
- programmers and people work with computers spent all day on the command line these sort of terminals issuing little short commands back and forth with the computer to like tell it what to do right and that sort of old school
- approach to like directly commanding your computer is something that most computer users never do or don't have to do but the computer programmers do
- pretty much all day because you have a hunk of code that you want to run you want to do something really sort of specific and precise to your computer as
- you're manipulating data or whatever issuing these commands to the command line is still kind of the like um the the way that people do it and so so to become a programmer and do this
- type of work even as as a casual data journalist does require that you become to at least some degree familiar with working on your com your the command
- line of your computer and uh every Pro every operating system has its own kind of way of dealing with this and there's a lot of different programs that can
- operate as the command line but the first thing we want everybody to do is open up the version on their computer and just get ready to start telling the computer what to do and so
- I'm on a Linux computer and so I have my terminal here on the right which I hope you can see right if you're on if you're on an Apple computer you'll want to go
- to the uh the upper right hand corner I think where there's a magnifying glass and you'll want to type in terminal and then that will give you access to your computer's command line if you're if
- you're on Windows we had instructed you to install the windows subsystem for Linux which will create its own terminal inside the uh the program we want to get
- that up and running and I just ask everybody now to open up their terminal and get ready to go and you'll see here this sort of blinking eye of the
- computer as it waits for you to tell it what to do so I'm going to give it a second for everybody to get there
- when you first encountered the terminal it's a little mysterious right there's just this sort of blinking cursor and an
- empty command line and nobody really asking you what to do or giving you any instructions and that is uh because
- these are sort of these systems are are built to um they expect you to come to it knowing what to do and be learning how to use
- the command line just means gradually learning about and practicing and using the hundreds or thousands of little itty bitty commands that can be used to
- interact with it and no one is born knowing these things they were mostly invented like in the late 70s and early 80s by like sweaty nerds at like Bell
- Labs or wherever and they're a little weird and it's not your fault that they're a little weird they just are because of how they were created and because of how computers have evolved
- they are still with us 40 years later and so getting comfortable doing this and learning how to do it is just sort of gradually embracing the process of learning what these odd things are and
- so I'm just going to start off with just a couple of the common ones that that are good to know and that kind of illustrates you know how this how this operation works so there's a command
- that you'll be able to run called PWD again pretty cryptic right so if you type PWD and you hit enter you can see that it's going to print out
- something just below okay PWD it so happens stands for present working directory duh right and it's really it's
- just a command that prints the current location in the file system where you are on the terminal and this illustrates a key point is that whenever you're on your command line you are also at some
- position within the folder structure of your computer from which your commands are operated and so my term when I run
- PWD the terminal is telling me Ben you are currently in your home folder and if I were to open up my uh my finder on my
- computer here next to my terminal you can see that this is the user interface that does the same thing here in my finder it's my home folder for my user palewire and here's all my stuff right
- and so at the terminal currently my um my user on the terminal is currently in this folder here that you see on the
- left and as we move through the class you are navigating through the folder structure on the terminal in the same way that you do when you click around inside these interfaces that I think
- we're all familiar with for moving through folders okay this brings us to another command this one called LS and LS is a command that stands for list
- and it will just list everything that's in the directory and you can see that I have this very sort of verbose output of like crazy things because I'm a
- programmer but what this is really doing is it is listing everything that's in my home directory and so we can see here on the left it says there's a pictures folder there's a Purl five folder hell
- yeah there's a videos folder right and those correspond to the folders that are over here in in the window and so LS is a command that just lists everything in
- your current directory and it is effectively the same as looking at it in the user interface on the left and so for almost anything that you would do in the finder open a folder move into a
- folder create a folder delete a folder rename a folder copy a folder all those things that you would do in the user interface there are equivalents of in
- the terminal they just have like very cryptic ancient computer programming names for the programs that do it with PWD being my first example and LS being
- my other example another one that might be helpful like at this moment is there's one called clear if I type clear it's going to zap
- everything that's in my terminal and just bring stuff back to where we started so it it kind of tidies things up a little bit so those are those are three examples
- um my favorite terminal command is is what I think of as the most existential of all commands there's a command called who am I and if I simply type who am I
- it says back palewire which is my username right and so this is perhaps not a very useful command but it's one that I enjoy um Andrea what's your favorite terminal
- command uh that's a good question
- I think I use LS the most LS it's very useful I think my second favorite one is called cow say
- so there's a command called cow say and then anything you say after that is then broadcast by a cow super useful I love it what's that PG CD is a great command
- that's for changing directories from moving One Directory to another is CD and that's the same as double clicking on a folder in your finder and moving
- into the next directory and we're going to do that together here in a minute Cal State is probably the best now that I think about it come on actually
- there's one where if you mess up I think it's called the where if you mess up typing a command you just type and it it guesses what your
- command was supposed to be very useful I did not know that one okay learn something every day uh all right very cool so we're on this
- command line we're going to use it to create folders move within them install things and just kind of navigate through the class all right everybody's got that
- cool uh as we know everybody needed to have a python version installed before they got into class today and you know the python command on the command line
- is really just the way of invoking the Python program to run code or to do things right so if you do python space and then the name of a file that has a
- hunk of python code in it it's going to run that python code right and the command that I asked everybody to run as part of just preparing for this class is
- python dash dash version which is just a command that asks python what current version of python is available on your computer right and this can verify it's
- there I'm going to use another command just to show off a little bit called witch Python and this will actually show me where python is installed because python itself is just like a program
- that's somewhere on your computer right and by running the which command I'm asking my computer where is python anyway and this actually reveals to me this has been where your your current
- python installation like is at right like inside your file system so Python's one everybody should should could run python
- version if you haven't already just to make sure you've got it installed okay and then our other kind of uh prerequisite and type of thing we need
- to have installed is a little more conceptually complicated but you know as we were as I was talking about earlier with pandas and Jupiter and Altair
- um python alone is not enough to do what we want to do right because the python tools are so rudimentary and basic that to conduct a more complicated data
- analysis or make a chart would require us to write a lot of python right so there's these other python tools that are out there in the world where helpful
- open source programmers have developed and released sort of hunks of code that have all these shortcuts and make life easier for you right and those are
- available in a place called The Python package index which is at Pi Pi p y p i Dot o-r-g and this is kind of uh a phone
- book of sorts of thousands probably hundreds of thousands of little people clumps of python that can be used to do
- different things and so for instance we're so pandas is um an entry in the python package Index this is actually not it sorry
- um that uh that is um basically a offers a way for you to
- download and install this this big chunk of code that has all these shortcuts into it onto your computer so you can begin using pandas yourself right and every programming language has some kind
- of directory like this of helpers and tools that you can install that are out there on the web and every programming language has a tool to help you do that
- so basically it's a tool that helps you install other tools right and this is typically called a package manager right
- and this is a way in which you can basically ask your computer from the command line to go to the phone book or directory of packages grab the one you're looking for and install it on
- your computer right and if you've taken like python scraping classes you've probably learned about tools like requests or beautiful soup right or if you've tried to build a database website
- with python you may have heard of Django or flask these are really helpful python tools that are out there that you would use a package manager to get installed
- on your computer so that you can take advantage of them right and whether you're writing node code or Ruby code or Pearl code hell yeah or whatever else they're going to have their own version
- of a package manager to go get stuff and get it going okay and so Pippi and V is a package manager which everybody had to have installed and so it has within it
- everything you need to do to go get something and get it on your computer right so it's a package manager but it's it is also a second thing
- and the second thing it is is an environment manager which is where the EnV and its name comes from right and this is this is a a more advanced concept but
- um you know these tools that we download and install are all constantly upgrading and constantly changing it could be in conflict with each other or be in
- conflict with the version of python we have installed and so over time if you just use a package manager to install whatever you need on your computer right the different projects that you're
- working on are going to start to crash and have problems because different code is using different versions of programs and things just kind of go wrong and you can also end up in a situation
- too where you might want to take your code and have it run on someone else's computer maybe one of your colleagues right or maybe on a computer server that runs in the cloud and creates a website
- then when and in order to make your code transferable from your computer to another computer it's helpful to have a system that is sort of like a um
- a a contained little space that has everything necessary to execute your code so it has the version of python
- that you're using it has all of the outside packages from Pi Pi that you've downloaded installed that has all of their versions carefully managed and then that way you can have this little
- um like Tupperware like container that has everything necessary to do what you do and so people who get serious about programming will install things in a way
- so that every project that they work on has its own little modular space with everything necessary to be installed and isn't dependent on other things on their
- computer and the tools that do that are called environment managers or sometimes it's called a virtual environment and so you're creating a sort of computer within your computer that has everything
- it needs to go right and pit bnv is also that and so we're going to use the pippy Envy tool to create a little container
- on your computer for everything we're going to need in class right and then we're going to use its package managing functions to install pandas and Jupiter and all the stuff and all tear so that
- they can run our code and execute our analysis inside that environment and this is a hassle right and it is strictly speaking not 100 necessary to
- get code running however it is a very good uh code hygiene I suppose or just the right way to program and we think it's important enough that it should be
- part of kind of a basic class like this okay and so that's what we're going to do with Pip Envy here in one second and we're going
- to get our hands in it and actually install some stuff but before we do that uh as my uh long-winded Spiel is winding up winding down I just want to ask are
- there any questions about the topics or concepts we just covered
- okay now we're going to code spiel's over so we're going to go back
- to our terminal right and again just like I did with python version I can do Pippi and V dash dash version to verify that our buddy Pippi and V is installed
- right so we have python we have Pippi NV we're on our command line it's time to get cooking so I'm going to type clear to just clear out my terminal before we
- go um is is my uh is my terminal text big enough that you
- guys can read it or should I pump it up a little I'll pump it up one step is that better yeah that's better okay thanks uh now um
- and if if uh if I go too fast or if you can't read anything just shout okay so again we can use PWD to figure out where
- we are we're in Ben's home directory or in your home directory or anywhere else before we begin I would like to create a new directory in which I'm going to store all of the
- code that we write as part of this class right and to do that I'm going to use a command that we haven't used before which is called mkdir
- which is just a strange abbreviation for make directory and so this is a tool that will make a new folder in your computer all right and as you can see in the docs
- we advise that you make a directory called code to keep all of your code in I already have a directory called code so just to make the point I'm going to riff and I'm going to make my directory
- called mooc because we're in a mooc right now guys all right and if I go back to my finder in my home directory we'll see that there now is a folder called mooc
- can you see that so because I typed make dear mooc from my home directory boom there's the folder right and so I
- would encourage you to make your own move or make make a code directory for yourself and that will make a new folder where we're going to store stuff and then inside that folder I want to
- create another folder that's just for our class and so I'm going to do mkdir mooc slash and then I'm going to call it
- first hyphen python hyphen notebook right and you'll see that I have a little slash in there so I can create a subdirectory for my current position
- right now if I go back to my finder and I look in the mooc directory we can see that there's now is a folder
- called first python notebook which we just created and is empty all right so everybody should do that make those folders
- we're going to use another command that one of our students mentioned earlier to move into the directory that we just
- created and this command is called CD which is short for change directory and so I'm going to type CD m o o mooc I'm
- going to try to go into that director just created and here we can cover one of the most important terminal tricks is if you type the first few letters of
- something and then you hit the Tab Key it will auto complete what you've just been typing see how I did that moo tab it finished typing it for me isn't that
- nice and in this case since there's only one subdirectory I can hit tab again and it types everything for me love it so I'm going to do CD mooc slash first
- python notebook hit enter and now you'll see if I type PWD again it will say Ben you're now inside a
- different directory you're now inside first python notebook and so if that is the same as being inside this folder right here in my
- finder so we now have positioned our terminal inside a folder we've made specifically for uh this class and this
- is where we're going to begin using our tools to work if you have trouble with any of this feel free to raise your hand and one of our coaches can message you all right
- now we're going to use pip bnv to install our first program and we're going to install a program that is kind of irrelevant and not that useful just
- to kind of get some practice and understand the fundamentals of what's going on so the I'm going to type Pippi and V and then install which is
- going to ask pipian V to install something and then whatever I type after install is the name of the program that
- on the python package index the Pippi and V is going to go get and try to install right and so I'm going to install for the purposes of this demonstration a program called yoke 3K
- and yoke 3K is um it's just a very simple python package that it will print out every other package that you have installed so
- it's a tool for listing all the stuff that is installed currently within your environment right and so it's it's it's it's not something we're to use for our analysis but it just helps us kind of
- get the concepts oh that's wrong sorry Pippi and V install yoke 3K is what I'm going to type in and I know it's yoke 3K because that's its name in the package
- Index right and here's the documentation for how it works here's here's the command that will list all of our python packages which we're going to try in a minute but first we need to install it
- so I'm going to do pip EnV space install space yoke 3K hit it now Pippi and V is going on it's first
- off it's creating a new virtual environment that we talked about earlier so that they're inside our current folder there was no existing virtual
- environment or little modular Tupperware to stop to store all the stuff we're working in because it's the first time we've run the command here so the first thing that pit BNB did is it installed
- it and it actually installs an entirely separate version of python inside this that will run separately from your systems python installation
- after it's done that and here's the location of that on the on the computer system it shows here that this virtual environment is sort of hidden away in my computer at this location
- and then it goes off to the python package index it looks up yoke 3K in the phone book and it installs it so this is
- successful right and then um so it's success yes and then at the bottom it gives me a little hint here it says if you want to run a command inside
- of your virtual environment do it with Pip EnV run and that's the next trick we're going to learn is how after you've installed a python package into your environment how
- to execute it right um and we're going to do that to run the Yoke command that we saw in the documentation so in the documentation
- yoke said if you want to list all of your installed python packages you should run a command called yoke Dash L and so I'm going to run clear to clear
- out my terminal and then I'm going to do pip EnV run and then I'm going to run that command yoke Dash l and it'll take a second but you can see
- it then prints out here's all the stuff that's installed inside your virtual environment Ben so here these are basic python tools that are used by the
- package manager that are kind of always there and then here's the new tool that we installed ourself yoke 3K right and so we've now used pit BNB to create our
- virtual environment we've installed our first kind of demonstration package and then we've been able to then run code using that demonstration package
- okay if we look inside our directory if I run LS and hit enter
- we'll now see that there are two new files that are in our directory that were not there before there's the PIP file and the PIP file lock right I'm going to
- open the PIP file real quick and say well what the heck is this that just showed up on our computer and this is Pippi and B's way of sort of
- recording what it's doing so let's say we want to take what's inside your folder and we want to run it on your colleague's computer or on a computer server this file contains the
- instructions that the pipi and V on the other computer needs to install everything to do um and to do do what it needs to do and run your code and so we can see here
- that it includes a list of installed packages like yoke 3K which we just installed it has my version of python Etc
- for what we're doing today there's no reason you need to look at this file or do anything with it but it's good to know that this kind of exists this is what's known as a manifest file and it's
- just the list of everything you need to run your code right and the lock is sort of a more sophisticated version of this file that is intended for the computer to read not for us the humans so when
- you add or subtract different programs the human would do it in the pivot file and then the PIP file lock will be updated by Pippi NV so that the computer can keep up to speed with what you're
- doing okay okay um and with that we've covered the basics of Pipi and V and we're now ready
- to use it to install our first analysis tool the Jupiter notebook and begin work doing math and doing data inside of our
- browser before we do that I would just pause for a second and ask are there any questions about Pippi and V or these terminal Maneuvers we've been doing
- and there's one in that just showed up in the chat that's probably it happens to a bunch of folks uh I know it's happened to me essentially where your
- your pip file and your pip block pip block file uh end up in your main directory not in the first python notebook directory
- um is there like do you have a recommended solution for dealing with that spec you know I'm guessing a little but I suspect that happened because people
- ran the commands uh without having changed a directory into the code directory right so the picture is made at whatever location where you
- issue the commands right so if you're in your home directory and you do Pipi and V install there it's going to create a
- new virtual environment there and it's going to put the PIP file there right and so um you could proceed with the class and work that way there's nothing that would stop it from working it's just that you
- would have sort of all the stuff living at the root of your computer you know the reason I made these directories and moved into them was just to kind of have a little corner of my computer where
- this stuff was happening um but it wouldn't be necessary so if you wanted to avoid it I would say uh look back at uh the the the chapter two
- make sure that you make directory the code and first python notebook directories and then change into those directories with CD before you run pit
- BNB and if you did that now again I think you would probably be all right any other questions
- um I'm stuck in my home folder and I'm
- trying to move do I just hit CD code yes but the code directory has to have been made right which is what make directory does it creates the directory
- and then you use CD to move into it after you've done that the CD code just gives me no such file directory even though it exists uh then
- I'm guessing there's a typo right so one thing I like to do if I get confused is open my finder and just look so like I open my finder here and I look
- there's the code directory I know it exists so you might want to take a look at your finder make sure it's there you can type PWD to then kind
- of Orient yourself where am I in the terminal this happens to me all the time sometimes I think I'm in one place in my terminal but I'm in another and I get a little turned around and using PWD LS CD
- are the tools to help you kind of sort that out I think one of our coaches could probably help you with it if you're if you're turned around thanks
- okay great we can move on so now we're going to head into chapter two of the course which is called hello notebook
- and in this chapter we're going to install the jupyter notebook tool we discussed earlier which is a browser-based system for writing reproducible data analysis
- um many newsrooms are using this now and some are even publishing their notebooks which have all the code that execute their data analysis on open source publishing sites like GitHub so that you
- can read and run the analysis yourself you know for instance you may remember some of the machine bias series by propublica
- um or um the the um the tennis racket by BuzzFeed where they try to figure out who is throwing tennis matches using data and all the code that actually generated those stories is
- available in Jupiter notebooks on GitHub and I have some links here and you can go and read those and see them yourself it's not just journals to do this scientists Scholars uh Wall Street Bros
- as discussed and many others uh do this as well and so but before that you can write your own Jupiter notebook you have to install it right because it's just
- one of these hunks of python code that's out there waiting for you so we're already in at this point our first python notebook folder which I can verify by typing PWD which has been okay
- I'm there and so now I'm going to use pit bnv for real to install something I actually need and I'm going to install I'm going to do Pit bnv install
- Jupiter lab and the reason it's Jupiter lab is because that is how it's been defined on Pi Pi by the creators of Jupiter who
- were at Berkeley and uh the University of Colorado and Bloomberg and all these places around the world where there's these nerds who come together to write the software they decided to call the
- tool Jupiter lab and and put it into Pi Pi this way and so if that's the term that we need to use to install it from Pi Pi so we're going to do pip EnV
- install Jupiter lab and hit enter now because this is a bigger hunk of code than yoke 3K and it has to install
- more stuff it's just going to take more time and so we're going to hit this and it's going to depend on your internet speed and your computer speed and other
- factors but it might take even a minute or more for this to finish and so I'm just going to kind of let this chill and you guys should too and just give everybody's computer a chance to get
- this done and and I'm going to take this opportunity need to jump in and tout the fact that one of the newsrooms that really publishes a lot of their
- notebooks is the LA Times which Ben didn't didn't go into but um I actually point my students to the notebooks that the LA Times Daily desk publishes all the time it's really
- useful uh you walk through that code and it's just great practice yeah so here on GitHub is a list of all of the notebooks we've published here's
- one that analyzed uh the the speakers and Star Wars movies to show sort of the gender disparity there uh here's where we scraped Deadspin for some strange
- reason I can barely remember here's one that Iris one of our instructors wrote that analyzed the rising number of uh of women being killed in Los Angeles County
- in domestic violence episodes um and they live within GitHub which is sort of a social network for code so here is here is the python code that
- Iris wrote to do that story um inside the new The Notebook interface here's pandas here's Altair right and we're gonna learn how to do something quite like this here in a minute to
- analyze money in California politics
- you can see that mine is still going locking step often takes a while that's updating that pitfile lack I was talking
- about is what that is and now it's finished success I love it I love it when they have like little colors and emojis and things that make
- me feel good that's how you know it's a newer program it's been written by millennials right and mine is finished
- so it might take other folks a little while to finish so I'm going to give it a second but what I'm going to do just to illustrate the point again is I'm going to run yoke one more time you do
- not need to do this I'm just kind of making a point but if I run that yoke Dash L again with Pippi and V and it lists all the install packages look at that remember there used to just be
- three or four things now there's like 20 or 30 things because these are all the other packages that got installed on our computer by Jupiter lab because Jupiter lab is
- itself one program but it then calls on many other programs to do what it does right because it you know if it wants to
- run python code or it wants to have sort of uh Crypt use cryptography to secure its website or it wants to send files to the trash of your computer rather than
- write their own tool for doing that it's going to install other tools that other people have written on Pi Pi to help them do what they do right and so installing Jupiter lab is actually
- installing dozens of other python programs that we see here right and you can see they often have really terrible names like sniffio that's
- pretty bad pickle share how about that one you know okay I'm gonna type uh hold on
- I'm going to type clear to clear out my terminal Jupiter lab install for everyone uh no not yet
- okay so there's one issue which is you might be waiting and then this is also one of those moments where something could go wrong right like if your Pippi
- and V isn't configured correctly or you have some underlying issue with your computer this is a moment where it might like bark at us and say there's a problem so if somebody if you're not if
- if you actually have an error message if something has gone wrong this would be a moment to raise your hand and our coaches can reach out to you and we can try to figure out what's going on if
- you're just waiting we could just wait another 20 30 seconds and I think everybody will catch up
- foreign
- ER lab is installed we're ready to run it right it's installed inside our virtual environment so in order to start
- it up and get it going on our computer we need to begin by typing Pippi and V run just like we did with yoke because again Pippi and V run is our way of running something inside of this little
- self-contained environment that we've created right and inside of Pippi and V run there's a command that actually basically is the start button for
- Jupiter lab this is the command that says okay I'm ready to spin up the notebook and start writing code inside of it and again this is just like an Arcane thing that was developed by these
- nerds and uh you just have to memorize it or learn it to go and so I learned this a long time ago from their documentation so I know that to start it
- I need to run the command Jupiter lab right um you know it's Jupiter like the planet but it's py like python get it guys
- some people call it Jupiter which is actually kind of fun because it's so nerdy but I think it's meant to be
- pronounced Jupiter don't ever call it you pyter never again I'll never do anything don't do that none of us should do that
- I don't know I kind of like it but I want for you for you Derek I won't all right so it's Pippi and V run which
- is saying run it inside the inside the environment and then the command I want to run is Jupiter space lab okay I'm gonna hit enter and that is going to
- sort of take over my terminal and like fire up a little little baby internet server inside my computer that runs Jupiter and depending on how your
- computer is configured it might even open a web browser for you automatically like you can see here on my computer Firefox just opened up hooray and here I am inside the Jupiter
- lab editing interface right now if your computer doesn't automatically open a browser that's okay not a big deal right
- what you can do is you can look inside the terminal message that's fired up and you can say to access the server it says either open this file or open these URLs that are at
- the local host or sort of your computer's position um and you could just say right click on that and say open link you could copy
- and paste it and paste it into a browser tab like you know if I say copy and then I'd paste this into my browser tab we
- can see that Jupiter lab is I think gonna fire up there too right there it is so I'm going to close Firefox and keep it in this other one right and if you've done this you now
- have successfully installed Jupiter and pretty much all the gnarly computer programming stuff or most of it is behind us and you're in good shape
- very good okay so I'm going to move that over here to the left we have Jupiter lab up and running and
- we're now ready to make our first python notebook right which is the interface like the one we saw with irises and there's a launcher right here in the center that says make a notebook in
- python3 and so I'm just going to click that button and that's going to make a new python notebook for here me on my computer and here is a new blinking Terminal A little
- cell of code where again I'm being asked to do something right we can see here on the left there's a file drawer or browser
- within jupyter to let you look at your files so we can see hey there's our pit file right and there's our pitblock file and here's a new file which is called Untitled IPI and B because the I Pi NB
- is an IPython notebook which is what Jupiter used to be called so that's the file extension and then the file is Untitled so it's
- named Untitled but I could just for fun you don't have to do this click the rename button and then I'm going to name I'm going to name it um G I don't know
- what should we call it naming things is hard I'm going to call Mine Derek right and so I now have derek.i pi and B if I were to go into my file folders go
- to mooc go to First python notebook we can see that there's my file Derek probably not a good name let me rename it I'm going to rename it
- notebook okay and you're able to interact with it here within the browser the same way you would on the file system okay now
- these cells that we see I'm going to hide my drawer then by hitting this little folder button just to give myself a little more space and I'm going to pump up the size of this so you guys can
- read my code a little better okay now the way that that notebooks work is um they allow you to segment the code that you write into little bite-sized
- chunks that are called cells and those are these little boxes we see just a single box here on the page with the blinking cursor that's waiting and
- unlike the terminal these cells are expecting you to write python code right they're not expecting us to do LS or PWD or any of those weird
- terminal commands they want us to write python which has its own syntax its own rules its own system right and so we're going to start with a very simple piece
- of python code which is just two plus two because when you're writing python you can take a number you can put a plus sign between it and guess what it will
- add the two things together so I guess everybody type two plus two and then you can see above it here there is a little button that's a play button like on a vcr and it will execute
- the currently selected cell of code which this little blue thing on the right indicates so if I click into my cell write some Python and push play
- you'll see that it then prints out beneath the cell the result of that line of code right and as I think we all know two plus two equals four and so now
- congratulations if you've never done it before you have written python code guys you've done it right your first line of python two plus two
- I see we have a few folks who've raised their hands and so um our coaches can maybe help them I'm hoping that this is broadly worked for everybody though have
- I gone too quickly is there any questions yeah I uh click two plus two uh and when I hit play I'm don't get the
- result I'm just taking to the next uh cell and I just noticed that you know your two your plus is highlighted in
- different color mine has not did I not save something or well it could be that um your cell is not a code cell so see how I saw I've selected the cell here
- and do you see there's this box here that says code there's different types of cells cells can be code cells or they can be what are called markdown cells which are the
- cells where you write documentation or you say this line of code is about to do this and you write little notes and you put hyperlinks in and you do stuff and so a notebook is almost always a mixture
- of code cells that run code and markdown cells that document the code right and so my first guess would be is that your cell kind of accidentally got turned
- into a different type of cell and then it's not being executed as python code am I right yeah I am on code though
- um well you could always there's a plus button here that will insert a new cell see how I did that look at me add these cells and there's a cut button that will remove cells and so if your cells are a
- little jacked up I would say just cut them all out add a new one click into it make sure it still says code try two plus two
- hit play that should do it would be my guess now there's hotkeys for doing all these
- things so if I do two plus two and hit play two plus three it now shows five two plus four it now shows six but you know you might get tired of hitting the play button so you could do if you hit
- shift enter it will run it that's a hotkey that's very hopeful on Jupiter if you just change there do shift enter and it re-runs the code right
- so that's a you know demonstration cell not especially useful you know what but you know so let's get into something
- that is still simple but it takes us a step further so if you're going to have all these cells which run independently of each other within your notebook you need a way to pass information from one
- cell to the next cell right so that you're able to create a chain of things that are useful and you know the the fundamental tool for doing that in all programming languages is the variable
- right a variable is a named sort of thing that stores information that can be reused in other
- pieces of code right and so you save some information into a variable and then you do stuff with it right and the information you save can be numbers they
- can be strings of characters they could be lists of numbers lists of characters a mixed list of both they can be what are called dictionaries where there's like a lookup of a name that keys to a
- value there's all sorts of different data types that can be stored inside variables depending on what you're trying to do so let's write our first
- variable which will be very simple I'm just going to call it number right so I'm just going to type number equals and then I'm going to say 2.
- and then hit the play button and so now you'll see that nothing got printed out because the result of the cell wasn't to print something it was to save the
- number two into a variable called number right and so if in my next cell I uh just type number and hit play you'll see that it
- prints out too right because this value has been saved in this cell right now if I were to make another cell
- an hour to do number Plus three and hit play see that it comes out as five because the variable
- was added to had three added to it and then the result was printed out right and this is just a very rudimentary example of python within the notebook
- now um the what the this this will seem very simple at first but what makes the notebook powerful is you can go back and change one of the cells let's say I
- change two to four right I re-run that cell and the subsequent cells and you can see now that number
- plus three equals seven and the changes that I made higher up in the notebook when everything was rerun rippled down throughout the code so that the changes
- I made sort of flowed down out and through the system again right foreign
- your Jupiter notebook is really introducing the python tools that are
- going to help you do what you want to do into the notebook and then writing rearranging rewriting re-running these cells to gradually get them to sort of
- get where you want to go as part of your analysis right and the notebook is your sort of friend and your tool for sort of doing all that
- all right with that we've reached the end of chapter two hello notebook we're
- ready to introduce some of our python friends to begin looking at how you would do more sophisticated data analysis within this environment does anybody have any questions about the
- notebook at this point
- I have a python question sure um is there an an option to liken JavaScript where
- there's a const to assign a value to a variable that does not change or is it just kind of like a quick assignment the way that you
- did it here by default in Python the assignments are sort of you know editable they're not permanently saved assignments so you're
- once you create a variable you're able to redefine it you know that's kind of Python's default behavior and this is one of the things that can vary slightly between different computer programming
- languages where most languages are mostly the same in terms of concept but there are little things about like or there are sort of fundamental qualities about languages that can vary and one of
- those is whether variables once you create them are sort of permanently saved as they're created and the other is what another is whether a variable
- has to be typed or when you create the variable do you have to say this is an integer this is a string this is a list like in a sort of very strict sense right and python is loose in that way a
- python variable once it's created can be redefined any number of times and it can really be any type of information
- um and that's I think the people who made python wanted it to be easy and like kind of fluid and that's why they did it that way other programming languages had implemented it in a more strict fashion right now if you want to
- write strict python which sometimes you do like let's say you're scraping data and you want it to be very strict about what you save into the database and if anything weird comes in the scraper you
- want it to throw an error right you know there are add-ons to python that allow you to introduce those types of rules to
- how the things work but by default they don't do it gotcha thank you [Music]
- your your audio was a little warped for me did anyone hear the question
- you could try it in the chat Maybe
- okay I see some people reporting some python errors I think our coaches can probably reach out and help with you it probably just has to do with your Jupiter starting up at some of those issues of just getting things kind of
- booting from your terminal we're working in our notebook is there
- anything that's happening in the terminal yes so if we reopen our terminal again underneath the notebook you can see there's all this gobbledygook right do you see that on
- yours yes yeah so this is basically when you started the The Notebook it kind of took
- over your terminal so your terminal is no longer yours it's been kind of possessed it's been you know it's now run by Jupiter and the the web server that is the site is sort of logging out
- what it's doing it's just giving you kind of reports on what's Happening and you can see that like when I renamed the file and the file got saved it sort of
- logged that to the terminal and I think it has an automatic save where like every however many seconds it saves the file for you as a courtesy and every time it did that it was kind of logging
- what's going on here right oh Derek I can see that it says notebook is not trusted
- the notebooks that I created that's not an issue that's just like a security feature of the Jupiter notebook that is trying to like sort of use cryptographic techniques to sort of like ensure that
- there's not a hacker inside your website like screwing around with it and I think that those features because the Jupiter notebooks we're running it on our own personal computer but they're the
- software is actually capable of being run on a server in the cloud where you just go to like a private website and write the notebooks so for instance this
- is 100 true if you've worked at Netflix for instance and you were on the data science team at the East Hollywood office or in Los Gatos and your job was
- to analyze um you know how many seconds into the stand-up comedy special before people bail you know what I mean or like um to write machine learning to determine if
- you could create a new genre of Chinese Noir or something you know what I mean they're using Jupiter notebooks to analyze the marketing and user data or
- or whatever but there's so much data and their organization is so sophisticated that people don't install and run the code on their own computers the Jupiter notebook is instead hosted in like a
- private website that's run by a whole security team and the data analyst goes into some private website and they see 100 the same Jupiter like interface that
- you're seeing but it's instead hosted inside this website and in those cases they might not want to get hacked or whatever and so there's these features
- to sort of secure the The Notebook but on our private computer it's none of it's on the web it doesn't matter
- make sense thanks hey Ben quick question um was that that when you like ran the
- entire notebook again was that the like fast forward button the like restart kernel then rerun entire notebook yeah so um I didn't quite do that but there are
- some useful things that are worth pointing out that are like that so like the play button will just run an individual cell right right but they're a really common case for us a journalism is like well I changed something up here in the notebook and then I want
- everything below it to like jam out you know what I mean based on like the change and we're going to do that in class is like there's there's little tools like in the Run menu you can say
- um run all cells and so that'll just do everything from the top right and you can do um restart kernel and run all cells is
- sort of like flushing everything and running it all again because if you run all the cells let's say you created a variable but then deleted it and then renamed it and used it again that's all
- kind of like living in the memory of the computer and sometimes things can kind of get crossed up and so if you restart the kernel it's just like flush all the python and just start over so that
- nothing can go wrong right and so if you if you want to be very you know so oftentimes when I finish a notebook I'll do that to make sure there's like everything still runs after I've like
- edited things and deleted things and goofed around a lot you know um but if you worked at a big data science company you might not want to do that because cell number six read in 200
- million records and if you restart it you'll have to do that over again you know yeah yeah okay thank you so much which is why they don't have a hotkey for it I'm like in a
- fight with them if you go on GitHub on Jupiter lab I'm like I want a hotkey to restart and run all the cells and they're like that's too dangerous we can't allow it I'm obsessed with the hotkeys too but I know so you should go
- and thumbs up my little feature request you know um and there's a bunch of other things like that that allow you to cut cells and
- paste cells move them up and down uh you can show little sidebars you can collapse them there's like a lot of stuff you can do that's kind of hidden in these six
- okay so for good there we're ready to move on to chapter three and chapter three is we're going to begin to write we're gonna begin to use one of our python tools to write the code so rather
- than just goof around with variables and numbers we're going to use pandas and we're going to introduce pandas to our notebook so that we can
- uh begin to take advantage of the many shortcuts and special features it has for managing data and and calculating statistics right and so to do that we
- have to go back to our friend Pipi and V and install pandas because just like Jupiter lab it's not on our computer by default right it's something we have to install and Pippin B is our friend to do
- that right now as we pointed out before our terminal has been taken over right so if we return to our terminal we see
- that Jupiter lab is in here making noise and if I try to type like PWD and hit enter nothing happens because I'm no longer on my terminal and so to get back
- to the terminal to run Pippi and V to install our other tool we have to either reclaim our terminal from Jupiter lab or
- we have to open a second terminal right that will operate in tandem with this and either of those methods are okay and I kind of leave it to you to pursue what
- you want but just to illustrate how you start and stop the Jupiter lab server I'm going to do it that way myself and so because it's not immediately apparent here at my terminal and this will vary a
- little depending on your operating system if I hit control C it will say it will then prompt me do you want to shut down this Jupiter server I'll hit Y and
- enter and then it should um I waited too long it should then shut down Jupiter lab and now I've returned
- to my terminal where I can type PWD right however my Jupiter lab is dead the thing in my browser has now been terminated and won't run anymore right
- so I want to close my browser too and so if you want to get some practice at this I would do control C or if you're on an Apple it might be a command C is that
- right guys okay thank you and then you just want to say yes and you want to stop it right so and
- um if you're if you feel you're already advanced and you just want to open a second terminal window instead that's fine too but you know this is good practice I think so Ctrl C we'll close
- it down I'm then going to type clear to clear out my terminal again I'm going to check where I am with PWD I'm in first python notebook which
- is where my virtual environment is so I need to be here to do the install right and now I'm going to install pans with Pippi and me install canvas
- okay and that's going to go again to Pi Pi it's going to look up pandas in the phone book it's going to take the code off the internet download it install it
- on my computer and get it ready to go and pandas is even more complicated than Jupiter and so it's going to take a while for this to work right and it's also possible that after
- waiting there's some configuration issue on your computer with installing pandas now one thing that was in our prerequisites was that on your Apple computer you need
- to have xcode installed right and xcode is the computer programmer add-on to an Apple computer right and it basically
- includes some tools that let the computer do nerdy things that aren't worth getting into and it may be necessary to have xcode installed for
- pandas to get on your computer is that right Andrea I think so yeah so if you have I could
- probably yeah so if you have an error with compiling the code or something like that it's likely going to be due to having not having xcode installed which is easily done if you're an admin User
- it's just an extra step and uh and I did we have that documented gosh
- we should if we don't I feel like we may have I may have overlooked that in my documentation huh guys
- I feel like we used to have it but I can find that line of code in our
- internal LA Times documentation for folks in case this comes up as an issue welcome to the LA Times private documentation site
- very exciting stuff [Music] we have a little how to set up your Macbook guide
- it just has a bunch of annoying commands like this because it's just too hard to do all this crap and so there is I believe here
- a command that will install xcode that if anyone has this issue I will post into our chat
- I suspect if you have Homebrew installed you have xcode installed would be my guess but
- okay so you can see here on the left in my terminal that pandas has been successfully installed you know if I were to look at my pit file right which which includes
- everything I've tried to install we can see that it now includes pandas right because these are the three things that Ben has used pipnv to install and so that has been logged in Ben's pitfile
- okay that's all done you know if I were to just again do yolk just for fun you would see all the underlying
- dependencies that get installed as well and we can see wow we've really installed a lot of stuff right now on the computer there's pandas right there all right
- okay again depending on your internet speed your computer speed this will take a little longer um and if you have any errors or buds that come up just raise your hand and our coaches can help while we're waiting
- does anybody have any questions about uh pandas or any of this I would uh Ben I would just say like uh for folks uh and we'll we'll uh coaches
- will be around to get to folks when you have issues or when you have errors like that's a super common thing the first thing you should tell us is not I have an error but here's my error
- um just you know the the specifics will matter especially when we're doing this remotely so uh uh so as specific as you
- can be that helps us helps us solve and even just copying and pasting what you see in your terminal would be a good start right Derek yep
- okay we'll wait for people to get a little help there um
- we have pandas installed and I'm gonna and that means we're going to move into chapter three hello panties
- we're almost done with all the painful installment guys this is the last one for a while
- okay while our coaches are helping let's discuss how we might get back into our
- Jupiter lab environment so that we can take advantage of pandas uh oh I have to uh who here in the class
- remembers what we need to do to start up our Jupiter lab and can share it with the group
- was that the hip Envy run Jupiter lab that's exactly right you got it and so
- if we want to restart our Jupiter lab which we killed a minute ago we have to use Pippi and V run which can execute a command inside our virtual environment and then the command that starts the
- Jupiter lab is Jupiter space lab you nailed it and so to get back into our Jupiter lab where pandas will now be available for our use after we've
- installed it we need to run this command another time okay and so I'm going to hit do Pipi and B run Jupiter layup
- hit enter same as before and you can see my Firefox is booted up
- but because I want to stick in the same browser as my other thing which is in Chrome I'm going to copy and paste the link into a brave tab
- and that will start up for me you can see here in my case The Notebook
- opened right back up where it began again and again just to kind of make the point if I were to advance down to number plus three click on it and hit play it's going to throw an error and
- say look number is not defined does anybody know why that is if you haven't rerun your notebook
- exactly I haven't rerun The Notebook even though all the code is still in here because we killed the Jupiter server and restarted it it all needs to
- be run again from the top right and so in order for number plus three to work I need to run all the code above it which I'm doing with my hotkeys and then voila
- it does so everybody who has their pandas installed and as jupyter backup should just try to rerun their cells again
- and get it there I feel like we have a few people who have uh bugs the coaches are helping them with so I'm going to stall for a second and I'm just going to do a little
- curlicue on a feature that is not essential but it's just a useful uh Jupiter thing that we could spend some time on you know a minute ago we were
- talking about how there's the cells could be code cells or they could be markdown cells right so for instance if we look again at iris's notebook we can
- see that when she was finished with it it wasn't just python code that was all chained together she also had written some documentation that explained what
- she was up to right so we can say this is the women homicides analysis by Iris Lee she has like a summary of her findings she links to the story and then
- throughout the python code you can see she's created section headings she's like described what she's doing and she's done that by mixing in these markdown cells which is kind of a form
- of HTML with the code so into kind of a little um you know a little white paper a little thing of Art and we can do that ourselves by taking one of our cells and
- converting what it is so I'm going to click on my first two plus eight cell and you see where it says code there I'm going to pull down that I'm going to hit
- markdown and that now you'll see that the little thing on the left is gone and the cell looks a little different it's not formatted if I hit the play button it literally just says two plus eight
- and that's because the notebook is now interpreting the cell as a markdown cell or a documentation cell and not python code and so if I can come in here and I can start writing like a headline for my
- notebook so my notebook is called first python notebook by Ben Welch right if I do that and run it wow it now has like a little bit of text
- there right and if I click into it again and now this is where markdown comes in markdown has ways of making headlines a single hash before a line creates a
- headline and so I could say boom hash space first python notebook play right now I have a headline you know I've got my little blog that has my profile page on it so
- I'm going to go get that URL that's how I used to look before kovid and I grab that link and then I go in here and then markdown has a way of making
- hyperlinks which you can look up just how markdown works and you can learn it you put a little bracket around the text you do a parenthesis and then you get a little hyperlink and so now the top of
- my notebook has a headline and it has my name and a little hyperlink and it's by mixing and matching these coded markdown cells that you can kind of document what
- you're doing we're not going to get into a lot of that in class our class is really focus more on the coding part of it but this this this markdown feature is just part of the art of making a sort of nice
- notebook and I find it quite helpful to put like a little line of text around above everything I do like you know to find number variable you know like right
- there above what I do and then it's like you know I could say prince number variable right and now I can sort of document what I'm doing as I go for my own future reference
- okay that was my stalling routine coaches do we need more time or are we ready to advance
- I'm still stuck we've got a yeah we've got a couple people who like the the
- pandas install just isn't working right and it's not I'm not clear like there's two there's two like Avenues here two
- types of failures we're getting one is it just like it the Locking doesn't complete uh and then the other one there's a very specific one that at least a couple people have run into that is requiring a
- [Music] um uh what's the way to say this essentially like it's it's
- raising a very specific error uh to install numpy uh that references pep 517 that I've
- never heard of before but I've Googled and it seems like it's a real thing but like the the instructions for getting out of it aren't have not yet yielded any success I don't know if you run into
- this but that bug I haven't seen now if your issue is the lock file like the Locking can take a long time and if your computer is otherwise taxed with other things it can take even longer and it
- can be very frustrating the lock file itself is in essential to installing the tool it's really just about creating the record in the pipfile.lock which is not
- necessary and so if you're having trouble with the Locking what I would advise is PIP B and B install pandas and then dash dash skip Dash lock and that
- will install the tool without doing the lock right which can just lead to it being more quick and avoiding whatever your bug you're having for this time so I'm going to paste that into the chat and I would say if you're having trouble
- with the lock taking forever or crashing I would say just try it with that variation for Thing One Thing Two you sounds like something I haven't heard before I would bet it it has to do it
- might be a 3.9 thing that's just new to us um is there not a way to skip that verification
- it just doesn't complete the install but it fails that's what that yeah exactly and do you have xcode installed
- yep okay because my first thought was maybe it was the component that was my thought too but it doesn't solve so
- um that's a new one to me these are on Apple Computers yep oh I actually might know this
- um so the new Apple M1 chip has been causing problems with certain open source software yeah this is actually isn't an M1 thing this is like a 2017 MacBook Pro
- okay then everybody yeah you're right you're right if you're on a new M1 you could you could run into this but like uh but if you're like uh but yeah it's yeah that's the one
- then right there yeah essentially if you're on Big Sewer and and M1 in particular you could have
- an issue here and I've for folks who have run into this I've tried this without you know without using the the command that says don't use pep 517 and still no lock
- well this is one I have not seen before the zoology of bugs has expanded yet again
- yeah I I mean I think like I think like the thing to do is just not to run this on three nine for most people like I I like I I my suspect
- suspicion is like if we could essentially like have rolled back the clock and be like let's all make sure we start on either three seven or three eight we probably wouldn't run into this
- well if that is the issue I think the route to correcting your python version is using a tool called Pi EnV which is probably too much to get into in this
- class but is a really great python tool so in the way Pippi and V is a tool that lets us manage different versions of python tools Pi EnV is a tool that lets
- you manage different versions of python because over time it as you code more just using the default installation of python or trying to change the versions yourself become such a headache as we're
- encountering here that it's useful to have a tool that will let you install multiple versions of python and switch between them which you can see is a tool
- that I had installed when I did my which python it showed that my python is installed via Pi EnV and I'm able to jump between Python 3A 373639 depending
- on like what's going on in my computer and I just are on three eight because that's where I've been for a while and so I think if you're if you're if you're
- stumped by um by that I think um ultimately installing Pi e and V using those techniques and then using it
- to switch to say python 3.8 which is the most kind of common one currently um and trying it again there is going to be the route to go I think that's
- probably too much for us to cover here as part of this class and so I think um we probably need an alternative installation method for people
- um there is you know uh I uh there are ways to run Jupiter notebooks in the cloud without installing them on your own computer
- there's one that's called uh Google collab if you just Google this real quick if you wanted to try you could install a sort of simulation of a
- Jupiter notebook that runs inside of your Google Drive using a tool called Google collab and it will have pandas already pre-installed and you could uh
- you could try to make a notebook there and follow along with class using this sort of cloud Tool uh for the time being but I think short of that we're going to
- need to do the pi NV fixes um which we just don't have time to do as part of this class here today I don't think I would encourage people to Endeavor on it anyone who is currently stuck at that
- stage now in class if after class is over you need some one-on-one help getting your computer fixed I would say you should join
- um the first python notebook slack channel that we've set up on the news rotary slack and I'd be happy to at a later date help guide you through trying to do the pie EnV stuff and getting that
- stuff ironed out okay um collab also comes with Altair installed by default so
- that would be that would be a way to not have to install anything else right so Google called collaboratory is the name of the tool
- you could try to jump into that now and I I bet if you fired that up you'd be able to catch right up with where we are in class okay anything else before we keep moving
- with pandas I wonder if we should just give like a couple minutes for the folks who want to try to jump into Google collab because I think there's quite a few
- mm-hmm I'll do it right now I'm actually still having an issue
- um I think someone else in the chat wrote this but when I did the Vim uh pip file I got the this arrow that said found a swap file by the name Dot
- file.swp and it asked me um it said it already exists and if I want to open read-only edit anyway recover quit or abort that does not
- matter you could just abort I mean you know this the swap file is basically it's it's saving your unsaved progress it's sort of like backing you up in case you forgot to save and you want to get
- to it later or you want to undo something and it's really an irrelevant file um editing the pit files nothing we need to do as part of this class I think you can just abort and move on and you're
- fine okay thanks so I'll go into Google collab here myself I'll say new notebook right and Google collab and you'll see that they
- have their own um proprietary spin-off of Jupiter notebook that they probably hope to make money off one day uh and in here I can
- do number equals two right and I could do number plus three and I could do all the same stuff we were doing in the Jupiter notebook right and pretty much all the code that we're
- about to write will work in here just fine for the time being I can name my notebook I hate python 3.9 on OS X oh
- sorry Mac OS sorry and um blah blah blah okay
- any other questions and Ben can you do that trick where you gave yourself a title in the code and
- then your author name and yeah totally we can look at that again real quick that was just taking one of my cells and making it into a markdown cell right so
- I hit the plus button which will add a new cell by default it's python that runs like python right but if I click on it select it and then I go to this pull
- down menu and I say I would like you to be a markdown cell instead I can then start writing markdown woohoo right ah okay that'll make it uh italic because I
- put little things around it and if I do underscores what will it do same thing what is bold two asterisks I can never remember yeah bold right
- cool and how did you add the link it's a it's a little bracket on each
- side of the text and then a parenthesis in the link so this is this is not a Jupiter notebooks thing this is markdown so markdown is just like a generic way of writing kind of HTML ish things
- without actually having to write HTML you can go and read about markdown on markdown guides and it has all the cheat
- sheets for how you do headers and bold text and ordered lists and black quotes and all that kind of stuff I believe you also can just write HTML right guys you can just do I want a b
- tag and you got to be 10 right or I want an H2 tag and I got an H2 tag right
- foreign what was that um News Mercury slack Channel again you said first called the
- news notary slack channel is called uh I think let me find it I'm going there right now it's called fpn
- nightcar 21 I think here's the LA time slack it's uh it's fpn hyphen nikar 21.
- and we'll get to the bottom of this bug uh in that channel once we have some time to kind of focus on it okay here it is right here awesome thanks no problem
- foreign okay I think we're ready to move ahead now into chapter three hello pandas in
- this chapter we're going to import pandas into our notebook so that we can begin to take advantage of its many powerful tools and we're going to get an
- introduction to some of the really basic uh features okay and we're to do it we're going to use a little sample fake
- data set that doesn't have much going on all right but before I begin in my notebook I'm going to clear out some of the stuff I've been doing you don't have to do it but it's just going to make it easier for me and you is I'm going to
- delete all these cells that I was goofing around with and you can do that with the cut feature right here to cut selected cells or by hitting the X key
- and so I'm just going to hit x a bunch of times maybe hit the cut a couple times and now I've kind of gotten rid of all that stuff I can hit the plus key
- and now I've got a new empty cell okay now in my empty cell I'm going to use a python command we haven't used before which is called import and import
- is the python command for bringing into your current script one of these packages that you've installed via Pi Pi right and so if I type import pandas and
- hit plus right it will run the line of code and nothing will happen that's a very very good sign right and that means
- that my python was able to read in pandas and now the variable pandas that has been imported contains all of the
- many many many shortcuts and tricks and tools that Panda has available for us okay so I would ask everybody to type import pandas and hit play
- foreign if you get an error message it's
- because it either hasn't been installed or something else has gone wrong but if nothing happens that's very good now before we continue we're going to return to that cell and we're going to
- make a slight variation to it we're going to go import pandas space as space and then PD and we're going to
- rerun the cell and what this does is it does what's called an alias this will import the pandas library at the shorter variable name that we put after the ads
- and so it's going to be pandas is effectively renamed as PD this is not a required thing to do but it's standard
- practice in the pandas Community when you look at examples of pandas code or pandas documentation you're going to see that there's this magic variable called PD which is doing all these different
- crazy things right and what it is is it's an aliased version of pandas the reason that this is done is because the creators of pandas even though they chose the name felt it was too long and
- so when they're writing their code rather than having to write pandas all the time they would rather just write PD as an abbreviation and so their standard
- practice is when they import the library to do this Alias it is of course optional but you know I think it's good to get in the habit of because it will help your code be understood by other pandas programmers and you're just kind
- to go in with the flow I personally think it's sort of a flaw and now pandas was initially designed but you know what I can live with it you know and so when
- I make a new pandas file I import pandas as PD okay now in the next cell this PD variable is now
- um able to offer us just a million different like things to do and you can access them excuse me
- bye typing PD and then if you type a period um and hit the Tab Key you can see that
- Jupiter has a little helper function that reveals to you all of the many many many many different features and functions that the pandas Library offers
- including ones we're going to use like read cxp or read an Excel file right and so the panda variable followed by a DOT is sort of your way of then accessing
- all of the different things that pandas can do and so when you're reading files you're writing files you're converting files you're doing things you'll often be typing PD dot whatever it is that you
- want to do and learning to use pandas is learning is learning to know what the key different features are inside of it
- and then taking advantage of them while there are thousands of them there's probably only 10 or 15 that we use as part of any real data analysis and you don't even really need to know the rest
- okay so we're about to do that in a minute but before we do that I want to create a really simple data set that we can use to do some basic analysis and learn some
- of these tricks right and so I want to create a list of data that is just some numbers that we can analyze right and I'm going to do that by making a new
- variable called my underscore list equals right and then I'm going to do two flat brackets and this is your way in Python of saving a variable and
- storing within that variable a list of of things and inside my list I'm going to do a few even numbers 2 4 6 and 8 right and
- then I'm going to hit the play button and so I now have saved into my variable a list with four numbers in it two four
- six and eight right and if you were a skilled python programmer you could do all kinds of stuff with this you could Loop through it you could multiply them all you could sum them together you
- could do all kinds of stuff but you would need to know quite a bit of python to do that navigation and what makes pain is good is it has shortcuts that makes doing that kind of stuff easier
- right but in order for your list to take advantage of them you have to sort of feed your datas to pandas you have to say pandas here's my data I want you to
- to like level it up and make it easier to work with right and so inside the world of pandas the sort of like super list the list that's able to do a lot of cool things is called a series that's
- what they decided to name it and so in order to take advantage of it we need to convert our python list into a panda series which we're going to do in our
- next cell right and so I'm going to name I'm going to create a variable to start it's my series my underscore series equals right and then here's where I'm
- going to hit that PD so PD is the little pandas tool belt Dot and then it's capital S Series right so this is the
- pandas series maker it's the thing that can create a new series for you I'm going to open a parenthesis which is going to sort of feed
- information into the series and then I'm going to pass into it my underscore list and I'm going to hit play and so what we've done here is we've taken our previous variable my list we've sort of
- fed it into a pandas series and then we've saved that series as this variable called my Series right and then
- I've hit the play button to run it nothing happened very good now once we have a series we can
- immediately do all kinds of cool Panda stuff including running a lot of different descriptive statistics right and so I can do so my series the variable it now has its own little Dot
- and magic things it can do and so if I do my DOT series sum open parenthesis close parenthesis and hit play it will
- sum all the numbers in the series and print out the results right so these four numbers two four six and eight added together are 20 right and that
- worked for four numbers in this case but it would also work for four million numbers right or whatever database of information you were to pass in right which we'll get to later with real data
- right there's a bunch of other you know we all know this descriptive statistics right my series dot Max open paren close print to execute that we'll then return the
- maximum number in the list which is eight right my underscore series dot Min open paren close paren play we'll return
- the minimum number in the list right what's another descriptive statistic I bet somebody knows one
- median the median yes and like calculating a median is kind of complicated right and like and like uh and you might in writing python code to do that's a real hassle but in pandas if
- you just do my series dot median open paren close paren it'll return the median number which using the default
- median maker there's different strategies for this it took the split the difference between these guys right and it returned five right the average is another one like
- that right in that case it's called a mean in pandas is what they chose to call it right and then uh another descriptive
- statistic is of course the standard deviation and unfortunately they chose to call that the STD and so you do STD open paren close paren and it'll return
- the standard deviation and so on right um if you were to uh do the dot and then hit it and then hit hit the tab key
- right afterwards you could see you could then preview uh all of the many different things that a series can do right so Series has a
- lot of different things you can do kurtosis right uh and all of these are documented inside the pandas documentation if we were to say Google pandas series Max for
- instance we would find that there's a page here in the pandas documentation that explains to you how that Max
- feature works right and it links to some other ones you might want to do like there's some Etc and you can kind of go through this documentation and you can learn kind of
- all the different things a series can do right here's pow which is going to do an exponential power of the series right and stuff like that there's a lot of different mathematical things that can be done
- but these basic ones I think are ones we use a lot in journalism what's the sum what's the max what's the Min what's the central tendency right and then there's kind of the helper
- function that I kind of like called describe my series dot describe open parenthesis close parenthesis run it and it actually
- prints out a bunch of them it gives you the count the mean the deviation it gives you a couple different cut points on the percentile including the median which of course is the 50th percentile
- right and you can kind of see all of them like at once and that's like a quick way to get a look at your column of data to like see what it says
- um any questions about these kind of basic series tools um I have a question yeah when I ran my
- series um it gives me a vertical list of odd numbers and even numbers and the D type just I'm assuming it by describing it as
- ink 64. yes so this is an array of four numbers that we're passing through and I'm wondering does that generally happen
- it just fills in the gaps oh no this is just it's a coincidence so okay so so what you've done if I understand it right is you've printed your series out
- as I've just done here is that right yeah is it just numbering the it's it's saying what index each number is that okay exactly so this is a panda this is
- a panda's feature we haven't talked about which is every row is indexed with a sort of incrementer by default right so Row one is zero in computer
- programming they often start counting with zero for some reason and so this is just the unique sort of num you know um incrementer of it within the series right and then these are the values
- um as you get more advanced you can set the index yourself like let's say you have a unique identifier for instance or you want to group a a hierarchical index
- there's all these fancy ways to do that stuff but we won't get into that in class the index you can do pretty much everything you want to do in pandas without even knowing the index exists
- but there are cool things you also can do with it okay thanks and the data type is is an integer you know data type
- um one reason why pandas is fast and people like it is it doesn't use the default python data types it uses these underlying custom numpy data types which just more quickly process data and they
- have sort of strange names like in 64. other questions about the series
- okay so to illustrate kind of the the some of the earlier points about the notebook itself and how it can be useful I'm going to go up here and I'm going to
- change these numbers right I'm going to do 1 5 10 33 106. just some numbers I made up right
- and then I'm going to go to the top I'm gonna hit run and I'm going to say run all cells and everything will run again from the top right but you can see that
- the result of all the functions has changed and updated to say the new sum is 155 the new Max is 106 right and we were able to just really quickly repeat
- all of our work by by changing one cell and then hitting a single button to rerun it all right and as you iterate and work quickly and fudge around with
- numbers this is just really really useful because you know um and it's and it's also um a way in which what you do is reproducible you know if you were to do
- this kind of thing in Excel and write a bunch of custom pivot tables or little functions you might have to go back and do all the clicking and remember the order of how you did anything to like reproduce the work but because you've
- done it in this way you can make a change kind of Upstream in the information and it will all flow Downstream again when you rerun it
- um and then imagine that your data stream coming in is live on the internet and updates daily right you'll be able to just flow it through every single day kind of in this automated way without having to click click click click click
- the same 10 things every day you know which we'll get to with the real example here in a minute other questions about the series or pandas before we get into some real data
- okay great so we're going to move on to chapter four uh hello money in politics this is just kind of um there's this more written here than we need to say
- for this class but um you know part of the mission of this curriculum initially was to educate journalists here in California about how California's campaign Finance system works and so
- here you can find some information about uh Cal access which is California's database of money and state house politics this is the money that goes towards races for the Governor State
- House races for State Assembly and State Senate and crucially here in the state of California are many many many many many many ballot measures as you may know California is famous for loving to
- put yes or no questions on the ballot which are often uh irrelevant or rip-offs but are occasionally incredibly high stakes and very very important uh
- famous ones in the past have included um uh Banning gay marriage in the in the one we're about to study legalizing
- marijuana uh this last election cycle we had a very important and controversial one about the employment status of gig workers um and every now and then some really
- important stuff kind of goes through this and all of the money that goes into it flows through this database called Cal access which which was actually created as a result of a ballot measure
- itself uh in the 1970s and um it has a kind of an accounting system where the Committees that are formed to raise and
- spend money to Advocate yes or no in these ballot measures have to have sort of transparent bank accounts where they post into this government database the
- amount of money they've raised who their contributors are the amount of money they've spent what they've spent it on whether they support or oppose the measure and then that database becomes
- kind of our really only public record of the money that's flowing into the system and in the case of some of these high stake measures like ones that have to do with regulating Pharmaceuticals or
- genetically modified food corporations can arrive and spend hundreds of millions of dollars to try to influence the outcome of the of the measure in
- fact the the gate worker measure last year uh Uber and other companies spent some much money it shattered the record for the costliest ballot measured in California history with hundreds of
- millions of dollars uh spent for and against um and so if you're a reporter in California this database is a way to try to cover what's happening with that money now there are databases like this
- I think in all 50 states there can correct me where each state has passed its own version of campaign Finance regulation usually these laws followed
- the Watergate scandal in the 70s and what's resulted is this sort of fragmented system where each state has its own sort of variation on a database like this and on regulations like this
- that result in sort of the government puking out sort of gnarly data that you can try to like fiddle through to find and tell stories right and so did I get
- that right yeah and it's fantastic it's just as great as it sounds yeah and um
- and so in this course we're going to get into we're going to use some some data that I have sort of curated out of the gnarly database to avoid some of the nastier stuff you have to do to work with it
- um and we're going to have two files that we read into from the calexis system that are about the ballot measures in November 2016 when there were 17 on the ballot and there was a
- whole set of committees that were organized both for and against those 17 measures and then each of those committees reported lists of
- contributors people who gave them money to support their campaign and for them to spend and so the two data files that we're going to read into pandas next for our real work now are going to be a list
- of all the Committees from November 2016 and a list of all the contributors to those committees right and that is going
- to come up next in uh chapter um five hello data okay and this is where we're going to use pandas to read in that data before we do that does
- anybody have any questions about campaign Finance or the California system or any of the above um we have some people on this call not me who have uh Decades of experience
- with this and so this is anything you want to know that you could probably get an answer foreign
- out the federal system is in fact entirely different as well for presidential and for congress that's its own Federal tracking system
- no okay well it is campaign Finance um so before we begin again I'm going to clear out my notebook of some of my
- cruft you don't have to do this you could just keep writing but I just don't want to distract people and so I'm going to use the X key to clear out everything except import pandas right so I back
- down to import pandas I'm going to hit plus to make a new cell and now I'm ready to go okay at the top of chapter five hello
- data we have a table that has two hyperlinks and these hyperlinks go to the two data files I just talked about one all of the committees or the quote bank accounts that are raising money
- right and the other the contributions these are the individual checks that people wrote to these committees that had to be disclosed according to the campaign Finance law right and what
- we're going to do is we're going to use pandas to read those files into our notebook so we can begin to manipulate them right and the tool that does that
- is called read CSV and so I'm going to go right down to this section called creating a data frame and I'm going to copy this whole line of code which is long because it has a long URL
- and I'm going to paste it into my cell and I'm going to hit play and what we can see if we look at it is we're calling in that PD Alias like the
- tool belt right of all the pandas tools and we're taking advantage of a function that's inside there called read CSV does anyone on the call know what a CSV is it
- could define the acronym for others comma separated values oh beautiful and what are those it's a type of data set in order where
- um the machine reads like each comma as a new cell is that correct exactly exactly and it's sort of a neutral flat way of storing a
- spreadsheet of data and so here in my web browser tab you can see the CSV or comma separated values that is the list of committees and it's really just a
- gnarly term for a spreadsheet and we can see that the first row is all the header rows of the spreadsheet which are each separated by a comma right and then below that is
- each row of the spreadsheet where each value is then also separated by a comma right and so what we're doing with pandas read CSV is
- we're giving it the URL to this file which pandas is then reading in and parsing as a spreadsheet and this is effectively the pandas or
- python version of opening up a spreadsheet in Excel or another spreadsheet program right you're having the computer program read in the data and parse it into a table of rows and
- columns right and what pandas calls that spreadsheet is a data frame a data frame
- is a kind of generic computer programming tool for a table of columns and rows or a spreadsheet that has a lot of like fancy special properties that you can use a data frame can be filtered
- it can be sorted it can be grouped it can be aggregated and so all the things that you do in an Excel spreadsheet with sorting filtering pivoting right you
- could do with a data frame and you can actually do quite a bit more and that generic data for interim isn't exclusive to pandas it's also used in R to describe how you read in a table of data
- into our statistical software too and probably other programs that I'm unfamiliar with and so what you're doing when you're reading it in is you're reading in a data frame which is
- effectively a collection of those series that we were looking at before with each column in the table being a pandas series
- okay and you can see that by running the code it just puked out the result of the table here into my browser and if I widen my window we
- take a look at it we can see at the end of the day it's just a spreadsheet right here are the columns here's one called the prop ID this is the unique
- identifier of the proposition remember there's 17 propositions that year 's the Cal access prop ID this is the ID of that proposition inside the state
- database Cal axis right here's another idea that ID that our team created here's the name of the proposition right proposition 51 School bonds proposition
- 52 State fees proposition 67 referendum blah blah blah right so here on the left part of the table we
- see we have all the columns that describe the proposition that this committee was formed to advocate for or against and so because there are
- multiple committees for each proposition you can see that the propositions repeat because we have one row per committee
- the proposition information is on the left hand side if I scroll to the right we can see that the committee information is on the right hand side
- there's a unique identifier for the committee and you can see that the committee's names are here and these are these hilarious euphemistic names that you see at the bottom of television commercials you know save the bag ban
- yes on 67. you know yes I'm 51 Californians for Quality dog food or like whatever right and so those funny names that you see in the television
- commercials oftentimes correspond to the Committees and bank accounts that you see in this data and the regulations that require them to be disclosed on the
- commercials right are connected to the regulations about these committees and so the Committees oftentimes have the goofy names because that's what allows them to do it on TV ads right those two
- things are directly connected and then you can see we have a final column that's crucial that for each committee they have to file and it is recorded
- whether they support or oppose the ballot measure that they're advocating on so are they four legalizing recreational marijuana or are
- they against it right that will be logged in here are they four Banning gay marriage are they against it Etc support or a post so yes on 51 of course supports proposition 51. and this is our
- first data set all right now you can see that the result was just puked out into the the console here in
- the notebook and it's not going to be reusable in future cells and that's because we didn't save the result of the read CSV function into a variable and so
- if we want to take this data frame and we want to do stuff with it as we go along right we need to save it into a variable the same way we saved the
- number earlier on right and so what I'm going to do and there's an example here you can copy and paste is I'm just going to save it into a variable that I call committee underscore list and I'm going
- to rerun the code and so now the result of this function which is that data frame of data has been saved to the committee list
- variable okay and it's going to be available for us to use in our subsequent cells right now just like the series had all those fun functions like Som and Min and Max
- right the data frame has its own set of fun functions let's cover a couple of those really quick there's two that people use all the time right and you
- access them by typing your variable again I'm going to copy and paste it because that's the professional lazy way to do it committee underscore list dot
- h-e-a-d head open parenthesis close parenthesis and play will print the first five rows of your data set and
- this is just a common technique to kind of take a peek you know what I mean hey what's inside my data right by typing committee dot list underscore head I can take a little peek
- I could even substitute in different numbers like if I put the number one it'll just show the first record if I put the number 10 it'll show the first 10 records right and so anytime you have
- a data frame this head command is a helpful way to just get a look at it and kind of see well what are in my columns and rows exactly right
- the second the second kind of common command for inspecting a data frame is called info if I type committee underscore list dot info open and close
- parenthesis it will print out this which is a little report card about the columns or Series in my data set and so
- it says to me here Ben you have 102 rows in your data set you have these eight columns which we were looking at earlier here's the names of the columns right
- here's the data type of what we think is in those columns so the ones that thinks are numbers are integers the ones that it thinks are like strings or text are
- called objects and pandas that's what it's called is not a good reason for that that's just what they call it and so when you have an object column that means it's words it's letters it's text
- when you have an integer column that means it's a number which means you can sum it and add it and things like that and so this is a nice way to get a look at your table we can see we have 102
- rows we have these columns they're these data types and here it'll even tell you if you have any empty cell for any null values this will give you a report card
- there and so I can see that I have 102 rows and every one of my columns has 102 things so I have no empty cells in my data everything's filled in
- okay um okay so as far as reading in your data frame and doing this basic inspection with head and info are there
- any questions
- no okay so before we begin or before or the next thing to do
- is to read in our other data table right so we have this data table which is our committees and then we have our second David data table which is all of the contributions or all of the checks that
- were written to these committees right which are disclosed by a form that has to be filed with the state regulator and I have another read CSV uh line of code that I'm going to copy
- and paste out of the documentation and this one will read that second spreadsheet into another data frame and this data frame is called contrib list
- contrib being an abbreviation for contribution right so you can see that that takes a little
- while to run and then because the number appears here on the left that means it finished hooray who now if I want to get a look at the first few rows of the data frame what do
- I type somebody tell me untrib underscore list uh period head current yep and then open parenthesis
- close parenthesis you got it and just like in the terminal there's autocomplete if I type contrib and hit tab it will fill in the rest of the variable for me very nice
- and now we get a peek at the first five rows of our contributes there's a couple things about this table I want to point out and how it works but we can see that
- it has a different set of columns than our contributions list but some of them are shared we can see here on the left hand side of this table that there is an identifier called the Cal access
- committee ID this is the unique identifier of the committee that received the check right and you can see that their name is right next to it so here the first one is adult use campaign
- for proposition 64. right I think that's actually our proposition about marijuana so this committee which has this ID
- is reporting this check right and you can see here's all the information about the person who wrote the check and there's quite a bit in here more than many people may realize
- Gretchen Bergman of Spring Valley California right on September 18 2016 right
- wrote a check to that committee for over here on the right hand side there's a column called amount for 84 dollars
- right and her she listed on her form disclosing this check that her employer was a company called A New Path and that her job was as the executive
- director right and so this is one of the the checks that went to this committee and each one of these rows is a similar check each one has the name of the contributor it has their location it has
- some information about their line of work and it has the amount of money that they gave right and um this type of thing is where the data
- like this is where every campaign Finance story about donors you've ever read came from right journalists like the ones on this call
- work with lists like this to try to identify the top donors the top industries Etc to figure out and describe what's going on with our political campaigns
- here and you know the data is you want you you know you probably won't be surprised to learn pretty imperfect people are really able to put whatever they want into the employer and occupation field uh people
- inconsistently spell their names they give different answers to these questions over time there's no unique identifier for a donor over time at all
- and so aggregating up who's given the most money requires cleaning up variations making decisions about who to combine that gets more complicated when
- you get into things like husbands and wives decide figuring out how much money the oil industry is given requires looking at things like the occupation and employer field and choosing to
- categorize those things um based on what you read there which can also change over time as companies change over time and there's a very sort of messy imperfect process that goes
- into refining data like this into the the stories that we tell in this case our goal is to figure out in the case of uh the marijuana legalization measure
- which side supporter oppose raised more money and who were the top contributors to each side right and so what we're
- going to do in the subsequent chapters of this this class is we're going to merge these two data sets together we're going to filter down to the ones that
- were just limited to the single ballot measure that we're interested in and then we're going to begin to write analytical code that groups and ranks and sorts the data to ask and answer
- those types of questions that I was just articulating right and the two data sets are going to be able to be fused together to achieve this using their common identifiers and by merging the
- tables right because we have the identifier of the committee and the contribs file we can merge it to the Committees file and we can do what we
- need to do to kind of make this work and that process of wrestling with this data and then learning five or six core analytical techniques like filtering
- sorting ranking merging Etc and then chart making is what the remainder of the class is going to be and at this point I'm going to hand it off to my colleague Iris Lee who's going to pick it up where we are now and teach you how
- to use pandas for real to actually do this journalistic uh investigation that we're on okay this might be a good
- minute to just also give Iris a minute to take over the the screen and pick up where I've left off and also to answer any questions that we have from people in the crowd does anybody have any
- questions yeah um Ben uh I'm a little bit of a campaign Finance junkie I'm just curious if anybody has taken the time to build this database or something
- like this for local uh for like uh local and County races um or that is something that hasn't been done yet
- you know you you um you point out a hole in my description of the system right we have the federal campaign Finance system which which tracks the money in presidential and Congressional races we
- have these State systems which track them at the State House level right in the Statewide level but then there is a third layer yet below that for local races for races for the mayor of your
- city or for your board of County Supervisors or your sheriff or your district attorney all of those um are are collected and tracked in the
- hodgepodge of systems that are set up by city and county governments Across America and there's no one system for that and it really varies a lot from
- place to place you know and so there's not one database there are places that have tried to aggregate it I don't think anyone's done it with much success and the problem is is they're just these fragmented different broken screwed up
- systems and it's just a gigantic effort to do it um the fundamentals of how to like understand the data is kind of the same because the regulations are all very
- similar things like contribution caps committees some stuff we're going to get into a minute about dealing with how things are itemized those are similar
- but your data source and the structure of the source data is going to vary from place to place you know uh in in Los Angeles County um you know which is home to 10 million
- people and it's the largest most populous and in some ways one of the wealthiest places in the world you know the largest most populous County incredibly wealthy place
- um supposedly liberal and advanced our County Board of Supervisors five people who rule over 10 million still report all the money in their races on paper and it's not digitized and they have and
- based on a series of stories that my colleagues like Malloy Moore have done they pay people to key punch them into an electronic system after they're filed
- by paper and then they don't appear online until days or weeks after it's actually important to look at them and and I'll just add that like I think
- I think my the student the student story on this that La published more than half of California's counties don't even have that they can't local campaign Finance
- on online at all and that includes Los Angeles County which is just shocking yeah great um and so it's it's it's pretty bad out
- there um I I personally think that getting into your local races is a great way to is a great way to cover what's up in your city in a way that a lot of other reporters aren't and you don't have to
- totally Master it but learning a lot of the tricks like we're covering here today is going to let you um do that so I I don't want to give any spoilers but by the end of this class we
- will have discovered a lead to a story that nobody covered at the time in the state of California about the one of the top donors they've been just a mechanical thing you
- started with the simple list of numbers and I guess at some point it became a matrix that's right that's that's the that's exactly right that's the difference between the series and the data frame right so the series is the
- pandas jargon or terminology for the column of data for a single column right like a list of all the amounts right or
- all the committee names and then the data frame is the table or Matrix right which is it's a spreadsheet right it's a list it's a group of columns right
- um and that happened when we did the read CSV function because the read CSV function read in a spreadsheet of data that we had prepared which was then converted into this data
- frame got it and it just sort of knows what's the rule and what's the column by the way it's read in exactly because of the comma separated value data format which
- is the common generic format of spreadsheets and so if you opened it in Excel Excel would have parsed it the same way
- thanks excuse me all the columns should be of um made of one type I mean the old uh
- for example one column all record is the strings or another column all later is a a numeric the right
- uh they could be so really the data type of each series can vary right there's not like a um a rule or law but what pandasbus by default is it sort of it it
- guesses so pandas when I did read CSV behind the scenes without me knowing about it it did what's called sniffing the data it read the first 100 rows or
- so it sampled the data it looked at what was in there and then pandas made a decision and it said this column is an integer this column is a string etc etc
- right I think yes now sometimes it guess is wrong and so so there's a when you when you do read CSV there's an option called D type and
- you can specify I want this column to be a string or this column to be an integer and you can force it to the data type that you prefer right thanks thanks yeah the most common example of this is of
- course fips codes fips codes are the identifiers for States and counties in the United States and they have leading zeros so the state of California's unique identifier is zero six and pandas
- will strip the leading zero and make it an integer and then I have six and then it can't join to another data set I pull my hair out you know and so I have to
- always tell pandas no the fips code is a string it's not an integer right thank you I completely understand thank you yes
- zip codes are also a pitfall especially if you're in Vermont where every zip code starts with a zero right I bet that's oh my God
- so you have to write all this code to like put zeros on the front of government zip codes I bet sometimes like insert the zero back in excuse of like string padding function but you
- have to remember to do that otherwise you're gonna regret it down the road yep other questions yeah
- um quick question so I'm kind of a self-professed um our user but I'm really interested in trying to you know get up on python um and I noticed
- um like the way that R handles um different data types in a specific column is it will just like have a list in in like what what R's equivalent of a
- list is in that column does python or pandas I should say specifically have a way that it handles like different data types being in the same column so if they were mixed data types like it was
- like hodgepodge um Iris can maybe correct in here but I think it just says it's an object right and then you kind of have to figure it out object will say Tech would be
- usually string but it can be a mixed data type like a list and that that can be frustrating yeah interesting yeah there's many things that I think pandas can improve at I would say that's one
- thing right there um I when we're working on something that's like serious like you know our coronavirus tracker or something I'm uptight about I will explicitly declare
- the data types even though it's right code because I just want to be sure you know yeah plus for errors and stuff like yeah it's just probably better be more explicit but okay that's an interesting
- thank you it also will run faster the sniffing of the data set can actually slow down the computer especially when the data set is very large and so if
- you're reading in a large data set declaring the data types as a way to make it go quicker foreign okay any other questions before I hand it off to Iris is there um a method for a non-csv data
- types oh yes there are many many methods um so like if I do PD dot then I do read underscore and I hit tab it will show
- them all right so you can see the pandas can read an Excel file which is super handy right it can also read an Excel file from a URL so you can like just
- scrape government spreadsheets right off the internet right it can read a fixed width format file if you're like a 90s baby like me um it can read HTML tables it can read
- Json format which is really common and it can read SQL databases you can actually write SQL code in your pandas which is pretty cool or you can read in
- a whole table directly there's a bunch of stuff like that thanks [Music] anything else I just was curious can
- can python manage the data or would you do that in like your XLT or your wherever the data
- is coming from like if you wanted to change the data that you were at oh you could definitely change and edit the data here in pandas pandas has a lot of techniques for like selecting a row and
- editing it or one thing iris is going to show you how to do is how to compute a new column I think we'll get to that at some point where you want to like take a couple columns and like make a new one
- based on what's in there um and then there's functions for writing files out to different formats too so we could read it in as a CSV and we could write it out as an Excel file
- if we wanted to for instance or out as a Json file or whatever else but yeah the the there's there's all kinds of stuff like that
- okay I think with that I'm going to hand it off to Iris Lee I'm going to stop sharing my screen so that she can share hers I'm going to stay on and mute and
- be coaching uh as she picks up I think with what chapter six Iris yep chapter six okay
- hi everyone my name is Iris Lee can you guys see my screen
- yes okay cool um all right so we're going to start at chapter six if you're using collab by the way I
- really have to show you this um if you're on Google collab there's this phone there's this uh thing on tools that settings and if you go to
- miscellaneous you could get your notebook in Corgi or Kitty mode and then it'll have these little corgis
- running around on top I just love it so much um and you'll see some kitties too coming around so if you're in collab feel free but anyways
- um so let's start where we left off so we looked at two data frames the committee's list and also the contrib list and from this point on we're going
- to explore the committee list data frame first okay so I'm just going to start a cell on the bottom
- um and just start from the top so sometimes when I'm looking at like two data frames like I am now I kind of forget all the column names right it's
- like really easy to forget um so what I like to do is kind of like what Ben showed you the head method but I just kind of like to print out just
- the first line and if you want to just do that you could just put one between the
- parentheses on their head and then you can kind of see like the very first line of your data frame so sometimes I'll just put this up top here so that I know
- what the column names are and I know what to type so we're going to look at this column called prop name to select a column in a data frame it's
- pretty straightforward all you have to do is type the data frame's name which is committee list at this point
- and put a dot and put the column's name and we do that by saying prop name and now you can see this is kind of like the series object that Ben showed you
- where things are indexed on the left side but you can see all the prop names in this data frame and there's 100 to
- not different but hundred two lines the same line as the data frame right at times when you use different kinds of
- data sets sometimes the column names are not clean like this sometimes column names may have spaces or capitalization and one way and if it does have spaces
- in between this dot method will not work in pandas so what you have to do is you have to do like a bracket and a
- quotation and then put the column name and it's exactly the same thing so let's say this prop name had like a
- space in between that this is like the notation I would use but because it doesn't I can kind of do this like real quick shortcut of prop
- underscore name and then here um I'm not pressing play I'm Just Doing shift enter just to run the cell real quickly so here um
- we see a list of prop names there was like some props propositions that has to do with school bonds referendum to overturn ban on I don't know what that
- is something about hospitals but we are going to try to look at what kind of prop names
- are here and how many are there so kind of like the frequency of how often does this proposition 51 come up in this whole data frame okay and in order to do
- that we are going to select the column that we want to explore by typing the same thing committee list prop name
- and we're going to do this method called value count open and close prints
- and this value counts is like a method that you use on the series kind of like a sum or mean but it lets you see how
- many times each type comes up so let me just expand the screen so it's a little cleaner so as you can see that there is at least
- 13 rows that mention proposition 57 there's like 12 of proposition proposition 56 and all the types all the
- frequencies that this proposition name comes up in this data frame um so we
- are going to all right so sometimes this in this like series
- would you mind bumping up your fear of font size here oh yeah a little just like control plus I do it okay there you go is that better folks a
- little better okay great um so now um we're seeing the value count of each prop name but this is in this
- like series format very texty kind of hard to read so what we can do here is we could do a reset index so I'll just type
- the same thing copy and paste and then right reset index and pandas indexes it back into a data
- frame so now you can see kind of in this like clean tabular format how many times each proposition name comes up
- anything to add coaches no uh one particular question and do we always need apparentice after
- a value count or is it okay to cut out um yeah what's the difference that's a good question I've never done
- it without parentheses then do you have an answer for this you definitely need to do it it wasn't counting if you did it before you'll see oh yeah it was a list of 102. I don't
- know why that even returns anything I think yeah this is I think this is just a a quirk of how python prints stuff out because it's it's printing out the function as a
- bound method and then just the whole series following it it's like printing two things I see yeah so yes probably the string representation of the series if you want
- to get into it okay all right cool
- so we're going to the next chapter filters so the specific proposition that we're looking at that today is the proposition
- 64 which asked if marijuana should be elite legal or illegal in California um as you guys all know it's legal in
- California so it did pass but um we're going to filter this data frame to kind of just look at the Committees
- that supported or opposed this specific proposition and in order to do that we will have to filter this data frame um not this data frame but the
- committee's list data frame and just get the rows that mention this proposition 64. the marijuana legalization initiative
- so in order to do that we'll do a filtering method and one way to do that is to use an operator like double equals
- greater or smaller this is kind of similar to um in SQL like where function so double
- equals in functions mean that it's not like setting a variable but it's it's like asking a question is this on the left side the right side so if I
- do one is one then it's true so it returns are true so we're going to do this like similar method to filter this data frame so
- one so first I'm going to set a variable that is the prop name that I want to filter on so I'm going to call it my
- prop and I'm going to set it as this prop 64
- which on your notebook there should be um you can like copy and paste and I'd rather do that because I'm not very good
- at typing so I'll just copy and paste and I'll just and now the my prop variable is
- set as this proposition legalization initiative statue so if you print out my prop
- it is what I said it as and I'm kind of doing this so that I don't have to type this whole thing over and over again which will like introduce errors
- um and in order to filter the data frame on this specific prop I will name the data frame again which is committee's
- list and then we're going to put square brackets
- and we're going to call the column and we all know the column is called committee list
- dot prop name right and I'm going to filter it to the rows
- of the data frame where the committee list prop name is equal to this my prompt right so I'm going to a double equal
- sign say my prop so what this does is this filters the whole data frame that has all these different propositions in them but only
- targets the ones that have proposition 64. so as you can see this new data frame that we created from this filter has
- 13 rows or 11 rows because they were 11 times this showed up in the whole data frame and they're all proposition 64.
- and now we're going to name this into a new variable
- and I'm going to call it my um my committees okay
- is it my committees yes okay cool
- so I like to think of variables as like these like little jars that you put like labels on so now that I've like filtered
- this done this like whole filter I'm gonna just name it my committee so that I don't have to do this over and over again and I know that I just have to just pull out this my committees in
- order to use what I've filtered from the Committees list and now we can again kind of do the same thing that we did with the other two
- data frames and run the head and the info method to see what's in there my committees
- we can look at the first five lines or we we can also look at info
- and now you see this data frame unlike the other one it has only 11 11 rows but
- because we've only pulled out the proposition 64. um prop names okay
- but it keeps the same columns good okay all right moving on any questions I'm filtering
- no okay so now that we have filtered down to this contribute
- oh so our next job is to filter down the contributors list the contributors who have
- spent money to support or oppose this particular proposition so we're going to go back to our contrib list
- and we're going to merge the contriblist file into my committees so if you guys go up your notebook if you look at your
- contribute list you'll notice that we Ben had mentioned that there is an ID a
- unique ID called Cal access committee ID that corresponds to the committee ID in the Committees list so what we're going to do is we're going to merge these two
- data frames so that everything so that every person who contributed it to this specific committee will be added
- into this new data frame does that make sense this is just like
- joining tables and SQL if you guys have ever taken two SQL tables and join them together or like doing a vlookup in Excel where the two things come together the unique identifiers in each table
- link up and then you only keep the ones that match so the Venn diagram of the two tables are merged and it's just those ones in the middle that share an ID that
- remain that's a that's an inner join basically if you're thinking about it so I'm going to do a pandas merge
- command and we're going to name this new data frame that we're going to create from this
- join merge so let's emerged and then in pandas if you want to merge two data frames you just do pd.merge
- and open and close parentheses and then you're going to put your arguments inside this open and close parentheses so I'm going to try to merge this my
- committee's data frame which only includes committees that have you know supported or opposed this proposition into the contributors list data frame so
- what this will do is essentially it'll go through the my committees and look at this Cal access committee ID
- like one three eight one eight zero eight and try to find that exact ID on the contrib list and then attach that to
- the data frame so to do that we're going to merge my committee's data frame which is called my committees
- and we're going to merge that with contriblist and then you're going to tell pandas
- what do we want to merge these two data frames on and that's just called on and we know what it's called it's called
- Cal axis prop ID right you could type that or Cal access committee's ID excuse me you could type that up or you could copy
- it and paste it and this merge will work cleanly because the two data frames both have this
- column called Cal access committee ID but sometimes when you have two data frames and you want to merge on a column and they have different names you can
- also specify that saying things like let's say like on my committees the Cal access committee ID was called like just committee ID then in pandas you can kind
- of specify which I which columns that they could merge on by saying things like left on or right on
- but in this case the data frame has exactly the same name of the columns that we want to merge on so we can just name the column
- if we look at the merged data frame oop okay so a lot of times when I do
- pandas because I'm trying to type really fastly uh really fast a lot I get like a lot of errors so whenever you get an error like don't freak out just like read the error so
- right now it's saying data frame object has no attribute hat I obviously like spelled head Raw that's why it's occurring so just correct that and then you're good
- so now if we look at this merge data frame we can see that on the left side it's kind of exactly the same as my
- committee state of frame right it has like Cal access prop ID it has the proposition name um and it has the committee name
- but on the left on the right side it's The Columns that was the columns that were present on the contrib list data
- frame right so we have the filing ID the date it was received the contributor's name the contributor's first and last name and
- all their information right so now the two data frame has joined as one and it has become like a really long data frame that has both of the columns together now if you notice I don't know if you've
- noticed this but here we have like a new column name it says committee's name X and this is
- because the two data frames that we merged my committees and my and the contrib list had a same column name if you look at
- so if you look at the my committee's columns there is a column called committee name the contribute list also
- has a column called committee name right so in order to not mix these up pandas what it does it just attaches the suffix
- so I know the X came from the left side data frame and I know the Y name came from the right side of the data frame however they should kind of match up
- because we're talking about the same same proposition here or the same committees here but that's why this column name has changed
- any questions so far
- now if we look at our merged column merged data frame now our data frame has kind of increased in size with 860 new
- entries can you guys guess why um there's more personal contributions than committees exactly so there were only 11 committees I believe right so
- there were only 11 committees but many many people are contributing to each different committee so for every every like 5 10 20 people that contributed to
- like this specific committee pandas has created Rose for that contributor's name if that
- makes sense so we will still have only like 11 different committees but the people who contributed to that will add more
- committee names does that make sense so now that we have merged data frame what does this contain this information
- contains every committee and every person that donated to this specific committee so now we're going to work with this merged data frame to get more
- information about it okay so now we're moving on to chapter nine
- so first we're going to start
- by doing some math on the amount of money that was spent to
- each of these committees so if you look at our merged data frame it has this column called the amount and this is the amount of money that each contributor
- donated to a specific committee so this is similar to when we sum the series in the
- beginning of this class so we're just going to call on that column merged and we know how to call the
- column by saying dot so we're going to do the data frame name and we're going to put the column name the amount and then we're going to call sum
- right and what this does is it goes through this merge data frame takes all the numbers in this amount column and sums
- it up all together and now we can see that there have been what 35
- million dollars spent right so I'm going to be like hey Ben guess what I found out in 2016 there were 35 million dollars spent on this
- proposition can you believe it I'm gonna tweet this out put it in the headline yeah and then Ben's gonna be like not so fast my friend all right exactly
- yes and this is where we can there's a couple uh like quirks about how campaign Finance systems work that I think it might be worth pausing and just passing on to others in case they haven't which
- is uh one as you're like pedantic editor I'd be like well technically Iris it's not how much they spent it's how much they raised right because because this
- is the contributions table right and then there's a very Persnickety particular thing about campaign contributions that every newbie has to
- learn but like nobody teaches you which is not all contributions are disclosed right so when you get a list of campaign donors like this from a committee right
- it is not every donor it is only donors who gave enough money to trigger the reporting threshold right and in every
- state and in every race and in every campaign Finance scheme there's a minimum amount of money that's set and only people who give more than that amount of money have their identity
- revealed typically and so when you're looking at an itemized contribution like this list like this in most cases it's only going to be the people who gave more than that amount of money and
- therefore if you add up the sum it's actually not the total right it's only the total among the people who gave more I think in this case than one hundred
- dollars right and all the people that gave less than that have their identity shielded from disclosure and their money just goes into a lump sum of quote un
- itemized contributions and so when you're covering a campaign or looking at a committee to understand the total amount of money that they've raised you
- need to look at the totals that they report on different parts of their form which are called cover sheets and there they'll they'll report a big lump sum of the total amount they raised how much
- was itemized like the ones we're looking at here and how much remained un itemized because the names were never disclosed and so in this case 35 million
- is not the grand total we would be wrong to report that it is the total amount of money from itemized donors and if you've read campaign Finance stories Through The Years you may have noticed that uh
- that journalists like Derek have to have come up with a like um lingo to sort of talk around this
- problem okay have you ever noticed people talk about large donors to a presidential campaign and they'll refer to this group of people as large donors or small donors right that language is
- almost always just an attempt to humanize this legal distinction of which donations are itemized and therefore we know the identities and which are
- unotomized and we do not know the identities right and so in this case what the correct thing for Iris to tweet would be that there were 35 million
- dollars contributed by what we would call large donors or people who gave more than one hundred dollars is a hundred dollars a large amount of money debatable right
- but that's those are quirks and those are going to be present in almost every campaign Finance database including the one where you live and so you just have to be careful about that and kind of get
- a grip on that and recognize it when you work with it um there are quirks to this the ACT blue database that powered the Bernie Sanders Campaign which was mostly small donors
- did end up disclosing all of the small donors through a strange pass-through mechanism that they had to use to move the money and so it's not all as universally true sometimes you can learn
- the small donors um but we don't need to get into it okay thank you great so I'm okay well now I know then
- that 35 million doesn't cover everyone um so like Ben said this information usually includes people who have donated 100 or more so we could kind of check
- that out so maybe if I pull on the column once again merged amount and I called the Min it's going to give me the
- minimum value in this column so let's check oh it's a negative number what
- does anybody know why um because it was charged the wrong account maybe maybe that's the root cause but
- what we know from what's disclosed is that this was money that was returned to the donor right and so with it because we have to remember that these systems that we're studying are accounting systems they're not really public
- disclosure systems right and so they include things like correct Corrections so if someone wants their money back or has their money refunded because Iris
- does a story about how they're a criminal you know what I mean or whatever that might be charged as a negative value into into the itemized contributions right um there's cases where people give too
- much money there's the legal cap on how much they can give and they just accidentally give too much I mean it happens they might have that charged back down um there's other reasons why it happens
- Derek can probably rattle some off but this is another Quirk that you have to keep in mind is that we were thinking um maybe we could count this table and
- say how many contributions there were right well we can't because some of them actually are contributions they're funds right and so there's just these these
- little accounting complexities you have to you know you have to kind of wise up to and and work around and and you can great so now that we know the quarks of
- this data um this data that we're dealing with let's try to filter our merged database into the Committees that
- support and committees that oppose this proposition so if you look at the columns here there is a column called committee position and it tells us if
- this committee is for or against proposition 64. so in theory right now you're seeing all the ones that support but in theory there are ones that
- support and oppose and we're going to break them apart and put them into separate data frames okay so another way to check that would be
- merged committee
- position value counts right looks like I spilled this one
- position okay all right so if we look at the value counts of this specific column we see
- that there are 762 entries where the committee's position says support and 98 entries where it says a post so we're
- going to break this and set it as separate data frames so it's kind of similar to what we did when we filtered on the proposition name
- first we're going to create a data frame that has to do with the committee where the committee's position is support so we're just going to name that support and we're going to filter this merge
- data frame on the column committee position
- where this column committed position is named support or the value is support and they have to be exactly the same
- the way it's written otherwise it'll break and we're going to set that aside as its own data frame we're going to create another data frame called a pose where
- let's just copy this were the committee's position is not
- support but opposed and then we are going to create another another data frame
- so if you look at support [Applause] it's 762 entries that checks out because there were 762 entries in the data frame
- that it was filtered from right now we are going to run the sum once
- again to see how the committee's Committees who support and oppose raise different amounts of money so I wonder if one of our students knows
- what's to type oh yeah so if the question is What's the total amount of big donor money from the supporting side what would we type in
- support amount some our support dot amount dot sum parentheses yeah hell yeah
- and we'll do the same for a post
- and look at that what are the what are the numbers there so it looks like there are more money
- going into the support side then yeah like 32 million to 2.5 million pretty impressive so we've already answered a question right guys it looks like there's a lot
- more money on the supporting side than the opposing side which is a you know an answer to a question and maybe a lead on a story yeah so Iris what if I wanted to
- figure out like what percentage is that like I know I could get a calculator out but I don't know yeah can painters do it for me yes pandas can do it for you I would not use a calculator because I
- type things very badly um so let's get the sum here the support amount sum and we can just do divide it
- with the merged amount sum that we got before right yes and so you see how iris is using that slash to do the division that's the built-in python division tool
- and so this is a case where we're taking two like fancy pandas things like a series sum on both sides and then we're just putting a very simple python thing in the middle that division sign
- so now we know that 93 of all the money raised not all the money all the money
- raised from large donors there you go have gone through the support side yep wow what a finding
- I wonder who wrote those chicks hmm let's figure out
- Okay so we're going to now move on to chapter 10. uh before we figure it out we're gonna
- sort through some of these amounts so what were some of the 10 biggest contributions I'm I have a guess that it
- came from the support side but let's just look at it so we're this is how we found these support versus opposed money but we're just going to look at the whole um the all of the money raised
- so we're going to go back to our merged data frame which includes both support and opposed and we're going to sort the values
- and this is another pandas method that sorts sorts the data frame on a specific column and you name the
- specific column that you want to sort the data frame on by putting them inside of the parentheses here so I am going back to my merged data frame which looks
- like this right and I want to sort it on the amount column so as it stands now it's kind of sorted like haphazardly right
- it's like there's a hundred dollars there's like 25 250 000 but I want to sort it from top to bottom
- so I'm going to write amount here but if you do at first when you run the function amount it won't sort
- um it will sort from smallest to the largest so we have to give a specific argument called ascending false
- which means do not do not sort it in ascending order basically
- so now if we look at this data frame we can see that the amount has been sorted to the largest to the smallest so the
- largest amount given was four million dollars am I right
- and it can just look at the first top five by calling the method head again then we can see the top five here
- and we can see that it came from who is this top donor
- the name Parker someone named Parker who lives in Palo Alto one of Cheryl's neighbors yeah and his first name is Sean Sean right yeah we just live right
- next door to each other I'm sure does anybody know who Sean Parker is this is part of the fun of campaign Finance I just rewatched uh The Social Network
- so yeah yeah so he's portrayed by Justin Timberlake in The Social Network as I recall and he was the founder of Napster the notorious MP3 sharing platform and
- an early Facebook investor I believe he had a Game of Thrones wedding am I getting that right trying to think of other Sean Parker facts and uh he contributed four million
- dollars in a single check to which side Iris looks like the support side yep and so part of the fun of the of
- doing this type of data analysis is looking in and finding these names and thinking like don't I know that person are like can I Google them and so Sean Parker's brought the biggest check right
- yep and we can also see that the top five largest checks went to the support side because it's all support support
- here right and you'll notice that not all of them are people there's sort of an interesting Quirk here that some of these are non-profit groups like we see there's the drug policy action
- non-profit the fund and this is a common thing you see in California campaign finance and elsewhere where um basically shell organizations are created to move
- money between different committees and propositions and um we won't get to it as part of this class but if you were to use other research methods you could
- lose you could learn about campaign Finance you could try to read form 990s filed with the IRS other disclosure forms available on the state of California website and you would learn
- that the gentleman named George Soros is the ultimate source of the funds for that group great
- um so because the support site is so big this is already kind of giving us the top five support side large donors but we
- can also run that a different way which is getting to our support data frame which only has the support side and kind of running the exact same court code
- foreign ascending false
- and let's look at the head and again Sean Parker up top
- and now we can look the opposing side by going to our oppose data frame which only has the opposed
- contributions I think I named it a pose and we can see that on the post side
- the largest amount given was a million dollars from Julie Oaks in Pennsylvania
- so Julie I mean sorry Julie sure sorry from Oaks Pennsylvania right and so this is another like fun thing about going in this it's like the top donor is a person
- who lives in Pennsylvania who is listed as retired who I've never heard of I mean top four donors looks like yep
- and so Julie Schauer um if you were to Google her you could read some interesting interviews she's done with High Times magazine in other places but they were all done mostly after this
- election and so to me this is a case where by just fiddling around figuring out how the top donors did we uncovered this interesting person Julie shower someone who doesn't live in California who's willing to spend at least a
- million dollars to prevent marijuana legalization there and um no one wrote about her there was not a profile there was not a story nothing was done and her
- identity and what she was doing was was hidden in plain sight in these databases and there just weren't any reporters in California who had the skills and time and like kind of interest to to push it
- down and just by doing just this very simple analysis that we're doing here you can uncover stories like this for yourself in the areas that you cover just like who gave the most what's the
- big check let's Google them figure it out call him on the phone bam you know and also looks like Julie gave once in July 2016. July 20th and also again in July
- 25th are they the same person we don't know we'll have to check but looks like they they could be
- all right so now that we know um the amounts that went to oppose and support were going to try to rank
- we're going to do another thing called pandas Group by and this is a panda's message that allows you to group a data
- frame by a specific column and then calculate anything from like a sum to mean
- um to for each grouped set grouped column if that makes sense okay Yep this is just like a pivot table in Excel where you take something that
- has one row per item or check in this case and then you're going to roll it up using one of the other columns to get totals by name or state or whatever else
- right so let's say in this case Iris came to be her editor and was like Sean Parker's number one donor four million bucks and then she showed me this and I said well yeah but are you sure he gave
- the most overall because I mean that's just one check it could be this non-profit wrote five or six checks that added up to more money right so if we're going to figure that out we gotta group
- the checks together and add them all up right and that's what we can do next yes um so let's so you can Group by any column really um but first I am kind of seeing that in
- the supporting and opposing side there are a lot of other people from other states kind of contributing it seems like so I want to know um if I can Group
- by the contributors estate and see how much money is coming from that each state right so a group Buy in pandas so I'm going to
- go back to my merged one merge data frame that has both support and opposed also we're on chapter 11 now by the way
- if you are following along um I'm going to do merged Group by and I'm going to group by here and put in a column name
- so if I look at my I'm just going to take a peek at my merge data frame again
- I want to group by this column called contributor state
- so if I do a merged Group by contributor State you can see nothing really happens it just pandas has created the groups but it doesn't know what to do with them
- so it's kind of like waiting further instructions so what I want to do I want to group by the state and for every group so I'm guessing there's a group of CAS group of Pas group of like you know
- Virginia's we are going to group them and we are going to take that column amount and I want to sum them all up so I can
- see where the money came from each on the state that is available in this data frame okay so if we do this
- and then I'm just going to try to make it look at pretty so I can do reset index then the pandas will put it back into this like pretty data frame format
- and I can see that many different states in fact 29 different states have contributed to this proposition
- I have a suspicion that California will come up top but we can also kind of take a look at that [Applause]
- the virus wants to take this list and rank the states by which gave the most money she she needs to use a trick that we've
- already used so far in class and apply it to this data frame does anybody know what it is values
- that's almost it search values on parentheses
- the amount ascending goals exactly yes you need to use that sort values function to Resort it in descending order using the amount
- column and this is a case like so many with pandas were to like take that next step to ask that next question is just to chain on another one of these little tricks you've learned right so we
- learned how to sort before now we can take that trick and we can use it with the group thing and boom you know what I mean we can answer this next question
- so what's the answer so it looks like California did contribute the most uh raise the most and then next it came from Delaware New York DC all the way
- down to a b what's a b is that Kansas
- great question I don't know Alberta Maybe we will have to explore this so you know if you wanted to figure it out you could try to freelance and write a filter
- right on your data frame to find that record that's how you would like ask that question it's the same filter we wrote before different column different input right it'll be like merged
- contributor yep day equal equal a b you see how she's doing this and this like what iris is doing right
- now this is it this is being a panda's reporter is you're like what's up with that that's weird let's write a filter and like look at it and you're like uh it's an attorney in Calgary what is that
- even legal you know and you can see he had his money refunded right Iris yep so it looks like he gave on
- June and then he got refunded in July right so we're just guessing because all we have before is these records but my understanding is it's illegal for
- foreign Nationals to contribute to American political campaigns uh and so my suspicion would be is that the campaign got the money in through like a website or something and then they sent it back when they realized you know they
- couldn't accept it so we could kind of take this to a Next
- Level kind of like what Ben said we can see which contributor have um donated most and all we have to change
- is Group by because like we said like we saw here Julie's name came up like twice here um we know that she gave more than once um so we can kind of try to find people
- that gave more than once and then kind of sum them up all together and to do that um what do you guys think we should Group
- by on any columns stand out to you that may
- give us the identity of the contributor your last name right it's a contributor there is two
- columns here contributor last name and contributor first name I want to hear on the safe side because names can last names A lot of people
- could have like similar last names so I want to group by on two columns together the last name and the first name and in group by we can do that
- it's exactly the same code except we put two instead of one column we put two columns in here
- and to do that we put it in a list format so a bracket and then I can type in here
- contributor first name and then comma contributor last name
- so now I'm going to group on contributors first and last name together and we're going to run a sum on
- the amount column so if I do that oh I got an error because I spelled contributor wrong
- here we go there we go
- so I guess Sean Parker is just giving a lot of money but also Sean Parker and Affiliated entities
- right so he's in there in two different ways which makes it kind of tricky you might need to add those together yourself or clean up the data but you know you might look at this Iris
- and say yeah Sean Parker's number one right I add those two together he's given more than anybody else right okay you might notice
- that there's something wrong based on what we looked at earlier in the data that there's something missing does anybody notice it is it the support and oppose
- well that's true that's not there we could we could Group by that if we wanted to you want to add that Iris sure add committee position to the group that's like another nice thing
- but that will just improve what we have here it won't fix the hidden mistake does anyone see it there were Clues
- earlier didn't Julie have like a different will she listen to something else as well sheep might have been but we're not seeing it here there's something even
- worse well I mean one thing I always
- keep in mind with stuff like this is is um when you're working with a data frame you can move really quickly you can group and aggregate a lot of Records really quickly and things can kind of
- fly by and go wrong Without You noticing and one thing I made sure to point out earlier when Iris was sorting and looking at the biggest checks do you guys remember that list we looked at the
- biggest checks and we stopped and looked at that I always like to stop and look at things like that as I go and keep them in mind as I continue my analysis because sometimes things might go wrong um when we looked at those biggest
- checks remember there was drug policy action and these weird non-profit groups that had written these zillion dollar checks fund for policy reform and all those
- guys like those were multi-million dollar checks and they're not in our ranking after we grouped I mean what's up with that
- shouldn't they be there somewhere right something's got to be wrong and like this is one of these moments where you're just like uh your sanity checking you maybe once you've seen and and
- there's a reason why they're not here and it's not really our fault it due to it has to do with a quirk in pandas that I think is really crucial to pass along to all students of a class like this
- Iris you want to explain it four so if we look at let's see if it pops up
- so we are grouping from this data frame right so for this committee
- um yes on 64 California is to control and regulate blah blah blah we have a name associated with it here Laura
- Braden but right under it as you can see right here the
- there is a um a notation Nan which means it's not um it's not a number right it's a null
- value it's an empty value so when a value is empty the group by is just going to ignore this so what yes are you telling us we'll
- just delete the record from it's just one care and throw it away exactly and so this is just a really
- dumb default in pandas which is if you Group by a column and there are no values in that column by default it is omitted from the result right and it
- just so happens in California campaign Finance when a corporation or a non-profit makes a donation the name of the group is put into the last name field it's like a cluge on their own
- database that they wrote right and the first name field is left null and so the result of our group by without us really maybe even thinking about it is that we lost all corporate and nonprofit
- donations accidentally right and so we have to adjust for that and how we write our group by an account for it and like ways you could have identified this
- earlier right is one kind of looking at the data row by row and making sure the results are what you think they kind of ought to be like that's how I spotted this initially but there's other ways too when we ran those info commands
- about our table really early on right for each column it gives you a number of how many null values there are right and so we could see wow look of the 860
- entries uh only 750 have a first name right so that was like a little clue we've got some nulls like right there right and when you're working with the pandas you always you have to keep a
- careful eye on these nulls so how do we how do we avoid this how do we get George Soros back into the count Iris yeah so one way we can um take care of these nulls is there is
- a method called fill n a and you can put this on a column or you can put this all over a data frame it doesn't
- it depends on how you want to use it but right now I'm going to put it over this data frame so I'm going to tell pandas on the merge data frame I want you to
- fill all the N A values and N A essentially is um is blank but um
- it's like there's nothing in there but I want you to put something in there even like a blank text or blank space
- whatever and to do that I'm just going to say fill n a with a blank text so that it's just filled okay so I'm
- going to run this on the whole database and what it will do is it's going to identify these nulls in this column as well as these columns that have the nulls and just like fill it in with like
- empty uh strings empty strings and remove it from like the void basically so I'm just going to put that Command
- right before so that it's filled in and then I'm going to run the same exact thing so I'm going to group by contributor first name
- last name and committee position and then take the amount column sum it all up reset the index and sort values
- all at the same time copying and pasting like so if we run this
- now we can see sure Sean Parker still giving a lot of money but the second most is not or third most depending on
- how you see it is not Julie actually it's fun for policy reform right and you see if you then account
- for these spelling variations and kind of smooth them out on your own later you might see that if you took the different rows for fund for policy reform and added them together they actually gave
- more money than Sean Parker right and so just that little thing about getting of dropping a record grouping and Counting right made the difference between
- getting the story right and getting the story wrong right and so these tools are really powerful you can move really quickly but you got to be careful and you gotta like with each step just make sure nothing's going wrong
- and I think this really shows how we really have to interview the data like we would interview any person or source that we talked to you can't really take
- everything for as face value you have to know the limitations of what it can provide and possible errors inside
- um just because it's numbers it's not always very straightforward um well I think that's my part and we
- are going to take our findings and put something presentable together and it's going to be led by Andrea hi
- Andrea before we move on do you guys have any questions for us about what we did so far today
- I have a question that you don't need to answer now but if there's a resource you could point me to so like this example
- that you just pointed to how fun for policy reform has it's in there with two different
- um titles right I work a lot with school data and like I'll have two data sets and this the names are never written the
- same way so I have this issue a lot where could I go to figure out how to clean that up in Python how would you do it Iris
- um for me I guess um first I want to see how they're all written differently so I know that it would have some sort
- of like fun policy reform and maybe it has more or less things so one way to do that is to filter the data frame but on
- a very like a fuzzy kind of matching like uh on um pieces of the text if that makes
- sense so sometimes I'll look at my data frame and
- then we can do this command called string contains and then you can kind of put like a piece of text in there and it will
- return anything that's contains this text okay this is a pretty cool pandas feature is you can do little fuzzy searches
- so then um this has returned here we go now we can see like okay so
- this is a one with fun for policy reform this has returned this one and this has returned this types so I could kind of see okay there's like two different name types
- um that's just like for me to know and another thing you could do perhaps is um
- get all the unique names so I would call on the column which is contributor last name
- and then I will sort them or get all the unique values so if you
- think it's a new feature how about that but I could also sort them so I will sort the data frame
- first sort values [Applause]
- and contributor last name and then now it's kind of like sorted alphabetically so I could kind of like
- look at it and see like different variations like these two look like the same names but they have like a different variation right right so
- sometimes so sometimes what I do is I will put this into um uh an Excel spreadsheet literally just like
- all the unique names and then I'll go down each line and say okay for everything that says drug policy
- non-profit 50c Yes on Prop 64. I wanted to give it like a clean name so I'll type in a clean name right next to it and then read that database in and then
- merge it yep you can also use open refine to to Cluster and standardize names totally this is techniques we do all the time we call it cleaning the
- data or standardizing the data and iris's technique is just really it it's like you get the unique list of dirty names right you then put it into a spreadsheet into open refine you make
- another column and then you you standardize them so in cases where you want them to come out the same you give them the same value in that new column right you merge that back in and then
- you group using the clean column instead of the dirty column and now they'll add up because you'll have the dirty column in case you want to check your work and make sure that you didn't screw it up
- exactly this is really common with campaign Finance type stuff I think people have different slight variations on how to do it but it's really common one thing I like to do is when I do that unique output I like to count the number
- of contributions and sum them into the into the coding sheet because then I can prioritize the ones that are the most money because like let's say oh my gosh
- there's ten thousand names I'll never be able to do them all but if you do it that way you can say well we we standardized enough to account for 95 of the money you know what I mean or you
- could just focus your efforts on cleaning up the ones that make the biggest difference to the output you know that was very helpful
- and you could do that with all the techniques we learned here in class you would group it by the dirty name you would then use a write function which we haven't got to yet but that would write
- that out to a cdsv you then would do your cleaning in there you'd read that CSV back in and then you'd merge it on and group it hey Ben uh-huh
- uh we've been going for a decent chunk now it seems like maybe we want to give people a little bit of a break yeah we have the chart section coming up right with Andrea which we're about to break
- into what if we took a 10 minute break and came back and did charts would that be okay sounds good and and the other thing is like we can take a 10-minute break but if folks have questions they could they
- can also hang out here and ask the questions yeah we can maybe do 15. let's say we'll start at 10 40 or 12 45 Pacific Andrea will come in and she will teach you how to make some kick-ass
- charts out of this data yet too thank you Iris thank you we've got the
- the like fun visual part to come so it's like the dessert it is like the dessert after all the learning so for those who hang around if you're
- still into it there's a chapter we don't need to go through in class but it's the remix chapter if you were to now go back up higher in your notebook and remember when we printed out all the props and we
- then selected ours and named it as that my prop variable you remember that if you were to go up and substitute a different prop name the cigarette tax the bag ban the death penalty one
- substitute that name into that one variable and then rerun the whole notebook from the top you would get the exact same analysis but for a different
- prop right and that and that this is part of the power of the notebook right is that just by you know scripting it all out you can repeat it and you can do variations on
- it and remix it you know are the commands uh in in um in this uh
- in Panda are they if I'm using a text editor like Sublime text or something do I have to learn a whole other language are they pretty much the same commands
- you could run pandas in a traditional python script that is a text file it does not need to be run in a jupyter notebook environment so if you if you preferred to write like a traditional
- python script that was just in a flat file it would totally work it's just that each time you wanted to rerun the code you'd have to run it all from the top right you know so it would have
- would be one chunk of code that could only be executed like in a bolt which could be slow and require you doing a lot more like fiddling and re-running
- and losing time where the notebook is nice is that you can just work on that one cell you know what I mean and do it one little itty bitty bit at a time is it it's just for this is right really
- mechanical stuff but um like for fans of python crash course you know the text editor Sublime Text um do they all vary one from the other
- uh when you're text editors themselves yeah yeah I mean it's just like Coke and Pepsi it's like different you know variations on the same flavor got it
- yeah the big difference is if you get into like Microsoft Word or something where it's formatting the text and has like stuff like that that's not meant for code right you want these flat text
- editors that are just like you know text yeah yeah thanks foreign
- other questions
- hi can you hear me okay I just wrote a question in the chat um but I wanted to see if you could repeat the kind of best practices
- um um that you were talking about with cleaning like do you want to export that and then clean it up and then put it back in pandas or what do you suggest I mean Iris what would you say is the
- best way to do it if you were being really good about it I always keep copy of all the dirty data as is and then I'll just create a new
- data frame and then perhaps I'll match the new names into that but like Cheryl said I would still keep the dirty column I just like to keep everything right unless I'm presenting something so
- you're sort of merging data you're not overriding it right so you don't lose what you started with yeah and to me I think that's a good rule with Jupiter notebooks in general is that you're you're not you're not losing anything
- along the way you're just annotating and improving it you know Cheryl what do you think yeah I'm a hoarder I keep all the dirty stuff and and then also the the
- cleaned up file so yeah because I live in fear of making a mistake and that way I can always just check like did I
- really merge this properly did I really sum it you know the right things right right I mean Joe you're a professor now what would you advise well um I think what people said the
- biggest thing I think the way I think about it is that you want to be able to make sure that you can audit and reproduce it in case anything goes wrong both to explain it to your Editor to
- explain it to yourself in six weeks if you have to put the story on hold and so everything you can do to sort of make your process methodical and repeatable which includes not destroying data like
- always keeping copies and as is um Central to alike about it I I would add that and the nice thing about the Jupiter notebook is you're not but even
- then like when you if you save a a notebook like give it a good name structure right it's like you don't want final final Final on your name you don't
- want like five Untitled notebooks I'm guilty of that yeah okay you know the notebook is really uh this is I'm gonna reveal how pretentious I am
- maybe here but I think the notebook is like an emerging form of literature I'll be totally honest with you um and I think that the standards and structure for it is still fluid in an
- experimental State and this is one of the reasons I really like reading different people's notebooks because I think there's a lot of experimentation going about how to structure them you know like for me personally the thing
- that I've kind of stumbled into that I like is is um having multiple notebooks that operate in the sequence you know so rather than having like one huge notebook that has 8 000 lines of code
- and has like my all the little diddles I did along the way like strewn throughout there that's like the first draft and then the second draft is like 1 one little notebook that just downloads the
- data and writes it to a CSV like then a next notebook that takes that data and cleans it up and like does the tidying and then outputs a tidy CSV right and
- then like a third notebook maybe that then analyzes that tidy CSV and so that I'm able to sort of like separate the stages of the kind of pipeline of the
- data into little discrete chunks so that they kind of can like I don't know just make it it works better for me mentally you know I've been completely uh copying
- you in that band because I I yeah I think you've mentioned that to me before and it's so much more useful this is just easier for me to get my head around it
- I do see why it's called notebook now there's a lot of almost like scribbling of figuring out what you want to do it is for me I'm when I was a student I did
- have a class and learned a little bit about open refine and so I feel comfortable cleaning with that so if I were to start kind of querying data like
- we did today and then realize I need to group some stuff together I could put that in open refine and then do you recommend starting a I guess we'll learn how to for sure save a jupyter notebook
- here today but then I could start a new notebook and and have that clean kind of sheet and it would just be like the little Gap in the pipeline you know what I mean there's like there's the python
- part of the pipeline then there's your open refine part and then you pick it back up with the next thing yeah gotcha okay thank you guys the other thing I would add two more things and that is that um
- I I comment that I mean like I just I like really clear commenting right because I just think especially when I'm because I'm teaching when I'm teaching students like they may be their first
- time and looking at a Jupiter notebook I just want a lot of really clear instructions for somebody who might be coming brand new to it so they could treat it almost like a tutorial totally and so there's that and then I think
- there's the hey what's the final you know especially for the analysis ones like what's the published version what's the one that I'm going to put out there for anybody to access and kind of walk
- through those steps what that needs to be nice and neat less of a scribble yep and I I think to get to that point there's almost always like three at
- least three drafts of the notebook you know you know what I mean there's that that first one where you're just like uh what is this and thrashing and writing all this stuff and then you kind of like can break that into pieces so you sort
- of like Get Your Arms Around what you're doing but then there's this moment I think all danger journalists know where you sort of like I call it when you cut and run where it's like you've done all
- this fit you've looked at all this different stuff but like this is the story and like all that other stuff you know what I mean and like and to me there's like the third final version of The Notebook it's like I'm just gonna take out all the experiments
- and it's just going to be a pipeline that goes to the published work of what I publish you know what I mean and that it's until I get to that point I kind of hate my notebook like that's when I feel
- good about it Andrea how do you how do you set them up uh badly um
- I I always mean to go back and and re-order refactor I I try to do any kind of data set manipulation in one notebook
- and then the actual analysis and visualizations in a different notebook um if if there are like multi multiple prongs to like you know there's like a
- um you know something I'm working on for like for that's sort of like a fundamentally different analysis but for the same
- story right um I've found it's good to break that up into multiple notebooks um but I would definitely
- also emphasize the commenting thing not just for other people reading your work but if you come back to this notebook in
- six months you're not gonna remember why you filtered the way you did or groups the way you did um so using comments within your code
- cells but also using that markdown function where you can switch a cell to a text cell basically um and just like write down what you're
- doing and why um if there's a part where you're exporting exporting a CSV cleaning it in open refine and then bringing it back in
- write that down right um otherwise like it's a favor to yourself for the future totally I think you know there's a way in which you can feel like
- you're showing off with your notebook but you're actually helping yourself more than anything else I think that that's so true and I think a lot of the times those notes aren't really technical notes they're more like things from your reporting like you learned we
- have to exclude these records for some weird reason you know what I mean and like are you gonna remember that weird reason you know it's like it's really good to write it down
- other questions um I might have missed it but I had asked in the chat is there a way to keep track
- of the different variables and things that you're defining as you're going along and even just in the tutorial I was like did I make a variable for that do we have a data frame for that
- um so would you do that in like a separate area I can python just kind of spit out here's everything you've defined so far so python doesn't do it but notebooks do
- it there's a so we haven't really talked about magic but there's this whole class of things you can do a notebook called magix that are commands that begin with a percent so you can do percent LS and
- type return and it's just type just like you type LS in the terminal uh and you can do a bunch of others and one that you can do is present who who like who's in the room and I will show you all the
- variables that you've defined during the current that are currently defined in this session I don't actually use it much but it's kind of interesting but I think the main thing if you think about sort of the processes we've been talking
- about at the end of the day they'll all be there in your notebook and it's you can certainly like hop around and do things out of order which can be confusing but I think I saw someone
- mention in the chat like you kind of want to try and have discipline that says everything goes from the top and goes to the bottom so that if I do quit and start over and run it again it'll come out the way I want and so then
- you'll pretty much be able to see the variables and you'll learn some tricks about like white space and formatting and commenting to make that more clear so that who actually is not as handy as
- it sounds but if you're if you really are just kind of like bopping around exploring stuff it can be handy [Music] it's actually a pretty cool one there's
- a sequel magic so if you do the right setup stuff you can actually run that Ben mentioned and this is actually it's true that you can just do pd.read SQL and get a data frame from a database but
- you can also just do the SQL right there in a Cell yeah and that's that's kind of nice we might be showing our age with that one though Joe what SQL yeah I know
- well if you want to learn C so this is this is our fun uh people this took off way more than I thought of it but if you don't do Sequel and and you didn't get to learn a night card this year check
- out mystery.nightlab.com for a little fun SQL tutorial um that might entertain you yeah
- Hey Joe could I ask a follow-up question on that actually sure um because one of the things when I was first starting out with a lot of This was um just the like I I kind of also
- ran into that that weird thing where I get kind of like I like to see everything that I have like visually in the IDE um which is why I like loved our studio
- so much because it would it has like I think a little just especially if you're like getting used to the GUI like it has just like it's more visual uh is is
- there a resource of all of those like that percent who or percent LS is their resource of all of those commands because I feel like that would help me kind of like make the transition to a python notebooks or just like help them
- work help me work with them a little bit better I mean I don't think there's a resource as nice as what you're imagining um although actually the help in Notebook is pretty good if you notice um
- on there's a help menu uh that has a handful of things I don't think any of them is actually a magic reference yeah but there is like a reference for
- keyboard shortcuts which I like a lot I'm a very much a keyboard guy um so definitely know about that um but yeah I think mostly if you probably Googled for uh pandas sell Magic or iPad
- I think it would be IPython so the interactive python is the version that runs the notebook IPython yeah that's right that's right so I think I python
- Magics would probably get you the list um if you're a super debugger like oh my God I love debugging tools there's actually one of the newest things in
- Jupiter that's like brand new is there's a debuggable version of IPython I don't even know how to pronounce it it's spelled x-e-o-n Xeon or something and you could if you install this extra
- python add-on in your notebook you can create um a notebook where the individual cells will have like much more Trace backing and debugging kind of possibilities
- oh I didn't know about that one of the Magics is pdb yeah probably most people here are not python programmers yet but pdb is a python debugging tool and you can throw that in and kind of get it to
- stop when an error happens and do a little bit of an inspection without even installing anything else just um but that's probably not something most of you want to get into anytime
- soon because now Iris put a cheat sheet in the chat
- um as you were talking about structuring your different Jupiter notebooks is it a situation where you would like do one process start a new notebook and have to
- kind of input what you import what you did before to keep going in order to that's what it ends up having to be Yeah you sort of have to hand things off you
- know what I mean and like there's different philosophies about this but what I like to do is like have every notebook reads and files and writes out files you know what I mean in in that
- way it's kind of like functional programming in the old school way you know where and so like so to me it's like well my downloader is going to read from the web and write to a raw data directory and the cleaner is going to
- read from the raw directory and write to the clean directory and then the analysis one's going to read out of that is how I often do it but I'm sure there's other approaches that's just one
- way it's good to think about maybe folders where you put raw files and then like clean files a lot of different folders
- just to keep them tidy I guess I mean it's really up to you sometimes like my notebooks I like get to start with like zero one and then I'll put a name on it like clean up zero two I don't know
- analysis just so that it even looks like it's in order just for my like visual sanity you know and you can actually
- write meta notebooks that run other notebooks if you really want to get into it that's one of the magic commands
- um just just real quickly on that last Point um I was wondering if there's any kind of conventions or like standards in regards to how you should write and then
- call functions within a notebook um like should you write like a full function at the top and then call it later on or should you like write it in place where you're going to use it or
- anything like that I don't think I think people kind of it's the wild west you know what I mean like one feature we're not going to cover here in this class is the apply function which is a really common pandas
- tool and this is where you iterate over a column or over the data frame and then you you map every value through a function right so let's say you have like a a cleaner function or a data type
- conversion function and you want to run everything in a Cell through everything in a column through it right you you do dot apply and then you give it the name
- of the function and boom it'll just map them all right and so a common pandas pattern that's out there is you write the little function in the cell just like a standard python function and then
- you apply it in the next cell like you'll see a lot of people do that all right yeah thank you yeah there's cases where we'll we'll write the function in like a flat python file and
- just import it at the top too like you know if it's a really long gnarly function you know that might be nice or if you're going to reuse the function across notebooks like
- um so like you know our coronavirus tracker is just like 50 Jupiter notebooks that do a bunch of stuff in order but we have common functions for reading and writing files to the data directories so those functions like sit
- in a little flat python file and then the other notebooks kind of import them and use them you know Ben I've wondered about that so do you script the execution of that or does
- somebody want to watch those while they run just to make sure everything is going right it's a mix um uh we have GitHub actions so we have sort
- of scheduled tasks that run in the cloud that execute the notebooks and we've gradually moved to a system where the different data sources are sort of their own modules or apps so there's like a
- little app that is the data pipeline for the state prison data that Iris must of and there's another one that's for the HHS Hospital data or for the the
- unemployment data and each of those little modules will have a download notebook and a processing notebook right and then those those notebooks will get run as part of the module in the little
- scripted like cron and for the ones that are kind of like pure scrapers and set and forget we just let the computer do it and it'll howl if something goes wrong we put a lot of assertion tests in
- right like things to like make sure something hasn't gone wrong right but then in the case of our data entry for cases and deaths we actually um do that we we we watch that one
- because that's kind of the most sensitive one with a lot of human data entry thanks yeah I never really thought about executing notebooks automatically just
- because I know a different way to get the same thing done but I I can see why that would be valuable if that's how you start just to not have to read exactly I mean it's a whole world like running a
- notebook from a script or a scheduled task is a whole thing and it's actually pretty bad at it Jupiter by default is it cannot be run really from the terminal
- um there are huge Frameworks for uh scripting them there's one called paper mill which is just so overblown and Bloated it's like kind of worthless I think but I think that that's what runs
- inside of Netflix or runs inside of the marketing operation at A T or or credit fraud detection at Chase they have like these paper mill notebook like like
- pipelines but uh for us we just have like a little a little command line wrapper we wrote that'll just run them um but that even writing that's more of
- a hassle than it ought to be um okay I've probably drafted on too long there and I think we're we're past our break and so I would like to introduce Andrew suazzo from Vermont
- uh live and direct who is going to guide you through the remainder of our class which is how to install Altair and make beautiful charts take it away
- yeah um so we are actually at a little bit of a different spot than the first python notebook walkthrough has a set but it's
- actually great because it'll let us do a couple of additional fun things um without remaking a uh data frame so
- that's exciting um let me just get my windows set up uh
- screen there we go okay
- here we go okay can you all see my notebook screen
- okay all right
- so I am uh actually doing something that you're not supposed to do which is I didn't rename my notebook it's still Untitled so don't do what I do
- um because then you'll never remember whether you did your analysis and I'm
- Untitled Untitled one Untitled two Untitled three could be anything um okay so the first thing that we're
- gonna do though is um actually save your notebook and um pop back to and you can just close that window open your
- um terminal up again and shut down that uh that notebook
- server that you were running um
- so um what we need to do now actually is uh we need to oh
- um so to save it you should it should auto save but I usually just hit like
- command s before I close out a window and that should work for you um and it should also give you a warning if you try to close it and you haven't
- saved save um hopefully um so uh you should get back to your terminal and um and you can exit that by
- um I mean I know it's control I think it's Ctrl C Ctrl D on a Mac um probably something similar on Windows
- but just get back to your control C okay um so just get back to that um that uh
- uh command line where you can actually enter um enter commands and we are going to install a new library
- um so similar to what we've done before pip and install um and this is Altair
- and that is our data visualization that library that we're going to be using um it is a relatively new library
- um uh it will um it'll export um uh like interactive images for you
- um in in your Jupiter notebook that are actually like JavaScript um uh visualizations and we'll talk a
- little bit about the like like things you can once you have created them but functionally in our notebook they're going to be like basically glorified
- images um so once you have installed that um it should be ready to um go in your
- environment um and you're just gonna we're just gonna start back up our uh Jupiter lab notebook again
- um this just to make sure you remember how to do this because this is a good thing to do over and over again a couple times so you don't forget it
- and of course it's opening in the wrong Windows there we go Andrew we had a requests in the comments if you could bump the size of your browser font oh
- yes of course
- Okay so we've got a new uh Jupiter lab or we've got our old Jupiter lab uh notebook
- um and I have this fun little error that's not causing any issues but that seems to have to do with the new MacBooks but
- um uh so we have uh we have our import pandas um uh command up here uh you're gonna we're
- gonna have this these number readouts um which kind of makes it look like we have not yet run uh that like we have
- already read these cells into uh into your compute our computer's memory that's not the case
- um so if I hop down to committee list and try to run it um I get PD is not defined because I have not yet uh read the pandas library
- in here um so we're gonna need to run all of these cells again in order to be able to use any of the variables
- um and a handy way to do that actually um is run all cells in the Run menu
- um which will just go through and run everything um and I'm just gonna do that to read everything back in
- um but the one uh the one thing we're gonna add here um and technically we could do this anywhere in our document as long as we
- read you know as with any variable if we run the cell we have that variable to use anywhere in the cell um but usually we're going to import our
- libraries at the top of the notebook um so here I'm going to add an import line um and it's going to be Altair
- um and just like with pandas we're actually going to give it kind of a nickname um that makes it a little easier to type over and over again
- um and that is going to be as Alt um so when we want to use any Altera commands um to make a chart uh we can we just
- have to um use the alt shortcut um and then if you re-run that uh
- even if you've hit run all cells you rerun that we now have Altair that we can use um
- and I'm just gonna scroll all the way down um and so now I can use Altair anywhere in this document um we're gonna hop down to our
- um our merged data set here um and
- we uh we sort of ran it so that we could um pop it out into the cell see what was in it um but we didn't actually give it a variable name
- um and in order to uh in order to visualize the stuff in here we're going to want to give it a name um so let's call this top contributors
- equals
- I'm just gonna um also pop out the head just to make sure
- that what I think I saved uh to this data frame is in fact what I saved um make sure that you have those blank first names in there again
- um uh that you're using this fill-in fill in a to uh prevent that group by from dropping any of your stuff
- um because we definitely want to keep those in there okay um so we have all the all the
- um setup that we need to be able to look at um uh this in a way that is not just like a
- table um so what we can do is start main chart and to make any Altair chart uh
- the basic um syntax is alt dot chart the C is capitalized um and before we go any further we can
- just run that and see what it says um so the error we get here is that data is a required property
- um which we can probably guess means it wants a data set to actually be able to uh visualize
- so let's pop in our data set um I'm just going to run that again
- um so this time we get a different error and that error is Mark is a required property um
- what that means what mark means in the context of Altair um is how do you want me to show the show the numbers basically so Altair has
- a lot of different marks that are possible you can say like Mark Circle Mark line mark
- uh I think the map is Mark Geo um but in this case we're just gonna say Mark underscore bar
- um and it's it's always going to follow that uh basic syntax so Mark underscore and then the thing the kind of chart
- type to do um and then we get a shirt type and we're gonna run that and we got a nice little
- box um and so that doesn't look like a bar
- chart but that is a really good sign um because it means that it has a data set and it has a way that it's supposed to chart the data
- um and it is ready for us to uh to kind of give it more instructions it's no longer an error um but it's just sort of arbitrarily
- giving us a made-up like box because it doesn't actually have data to chart um and to pass in some data for it to
- show um we're gonna add on and encode function um and within in code we're just going to
- pass in like a list of the instructions um so this is where you're gonna give it anything it needs to plot
- um so the first thing we want to do
- um is give it the like most basic building blocks of a chart are usually going to be the X and the y axis um so in order to pass in what we want
- uh to tell it what we want on that X and Y axis um we're going to take a look at our chart and in this case we're going to
- look at um because contributor first name has a lot of blanks um we're gonna say contribute or last name
- we know that field is filled out in almost every case in every case because
- we looked at those uh we looked at that um data frame info um and then for why we need a number value
- um and in this case we're going to say amount and this is going to give us like a
- really really really long uh chart not really useful for anyone uh in any
- sense um but actually uh we have our we have saved this top contributor contributors
- data frame um sorted um so we already have it sorted first to last um and so we actually have a way of just
- showing the like those highest values in this data frame pretty easily um can anybody think of what we might do
- to uh tell Altair to only chart the like first maybe five rows
- uh the head command with uh uh parentheses yes indeed
- yeah so um so actually this is uh this is like a step before Altair so what we can do is just edit that sort of like
- pass in a different version of that Panda's data frame by just tacking that head onto the end
- of the data frame that we're we're telling altered a chart um so that's great we got our top
- contributors um and we got uh we got them sorted in
- an order that's not really very useful visually um but we're like we're we've got a good start uh if we wanted to kind of uh widen out
- what we're looking at we could actually type in another number into head and so
- head defaults to the first five um but we could say 10 and then it'll give us the like top 10 um values in that data frame
- um and then the other thing that we might want to do kind of before we look at sorting for
- readability here um is to change uh change where these uh change which axes are showing up
- um often especially if you have really long text um it's like kind of mean to make your reader uh like tilt their head to
- actually read uh read the important information on the chart which is who
- actually is giving this money um so in this case it might be maybe a little more readable a little more friendly to
- um the person who's going to end up looking at this chart um to just switch around X and Y um and as you can see here it doesn't
- matter which order you tell out there um to uh which order you pass in the uh encoding variables to Altair you can
- put X first you can put y first um uh what really matters is just like
- the why the y-axis is is the vertical axis the x-axis is the horizontal axis and what what column names you're telling it to plot there
- um so this is great this is actually um this is looking a lot closer to a chart that like we could maybe hand to
- an editor um but there's still some kind of uh visual um issues and one of uh one of those issues
- um I would say is that it's like pretty hard to derive
- any information from um a chart that is in this order um and the order that Altera is actually defaulting to
- um here is alphabetical order um so given no other information um Altair will always sort the y-axis by
- the Y values right and so the Y values are string values so it defaults to alphabetical order for
- string values um we don't actually want that because that's not useful I'm I don't like parse this information in alphabetical order I
- want to see who gave the most money um so to do that we have to use a little bit more kind of um an expanded Syntax
- for Altair um so it's very very easy um to just like pass in a column name
- um to Altair and it'll like try to guess what you want it to do um and like we have a pretty usable chart um but if we want to pass it additional
- information um we can use a slightly longer syntax um and once again we're using that alt
- shortcut that we declared earlier um so we are now saying um create a y-axis
- um and the first argument is always the column name um but then after the first argument we can pass in additional arguments to
- customize how how this chart appears um and in this case um we're gonna pass in a sort argument
- um so there are some like kind of complicated things that you can do with the sort Fields you can declare custom orders you can do sort of like um kind of uh like
- Things based on um you know things based on like groupings bins categories
- um but in this case we actually just want to sort um the y-axis by the x-axis values
- um right so we already have uh we have already declared the x-axis um so if we just pop in sort equals X and that X is going to be in quotations
- um it will sort by our access this is a relatively new trade it's like
- the year and a half old and it used to be way harder to sort a chart in out there um so this is very exciting still um and then to reverse that order you
- can just put a negative sign before that X um and we get our highest value at the top
- and so we know that we saw from sorting our data frame that um Sean Parker was at the top there so this is sort of confirming what we've already figured
- out um but in a way that gives us a little bit more of a like visual representation of like what the scale of these donations
- are um and again like we discussed earlier we do have Sean Parker in here twice
- um and that is definitely something that is not like ideal um but that's something that that you
- would want to sort of clean um in one of those ways that we talked about um and then you could chart your cleaned data sets um but we're good we're not gonna uh
- kind of touch that today um so the next thing that we probably want to do to get like a production ready chart
- um is to make these names a bit more readable um and so that to do that we're actually going to go back to pandas and we're gonna do something
- um that we haven't done yet which is to create a new column um so we have our top contributors column or data frame
- um and to create a new column um all you have to do is
- um tell it top contributors um whatever okay contributor let's say full name
- to distinguish it from first and last name um so we just give it a column name and
- then we say call this column name equals um and then we tell it what the sort of formula we want um is and in this case uh
- we already have uh contributor
- first name and
- what we can do is add in um so so to combine strings you can just
- use the plus sign and as long as those columns are both string columns and not like a number column that shouldn't give you an error
- it might not give you an error if it were a number I don't
- okay so we have contributor first name plus contributor last name
- um in cases where there is no first name this string will actually just start with uh space um which is not ideal but also
- um isn't going to have any impact on like the Integrity of this data so we're going to say we're okay with that
- and then you know we could clean it up later so if you run that you should have a new
- column um that has names that actually look a lot more um a lot more kind of uh
- like normal names for uh people who are not organizations um so we've got Sean Parker in here now
- we've got rich Goldberg um so that's good so that's that's kind of a more readable column
- and then I'm just gonna copy and paste what we've already done here just to keep the order uh
- just so I'm not like defining a new column and then going back up to an earlier cell and using that column
- um so here we can just Sub in contributor full name instead of contributor last name
- and now we have actual names um and so that's we're making a lot of
- progress and this is looking like much more like a chart that you might um kind of hand off to your editor once you have done an analysis you need to like kind
- of uh you need some visuals for your story um this might be kind of close to what you're looking for
- um but we do have um we do another column in here and that's committee position
- um and as we'll recall before actually we know that Julie's shower um was one of those folks who was not um not kind of supporting this ballot
- initiative um but she just want all the rest in this Visual and that doesn't tell us a
- whole lot about uh that doesn't tell us a whole lot about like what money is actually going to um
- to kind of which sides of this cause um so to do that we can actually throw another layer into this encoding um definition
- and that is just color one of the nice things about Altair is that like a lot of stuff is named very logically so sometimes you can just
- guess and it'll work Andrew before we do this we've had a question come up in the chat I want to just address which is some people are having a data type error
- on their contributor full name where the computer it thinks it's a mixed data type and I think this might be just a quick moment to show the Altair shorthand that can solve this
- issue so when you're providing that column into alt Y and you do contributor full name you can put a colon after full
- name and then the letter N right and that will tell it that this is a nominative category not a quantitative category and will I think solve that
- issue and so if you were to do it is that issue on the data frame or on the Altera chart oh I guess I thought it was on the
- chart I may have misunderstood um okay and you make the chart you're able
- to specify like Altair is guessing is this a quantitative scale is this a nominative scale is this an ordinal scale alterable guess based on the data
- but if it ever gets confused it has a little shorthand method where you tell it this is nominative data this is quantitative data which would be a queue for instance
- yeah and and that is a really important trick um that comes in when you're like um so you can import a data frame to Altair but you can also just pull
- directly um like a URL to um to like a CSV file that's stored
- somewhere else um but when you do that um you don't have the sort of pre-processing capability that pandas adds um so if you're pulling in a data frame
- or if you're pulling in a CSV from somewhere else Altair doesn't do the work to figure out what columns are are what types
- um so if you're doing that you do actually have to declare the column type um every time um so you know as as a sort of best
- practice um it is good to get into the habit of that so when you run into it um you do sort of remember so let's let's just throw in a colon and a Q on a
- mount um and that that's a quantitative value the n is for nominal so like you're categorical
- um or ordinal um value we'll go into that is just a string um there's also T which is temporal
- um there's one other but I don't remember it right now um yeah so if you do run into an issue
- where uh where Altair is giving you like uh uh um yeah encoding is special specified without a type error
- um just remember that if you declare the variable type that will usually resolve that um great so then we have our uh color
- um encoding and we just want to color it by the committee position so let's pop that
- in position
- and so now it'll just pick arbitrarily from its default uh color uh color set um it'll just like
- assign one category to um uh to one of the colors you can override that um using those sort of more
- um the using that Altair um kind of syntax that gives you some more options um but otherwise it'll just assign it a
- default thing um and as you'll notice it'll also give you that little Legend there
- um and uh so we're like we're like 90 of the way there um
- we could uh we could add a um properties function onto the end of this chart
- um and properties will let you access the sort of like more higher level um higher level kind of functions
- um like assigned to the Chart so um not specifically like data encoding related but for example
- um the title of the chart um and let's just call this um top contributor your title case
- um [Music]
- 64. so if we were actually printing this we'd probably want a um title that kind
- of uh gives people a little more background on like what proposition 64 is um but in this case we'll just go with that
- um this area is also where you would specify a custom width so if we want this chart to be like a little bit wider
- 600 look at this um and that that will actually um I believe alter the width of the
- chart so then anything else will kind of be tacked onto the sides um and we could do the same with height
- say 500 nice big chart there um yep uh no just uh sorry um yeah first
- yeah absolutely let's shipment um I just want to kind of uh we've got a couple folks kind of run into some issues where I think like uh
- it's it's important to note that like uh even though like this was uh Andrea's like like see it looks like she's starting a new section here essentially like you don't need to like start the
- process from scratch on your ends um and we should have like made that super super clear at the outset essentially like like you should you
- know the whole point is uh or super helpful to keep like that terminal window open that we were using before and keep like everything in place essentially because since we're working with a the end or virtual environment
- like if you exit terminal or close terminal window you essentially have to redo a bunch of steps and so like in order we
- should have explicitly said this but basically like you wanna you wanna just rely on the same stuff that we had already done earlier and not close stuff out and start another start another session you don't need to do that
- um I I also wonder maybe uh when you're
- when once you as you said ship it like once we get there like I also kind of feel like just maybe a recap of like the steps that we took might be might be
- useful for folks uh for for the chart specifically or the for like everything the chart the charts
- um yeah I think but yeah unless somebody else wants to speak up I just think it might be a little helpful yeah
- um yeah so we yeah we started with this this top contributors data frame um and we made this like kind of Fairly
- fairly straightforward um pretty kind of like chart but we're
- um again truncating our our data frame um a little bit so we're not showing every single data point we're just showing kind of the top things
- um we did uh decide to make it a bar chart um but I can't spell line uh we could
- just we could have made a line chart doesn't really make sense in this context but that is where you might say like um The Marked circles
- um so we made it we decided to make a bar chart um and then once we had decided on those sort of like uh overriding
- um values we just had to give it some data um and that's basically like passing in
- those columns um to uh to like where we want Altair to show them um so we picked uh a name for the
- y-axis um which is the vertical axis and the uh the amount donated to the
- x-axis and again you can you can play around with that so you can always switch those back if you have a reason to prefer
- those values on a particular axis um and then we did um create that new
- column name um so we are actually now using um the kind just the full name rather than the last name
- um and that gives us a little bit better kind of a little bit of a more um like readable um visualization
- and uh and then we added in color um to show whether someone was actually donating that money
- um to a pose or support um the proposition um where we are maybe interested in Julie's shower so it's interesting to
- kind of maybe show her in the context of the top donors um and then
- um we can hit this little uh this little menu um that's always going to be at on the right side of your
- um of yourself entire chart um and we can export um our to another format so
- um to say you can save it as a PNG and that would just give you an image um you can say that as an SVG and that
- would let you open it in Adobe Illustrator and make some changes um Altair almost never um Altera doesn't always export give you
- sort of the like prettiest defaults um so you can uh you know you you may not want to go like all the way to fully
- mind in Altair um there is a lot that you can do to kind of pass in additional styling things but sometimes it's easier to just pop it out and edit it in
- English the colors to your house colors um you can also open um this in the Vega editor
- um and this is where Altair's power like really comes into play so Altera is actually a oh my mouse just died
- um so Altair is actually just like an overlay onto a JavaScript library called Vega um or Mega light
- um they're sort of two um two different variants of the same Library um and and the Vega specification is and
- you do not need to know any of this um really to to use it um but the Vegas specification is in um
- Json which is a JavaScript which is Javascript object notation um and it'll you can you can kind of go through a Vega chart and see it setting
- all those things that we did in Python so we can see um uh you know it the mark Barr
- um that encoding the color is a nominal a categorical variable um that acts as a quantitative variable that Y is a nominal variable we passed
- in those customs and uh and width and title things you can actually edit those here and
- um see those see changes so um we wanted to change that to 65 we could and it would show up right on that chart
- um but from here you sort you actually have a JavaScript thing you could um you
- could pull the Vega um library and put that on the page um with this Vega light configuration um and you could have a chart
- um so uh if you if you wanted to do like an interactive chart you could do that from here
- um and you wouldn't have to mess with any of this stuff because you did it all in a python notebook um
- but to that end um you can actually so because you can actually um end up kind of exporting to a uh a JavaScript
- um like thing that you could embed somewhere else um we can add interactive elements to this chart that we have now um and the sort of the thing that is
- really nice for exploratory visualization um so like when we're just starting to look at the data we just like want to
- know what all the sort of uh columns associated with Sean Parker's organization are um we can add another encoding called tool
- tip um and then into that encoding we can pass any of the column names that we actually want to show on a tool test
- um so we'll want to do the full name contributor full name we can't actually see the full name uh in this chart right
- now so that uh so that might be kind of a way if you have really long strings um sometimes that's a good way to
- um see them if they're getting truncated in your chart um amount and then we probably do also want
- that committee position and so when you run it you shouldn't see
- anything change but when you hover over each bar you will now you should now see a tool tab um and you may also uh one thing that
- does crop up is that sometimes the tooltip command for some reason has a harder time um like assigning variables to
- um two types so um if you're running into weird errors there or overall generally good practice to
- just always throw in the type of variable um and there are also some way there are also like ways that you can format that
- tool tip so you could pass in um kind of a formatter function and get the amount with like a dollar sign and commas for the thousands
- um which makes it a little easier to read but um in this kind of quick quick um run down we don't necessarily need
- that um and then the last thing is that if you um if you have this chart and you really just sort of want to make
- you want to put it into Data wrapper you don't actually you just want to use Altair to like look at your look at your data kind of uh
- uh at first but you don't you don't really want to go on and and kind of export the Altair chart and put that into your kind
- of send that to your graphics team or if you're the graphics team you don't you don't necessarily want to kind of throw that up on your story
- um what you can also do and we just if you recall looked at the top 10 um contributors uh so
- um what you can do there is just say to CSC and then give it a name file name like
- top contributors dot CSV um and if you run that it will save out
- a CSV and actually we can check that out um in this in our file browser you
- should now actually just see um uh CS just saved out um we can open it too
- um it does it will save the index um the index by default um that those uh
- those index numberings that we don't really care about um you can turn that off by passing in index equals false
- um then we get a new CSV um great so so we just exported that to a CSV from here you could grab that CSV
- and put it into Data wrapper or flourish if you want to um kind of if you if your Newsroom uses
- a specific um tool and you don't actually want to use the Altair um chart if you want to uh export that whole data set also you don't need to
- just um just output that uh head um command you can just grab the whole
- thing and that will give us everything with the new column we've added this is a grouped data set so we have kind of manipulated it
- um uh so that will give you something that you can kind of then go on to your next step um with um or you can take this chart and uh pop
- it up on your website um and of course you can also uh like actually go to your
- erasing that there we go um so you can actually go into your finder and find that CSV file and that
- is saved out for you to open in Excel or anything else um and yeah I think that is uh that's kind
- of the the quick intro to Altair um there's a lot of other stuff that you can do with Altair
- um and uh uh and their documentation is extremely good so um it has like
- um uh so they have like a ton of
- um examples that you can just pull from and pop them into your charts into your um notebook you can make maps
- um but basically anything you might want to do there's there's going to be an example that'll get you like at least 60 of the way
- um and uh and then you can experiment from there
- um so I think that is all the charting that we've got questions about Altair I have a quick aesthetic question
- um so on the X and Y axes just the labels there just by default the variable names with the underscore is
- there and you change that without having to change the variable name or is it just easy to export it to illustrator or D3 or something like that and work with
- it that's a great question um yes it is um um I think it's just title yep so we can
- just add title and if we like this is top contributors we sort of know what that y-axis is we don't necessarily need
- to say that again um so I think we could just say title none it'll remove it um if we wanted to label that x-axis alt
- dot X um
- contribution amount um then we got a nice much more uh readable
- um we can even do the same thing with with that uh label so for the legend
- um amazing thank you just one quick follow-up um so the data set
- um are we just not is there no risk of the negative amounts because we're assuming that any negative amount is also supplemented by a
- positive amount for the return donations or is there a risk that the negative numbers could alter um
- the the chart somehow depends what you're trying to do with the analysis I would say in most cases you do want to subtract the refunds from your accounts because because the money was given back you know I so I think for
- the purpose of like who gave the most I think that generally most refunds you're going to want to x out you know allow to cancel out I think where it could come
- in is like let's say you're doing a count of contributions right that would it's actually two records in the table and they they cancel each other out but they would count as two checks if you
- just did a standard count and so it kind of depends on what analysis you're trying to do right and then if it's a gigantic refund you might want to figure out what the heck happened you know and
- so I think when you really dig it into a prop you're probably going to want to look at those refunds and really get a good grip on what the heck was going on you know yeah that makes sense thank you so much this was amazing thanks to all
- of you the other thing I'd add there is just like
- knowing as much as you can about how the data set is created um that's not a technical thing but just like making sure if you do see refunds
- that you know why those are in there and what they mean um is really really interviewing the data and interviewing the people who know the data yep
- and there's five or six things like that about campaign finance that every campaign Finance person has to learn and that nobody really tells you you know the UN itemized is a big one the refunds
- is another right um there's a few other things kind of in that ballpark as well um like a common thing in California is
- that committees will will pass money around between committees there's committee to committee transfers which some people call the washing machine for good reason and there's a way in which
- when you're analyzing a proposition you can double count money if you just look at incoming money to a a committee because it potentially could have come from one committee and then into another
- committee and to count it twice would be to double count the actual dollars right if it was an internal transfer within the sort of cause so there's things like that
- that are bugaboos that are out there in every state and in every system um another thing in California is a committee can support more than one ballot measure or oppose more than one
- ballot measure like a union might raise a bunch of money for two or three ballot measures it's very common and but it's only one piggy bank and so it's actually impossible to know how much money was
- raised or spent for any of the individual propositions because they're just like in one big kind of fund and so there's so like when you're trying to rank which prop got the most
- money or whatever it's sometimes actually impossible due to flaws in the accounting system aren't you glad you asked
- sorry to go on and on what else do folks want to know I feel like I want to thank all my other teachers Iris and Andrea our coaches Cheryl Joe and Derek but I
- also just want to hang on the line and take any and all questions that anybody has um because you know that's why we're here what do folks want to know
- hey um just a couple quick questions first off thank you guys so much this was um a treasure I really appreciate it
- uh because I noticed there was myself and I think of some other people are having trouble um maybe not trouble but like switching from the terminal editor and then back
- to yeah the python notebook can be kind of a two-step dance uh I I noticed in the bottom left and it's on mine and it was on yours when you screencraft there
- it seems like there's a way to open up a terminal from the the python notebook there is you're right there is so are we able to then install like if you didn't
- want to do that song and dance of like going to your outside Terminal which you can obviously do or something could you install your python modules from that terminal so you could just like keep it
- all in one space you know I actually think you could Iris does this more than me that that would work wouldn't it yeah
- oh sorry you can also maybe we should teach it that way in the future I just have never been a heavy user of the terminal in the notebook but I
- believe that you're correct and that might be a better way to teach it to be honest another thing you can do and this is this is applies for people using Google collab categorically is
- um you can do a command in the cell that begins with exclamation mark then everything after that is that as if you typed in the terminal it's called a shell a camera with the way the title is
- but so um so in collab the only way you can install their libraries is do exclamation mark pip install you don't have to do Altera because it's
- already in there but the ones that don't come already in collab you don't do pip and you do just pip um but the way that works you could do the same thing in the exercise the way we taught it today where you do
- exclamation mark pip and install Altair and it would run and later you'd get an error that it didn't need to run because it had already been done and you could delete that but um that's another
- way to sort of avoid switching between the notebook and the terminal um I think both of those are great ideas and they show people taking advantage of
- some of these features um I think you know the way we've done in the class I think is really just a reflection of of my personal habits and aren't necessarily always the best or most
- direct way though I do think ultimately there is no escaping the command line and you have to kind of make your peace with it yeah it's it's good to just get comfortable with it they're um
- I feel like I haven't seen them well I shouldn't remember there being some decent tutorials around like getting comfortable in it but I'd have to go hunting for references
- um sorry go ahead Derek I can ask you later okay I'm sorry um I was just curious so we've covered the pandas and and Altair are there any
- other kind of data analysis packages that are just really well known that we shouldn't be aware of that we haven't really covered Andrea what would you say
- um wait sorry I was I was he was asking what are other what are other like important python data packages people should look into besides the ones we
- covered hmm I think I mean some of the ones I
- actually some of the things I use the most are like um there's like busy data and um like CSV kit things you can use on
- the command line to like check out your data and kind of um uh examine it before you fire up a whole
- Jupiter or notebook um and often like if I just need to vet a data set and it's like not it's
- crashing Excel or whatever that's that can be a really useful way to um kind of check it out before before you get to the kind of analysis stage
- um and run some some summaries and do a little cleaning sometimes not usually great to do cleaning on that
- end so I think there's things related to web scraping are really important we didn't get to so the requests library to just grab a URL and the Beautiful soup
- library or some of its competitors to like parse a web page and pull data out of it those are both really crucial learning scraping is huge um on kind of the high end this is stuff
- that I I don't really teach and I'm only a newbie using really I think is going to be stuff like the natural language processing and machine learning toolkits
- that are in Python like the nltk and scikit I mean they're the The Cutting Edge of Open Source when it comes to you know artificial intelligency kind of
- stuff and you know we've used them from time to time actually to clean and classify campaign Finance data so you've got a hundred thousand donors to the Bernie Sanders campaign or whatever it
- is and you want to merge their names or have a computer guess what industry sector they work in you can train a machine learning thing to go over the data and kind of make its best guesses
- and so like that would be an example uh building off what we did here today I would also say that rather than looking for more libraries that do like
- advanced stuff like getting better at python fundamentals will really ask you a lot because a lot of the stuff that were like somebody mentioned this slack like renaming uh or reordering the columns can be a pain in the butt and it
- can be but if you know a little bit of python it can actually potentially be really simple and I think if I recall I shared a link to a census uh notebook that I used for a workshop last year at
- night car and just as part of doing that notebook like I used some of those kinds of tricks in part because I wanted to sort of show people that you know once you sort of understand how
- python flows you can really do a lot of things much more efficiently than you think you could and that's probably more important than fancy so like being able to write a little one-liner that gets
- all the columns Resorts them and then applies that back to the data frame right absolutely I had a question um about kind of organizing your
- notebook if you have a very long notebook maybe you have some functions in there is there a way to separate and
- like link notebooks or organize it that way yes so a notebook can run another notebook using the the magic commands we talked about earlier so like percentage
- sign run followed by the path to a notebook allows one notebook to execute another so I've seen examples where people will have like a master notebook that executes a whole bunch of them kind
- of in a string right it's like one way to do it also you know where we were importing these third-party libraries at the top like pandas and all pair but you can import your own files and so one
- thing we'll often do is The Notebook will live in a little folder and alongside it will be like a little utilities file and the utilities file will have a couple things that we're going to reuse across notebooks and
- we'll just import that at the top of the notebook thank you
- um so we we covered what the um with the markdown uh did and what the code did obviously what happens with the raw uh in the drop down
- menu I can't say I've ever used it do you guys have any utility for the Raw cell sometimes when I um write a code instead
- of commenting out I'll put in raw but because it's just like text and you can just dump it there yeah
- okay thank you
- other questions Katie yeah um and this may not um be
- I don't know it might not have a simple answer okay so I'm like a little bit hesitant to ask but like you mentioned that like a lot of campaign Finance
- documents are still on paper and so like if someone who was like a crazy person wanted to go and put those in like to um a database themselves would you use
- like SQL or is there a way to do that in Python um I'm just like I'm curious what that would look like this you mean when you
- say this crazy person let's let's discuss them a little more so this this crazy person hypothetically would be key punching them in manually or how would they be getting it in
- um yeah so that's kind of what I'm wondering what's the most efficient way to get data that's on paper or PDF into um I mean this is a big problem there's
- a bunch of strategies I bet Cheryl has some ideas uh well I mean you can you can try to use tabula right to extract it if it's in some kind
- of structured format um so there's that I mean I actually think Derek will probably have more ideas than I do because he messes with this all the time with Precinct results right
- yeah you um you do the data entry or pay someone else to do it I mean like like the the
- you can there are lots of different OCR libraries like uh that will give you some text like it depends on how texty it is or how numbery it is if it if there's a lot of text involved it's
- probably worth your time like uploading it to like document cloud and just seeing how what kind of text they dump out just to see like if it like that's a
- that's like a free OCR service for you basically if it's like a PDF and you upload a document cloud uh but like regular OCR itself like uh
- you know whether it's like free or paid will get you like you'll you'll still have to go and like verify you know go and do either spot checks or clean up
- anyway like there's no easy there's no easy route here so yeah yeah AP just released something at the conference uh I was just gonna paste it I just did it in the chat
- collaborative yeah Harvester and I'm gonna check this out because it is something we do a lot of right because I think there is an element to I mean what I like to call it
- focused data entry because I want to emphasize to our team and to others that we're not just going to enter everything on the planet you know what I mean that it has to be kind of focused
- and so to me there is kind of an art to a certain type of data journalism project where you're pounding into a spreadsheet or a database you know what I mean and um I think limiting the
- amount you have to do is key like you know you're gonna say like Iris worked on this one about we we put in all the um local investigations into police
- shootings from these letters this unstructured letters from the DA's office and to get it done it's like we're only doing four Fields guys you know what I mean and like you know and
- if the and we're not going to try to you know you try to limit how much you do and then I think if you do that and then you you pool the work among a large
- well-trained group of people um and you do some pilot testing where you kind of say well if we do five or ten of them how long does it take you like clock yourself you can kind of
- budget out how bad it's going to be and then kind of make a more informed decision about what to do but I think if you set that up I think a a well manicured Google spreadsheet can get you
- a long way and then I think beyond that like a tool like Django which is a python database tool that allows you to create an Administration panel with a very
- small amount of configuration can be the thing you send people into to like pound it through and have a little more validation and a little more organization to it and so for us those
- are kind of how we approach focused data entry and then your pandas can just read out of that you know what I mean that dumps something that goes
- out into the analysis I mean we're using Django for the California reporting project data entry or a police disciplinary records about that then the other piece of that
- members of the team the reporters can access the actual documents in document Cloud so they can they can use the data that's entered as a as their Tip Sheet
- and then they can go into the documents to do deeper reporting and I always say you always want a reviewer like a main entering Princeton
- and then like someone else to review just to ensure uh accuracy right I'm really glad AP is doing this because I think that it is it's almost like a discipline or like a type of product
- data journalism project that we haven't really kind of like formalized maybe as much as we should or like thought about what are the principles and best way to kind of do this sort of thing you know
- but oftentimes I think some of the most impactful work is that way because it's data that's being hidden you know or fractured I think there's sometimes there's an
- advantage to having I I did a big Elder Care Project where all of the reporters on the project were
- doing data entry as was I um and every having everyone know how the data was structured and having everyone like reading through these
- massive PDFs um so they knew what the interesting Fields were like it took so much time but everyone asked better questions of
- the data because everyone knew how it worked I wasn't just exporting a pivot table and like giving it to them yeah
- that was an air table you might have to be crazy yeah the
- uh data entry project would you have to add some sort of disclaimer explaining that there could be errors because the data was manually entered or what's the
- process behind that I mean you try to be transparent about how you did it and you try to make sure you minimize those before it goes out
- but to us it's just kind of yeah here's what it is there's here's how we did it and you try to you know that is why you do double entry though
- so that the double entry then you you flag the discrepancies and then you go back and check and see what's going on so you can try to get a pretty high level of accuracy yep and then a lot of using say a tool like pandas to like
- turn the data 10 different ways and like is there a weird outlier is something that makes sense are things what you know kind of what they should be you know really can help
- but in short there's no shame in building your own database and in fact if you build the database that's a sign that you're probably going to get a scoop because nobody else is totally worked yet yes and there's no shame in a
- great database and great story having a small imperfection you know we have to live with that sometimes
- I have a question if I had one of the weird errors from earlier uh to get Panda to run what should I do after this
- to try and get this was the pandas pip install thing with the weird numpy warning yes I think your your teachers are going to
- do a little research to see if we can replicate this bug ourselves and uh and and come up with what the fix is my plan is when we get the video up on YouTube
- to send an email to everybody who signed up for the class you know what I mean and if we can figure it out by then I'll try to include that message I'll try to include what we think is the solution in
- that message does that sound okay yeah that sounds good if you figure it out let us know this is a new one for us okay a couple folks uh on the side that uh
- news nerdy is like a place where people in our field help each other and so um also like just jump on there and I mean depending on how in depth the
- problem is it may be take some time to sort of schedule and find someone who can help you out with it but there's a lot of people who like to help there right yeah I bet if we asked in that python Channel somebody might know and
- and what the other thing I would say essentially like I encourage you to I mean it's frustrating for sure but I encourage you to try and figure it out essentially but when you do that like
- the best help the best thing you can do to help yourself is to keep a log of what you tried and and like every command you tried
- just copy it like when you enter it copy it into some other document or whatever just be like I tried this in this order because that will actually be meaningful to like like the reason why a lot you
- know like we weren't always able to solve stuff today is that like well one reason like we're not sitting in the same room and I can't just walk we can't just walk over and look over your shoulder to see what you're doing the
- other thing is like like despite like like it's hard to get like full context remotely um and so like there are things that
- like even despite like your best efforts and our questions like we're still gonna miss those things we're not gonna get and so the best thing you can do for yourself and for others is to like be
- like here's exactly what I tried and this is and then this was the outcome and then I'd be like okay well I I now I can now sort of try to replicate like
- what you went through and see if I get the same outcome right and so like that's helpful totally and there's actually a terminal command that can help with this there's one that's called
- history if you just type history and hit enter it will spit out your last umpteen number of commands in order and you can see what you ran
- you can even analyze it see which commands you run most often other questions
- maybe make an Altair chart out of it there you go who am I number one with the bullet I have a question about go ahead
- I have a question about installations so it seems like you know pandas and alter those seem like things that we have to install every time we uh fire up a new
- Jupiter notebook but what about Jupiter lab itself is that something that is now on my computer and I can fire up a new one anytime I want or is that something I need to install every time it's only
- inside that one virtual environment okay right so uh so when you when we did the Pippi NV install it's sort of creating that little self-contained module which is local to that folder so the folder we
- made first python notebook is sort of secretly connected to like a a a box of installed stuff that's like Elsewhere on your computer and so if you're in that
- folder and you do pippy and B run it's going to have access to those things if you're in a different folder it's not and so the way we taught in class today every time you started a new project you
- would do a new Pipi and B and a new set of installs right got it which is going to result in having maybe a bunch of different versions of the stuff on your computer it's my view that you're better off in the long run if you do that
- um people prefer to have like a global installation and that is possible if you look it up you can find ways to do that where you you install one Jupiter lab
- for forever um some people will make one repository that just has a hundred notebooks in it you know there's different ways to organize things and they're not right or
- wrong but I'm just a big fan of trying to because we have to juggle so much you know like in our Newsroom it's like you know in the course of a year I might have 50 to 100 repositories you know and
- so it's nice to be able to just have things be discreet you know yeah Tupperware method yes yeah
- like it does like if as Ben says it you do wind up with like multiple environments like laying around your computer but the which you might seem like a little bit of a mess but the
- reason the only reason like not to do that is if you trust yourself fully to do the right thing with your computer all the time and I've been doing this
- for 20 some years and I'm not even close to that point yet like I literally do not trust myself enough to do system-wide installs and
- depend on everything working like uh you know it creates a little sandbox for you and that sandbox is so that you can screw things up and not kill everything else you've got going on right but I
- mean then these tools have a bunch of bugs and they suck and that's like the downside you know so it's not perfect either and I think this is why to be honest with you I think this is a reason why R has is really popular with a lot
- of beginners is because you download it you double click it you got the thing you know what I mean and the thing's there right and like I get it you know what I mean that's cool like my dream is that that's going to be
- in the cloud is that the next generation of tools like Google collab and Jupiter are going to be like part of your GitHub account and you could make a new repository in GitHub and boom just have
- a notebook like right in there that's kind of my dream I think we will have that in in the next decade or something but we're not quite there yet well I'm where you I mean like but Simon Willison is doing with data set is you know is
- pretty interesting and we're actually integrating that with big local news so that people can just do their analysis from any data in that in a whatever
- project right there right they just open it up they do their analysis they share the URLs whatever their analysis is with whomever like that I so I think that's an example kind of what you're talking
- about man yeah phase I just wanted to say that Ben is right that things have bugs and they can
- be frustrating but also all this stuff is really methodical and it does what you tell it to do and so somebody said thanks Wizards in the chat and I appreciate that but I really want people
- to not think that this is magical and not think that when things don't work it's because they're buggy and it's frustrating and you know those of us that are teaching have been doing this for a long time
- but it it bet you it pays off to like recognize as methodical and to try and sort of develop the patience and you know and we like to help people because we know we got that help from people
- before but um you know it's it's a systematic thing yeah and I really think you know our work is getting more and more specialized and more and more advanced
- and I think that to really be a data journalist who is the specialist who does this stuff you just have to learn the terminal you have to learn this it's like there's not really a way around it but it's true
- that it's not wizard Wizardry right it's just a matter of like it's just learning it and it's kind of like you know you'd be a data journalist I think kind of like a crotcheted copy editor right they
- know all the rules they know all the things no you have to spell them up you just spell it that way well we just have to learn you just learn the various code which I'm not very good at but yeah I
- can also Google it and find the answer and I you know I can get there other questions and like yeah feeling frustrated and running into errors is
- like it's not because you're not good at it it's because it's part of the thing like I spend like half of the time I'm
- trying to do anything really frustrated and frantically Googling um but like that's sort of part of it and you get a lot better at Googling for the right
- things um and like troubleshooting um but it's not supposed to work perfectly
- yeah so it's not it's not a reflection on your your skills necessarily yeah you said that I did love to see to be honest with you data journalists take on the
- responsibility for improving these tools you know what I mean rather than just always being open source users being open source contributors and maintainers and helping you you know we've seen that with Altair and other things
- CSV kit invented by a journalist so I'll tell you another tool that we're we're building if you're interested and that's a fips
- it's a fips tool that allows you to um if you have a bunch of County names you can add you you use our fips tool which is not yet deployed but it will be
- uh and it will add in all of the fips codes for all your counties so so like yeah just little tiny things that will make it easier
- you got a name for it I can't remember the explosion kind of like fipsomatic that's kind of fun
- we have another one called the labeler we're working on it's like if you have a bunch of things that are like he's got like cherry banana broccoli you start
- create creating a label sorry fruit vegetable fruit vegetable you do the first 10 rows or whatever then you run that tool and it will auto
- populate and you still want to go machine learning for the masses that's the idea we're not there yet but we're getting there we're close
- sounds like Stanford anyone else have questions uh yes uh is it okay if I share my nut book with you and ask questions because
- I cannot find where I took a mistake um my my table uh doesn't seem the same
- as yours and the uh if I uh when I check are you in the the news nerdery slack uh yeah yes I I've just joined the you were
- so your notebook is just a file like any other file and so if you were to join our Channel which is fpn nikar 21 yes and just upload your notebook into there
- I think we'd be happy to look at it and give you some feedback thank you other questions
- uh if no one else has had any others uh if I could be greedy and maybe ask one more uh please as long as long as no one else had one uh then I know you
- mentioned earlier um that and I went to uh the demo that um Simon Wilson and your colleague Ryan Murphy did on get scraping which was
- brilliant um and really cool and they mentioned something that you mentioned that uh you run a lot of the Python notebooks some of them like automated
- through through goodscaping um totally okay if this is not open source or Publishers you guys already give so much I feel like to the community is it are is that code because
- I'm I'm trying to maybe eventually work in like automated runs of these of these totally python notebooks is that open open a small portion of it is and I hope
- to gradually make more of it so we have kind of a demonstration repository on GitHub it's called California coronavirus scrapers and it has two Jupiter notebooks that are that have
- Tableau scrapers that are run via the GitHub action um the notebooks are pretty standard scraping with the few Tableau Smash and
- grab that kind of hits um the trick of of scheduling it is uh is running the notebook from the shell which is just much much too hard and one of my personal open source goals is to
- help write the notebook shell script Runner I want to make that happen um but in that repository you'll find a file called run.py which is basically
- the file we used to call and invoke a note from the shell and run it which the action just like literally just runs that one line of code to run the notebook basically
- what you see in there is exactly what runs inside our system and I actually want to gradually move more of them out there because it's also free yeah and we can then we can then pull the data in
- our system without the cost either yeah just really really cool stuff like definitely gonna help me um a lot so I really appreciate it I think I think that in the last year
- we've seen a lot of GitHub action Innovation blocked up inside newsrooms and I one of one of my goals for this year too is to help try to open that up and see what other patterns people have and learn from each other and my hope
- would be is that the next night car we have a class called first GitHub action or first GitHub scraper and we we teach that as part of a class would be my hope
- any other questions
- okay hearing none I'm going to adjourn this call I want to thank again my fellow teachers Iris Lee and Andrew suozo our coaches Cheryl Joe Derek Sean
- green who's out there somewhere it's been helping people along the way quite modestly as well hello hey Sean and um thank you to everyone who came and and you know spent your time on this
- I hope it was worth it to you remember the whole class is scripted so you can go back over it again if you have thoughts on how we might improve it or change it for the future we definitely
- want to hear it you um you know uh you can feel free to email us or message us on news nerdery um we want you know this class has evolved quite a bit the last few years and we want to continue to improve it to
- make it better and better and you know your feedback is the best way to do that um thank you
First Python Notebook
By Ben Welsh • • NICAR21 After Party in Zoom