- Hey guys, welcome back, thanks for staying, thanks for sticking in.
- So before we get going, what I had planned, and we can always freewheel and freelance
- and switch it around, is a pretty serious coding man.
- We're gonna dig in, we're gonna write some code.
- There's a whole tutorial I have planned for us to work through where we're gonna do something
- called web scraping.
- But before we get going, I just kind of want to assess what kind of experience people have.
- The lesson is put together so it assumes absolutely zero experience, but I want to see kind of
- what's going on.
- So if I could just ask a question and people raise their hands, that would be awesome.
- So who here has done programming on the command line or the terminal of their computer before?
- Anyone?
- It's okay, it's totally okay.
- So who here has ever written code in the Python programming language?
- Okay a couple.
- And who here has Python installed on their computer?
- Okay a couple.
- So we're gonna have to install Python, I think it's part of the class.
- And so then ultimately, has anyone here ever written a web scraper before or even know
- what web scraping is?
- No.
- Okay so you know what it is?
- And keep your hand up if you've done it.
- How'd you do it?
- Scrapes in the middle.
- It's a button?
- It's a button?
- That's awesome.
- Yeah so basically if you guys don't know, it sounds like most of you do, web scraping
- are different techniques that basically suck the data out of a website where it's kind
- of encoded with all the other crap that's in the page and just drops it into something
- like a spreadsheet so you can access it like any other data.
- And there's a lot of different ways to do it and they're getting increasingly automated.
- But since every webpage is different and things are put up in different ways, it's often
- the case that you need to write code yourself as opposed to rely on some of these tools
- which are great and for simple stuff will work pretty well I bet.
- But for more complex websites that are just weird government stuff, probably can't do
- the job.
- So the goal of the class, which I'm calling a first web scraper, is for everybody here
- to one, get their feet wet and do a little command line mucking.
- Get Python going, do a little Python mucking, learn some of the basics of Python which you
- can use to all kinds of stuff.
- And then if we get there, write a little web scraper.
- And it won't be much more impressive than what the button can do but it'll teach you
- the fundamentals of how you would write one yourself.
- Does this sound like a reasonable proposal, does everybody think this is a good idea?
- But before we start, this is going to be painful.
- So in English we have a saying called a crash course, do you guys say this in Spanish?
- So a crash course is sort of like a class where things are moving really too fast and
- kind of out of control and a lot of stuff breaks.
- So there's no way you're ever going to master programming in two hours or even get all
- that far but we're going to try to get a lot of the basics in and a lot of stuff's going
- to go wrong and I'm going to run around the room and help people when they get stuck and
- all I ask is that you hang with me, okay?
- Sound good?
- All right.
- So I'm going to fire up my computer here and I'm going to give you the URL of a website
- where I have the entire class scripted out.
- So if you think I'm going too slow or too fast, you can move at your own speed right
- there, okay?
- So here's my text editor.
- Let's bump up the font size on this bad boy.
- Oh, by the way, is everybody, we have a mix of Windows and Mac, is that right?
- Okay, that's what I figured.
- That's fine.
- So we're going to have two different sets of bugs to work through, guys.
- You're not as excited as I am.
- Okay.
- Select.
- Let's see if this gets big now.
- Yeah.
- Yes.
- Oh, it's not up there.
- Let's see if I can, like, mirror the screen so I'm seeing the same thing you're seeing.
- Hey, how about that?
- Hola.
- Right?
- Okay.
- Is that big enough?
- Can everybody read that?
- Okay.
- Great.
- So what I'm going to do is I'm going to kind of write stuff up here and as we go and you
- guys can follow along on your computers.
- If at any point you have a question, just shout it, okay?
- And since I need to type, I might put down the mic and just yell for a little bit if
- that's okay.
- If that's not a big deal to anyone, okay.
- I'll move this mic up.
- Well, no.
- Whatever.
- Okay.
- Can you hear me?
- I'm coming.
- I'm coming.
- I need both hands.
- Okay.
- So the website where we're going to go, oh, thank you, I'm going to open a web browser.
- I'm using Linux, guys.
- Yeah.
- Anybody else?
- No.
- Get with it.
- Okay.
- So the website that has the documentation, we're going to walk through.
- Where's my mouse here, guys?
- Did it get lost on some other screen?
- Am I really mirrored?
- Look who's having the technical difficulties.
- Come on.
- Come on.
- Yeah.
- Where's my mouse?
- Okay.
- So the website you're going to go to while I figure that out is firstwebscraper.rtfd.org.
- You guys see that?
- So open that up in your web browser.
- RTFD, anyone know what that stands for?
- It's a nerd acronym in computer programming, which stands for read the fucking documentation.
- And it's a website that lets you host documentation for free.
- And so that's why we're using it.
- And you should land here.
- Everybody got it?
- Okay.
- And so this is the really long thing I've written that has all the stuff, including
- every command we're going to run.
- Oh, wait.
- I went to first news app.
- I went to the wrong site, didn't I?
- It should be firstwebscraper.
- That's a different class, guys.
- We're not doing that one.
- This is what you see, right?
- Not news app, web scraper.
- Great.
- And so this is everything we're going to do, all the commands, and a lot of the stuff
- I'm going to say.
- But we're going to go through it a lot slower, and we're going to kind of take our time.
- But if you get bored, you can move ahead.
- Or if we move too fast for you, you can hang behind.
- And our ultimate goal at the end of this, if we get there, is to scrape the data from
- the roster of jail inmates at the Boone County Jail in Boone County, Missouri, guys, who's
- been to Boone County, Missouri.
- Yeah.
- This is the University of Missouri, where I graduated.
- And this little dumpy website you can click on includes who's in lockup tonight.
- And so we're going to bust some data out of the Boone County Jail, guys.
- Jailbreak.
- Who's excited?
- No?
- Just me?
- Okay.
- But before we do all that, we got to get all our tools in order.
- And these are the basic tools that you're going to use your entire career as a computer
- programmer, or at least until Oculus Rift replaces, you know, the keyboard or whatever.
- And they're going to be necessary before we can even get started.
- And so we got a lot of boring fundamentals we got to get out of the way that you got
- to have in.
- And you can see that in this first section here called prerequisites.
- And the first thing that's required is a command line interface to interact with your computer.
- And so what's the command line interface?
- It's the goofy thing that looks like this, right, that you can type into, who am I?
- My favorite existential command.
- And it'll return for you a lot of different stuff about the computer, right?
- This is just like the hackers in the movies, right?
- That's what we need to be issuing commands directly to the computer and have it respond.
- Like say, run my Python script, right?
- That sort of thing.
- And so every operating system has one of these.
- They're just slightly different.
- And you can install replacements that are a little snazier like the one I'm using.
- But let's just now make sure everybody can open their command line interface.
- And no one's done it before.
- So this will be your first.
- Oh, this is awesome.
- Okay.
- So if you're a Mac user, right, you got that thing spotlight or the little search thing
- up in the corner.
- You know what I'm talking about?
- Okay.
- In the upper right-hand corner, in the upper right-hand corner, I don't have a Mac here,
- but in the upper right-hand corner, there's a little Sherlock.
- I appreciate it.
- Thank you.
- And you got to open that, and you're just going to type in the word terminal.
- Terminal.
- I'm causing trouble.
- And so that's going to then open, guess what, the terminal, okay?
- And you can go ahead and do just like I did, and type in, type in the existential command,
- who am I, right?
- And hit enter, and you're going to see it's going to return the name of your user, right?
- Now if you're in Windows, that might not work, because I don't know if Windows is that existential
- man.
- It doesn't look inward.
- It works excellent.
- Okay.
- I take it back.
- But now if you're in Windows, there's a slightly different thing you have to do.
- You have a start menu.
- I bet in the lower left-hand corner, right?
- And so if you hit your start menu, and this may vary depending on your version of Windows,
- there's a little box in there that you can type stuff into, right, and it'll pull up
- programs.
- Try typing in there three letters, C, M, D. And the Windows people can tell me if I'm
- getting this wrong, but you just type C, M, D into that little thing, and it should open
- a black DOS terminal, right?
- The one that made Bill Gates so rich.
- And so that is basically, that's the Windows version of the command line interface.
- And unfortunately, DOS and the Mac system, which is based on Unix, which is the same
- thing as Linux, have two slightly different ways of doing everything.
- They're very close, but we're going to have differences in what they can do.
- And so I'm hoping that now we can just have a brief segment where we walk through a lot
- of the basic commands that you will use every day once you get used to doing this.
- And if you become a programmer, you will do this every day, all day, that allow us to
- create files and directories, move things around, inspect the file system, and execute
- files, okay?
- And so that's going to be the first chapter of the documentation on the website that's
- called Act 1 command line.
- We're going to go through that now, okay?
- Everybody has their terminal up?
- We all got there?
- We're doing great.
- Boom County better watch out.
- All right.
- So now the teacher will remind himself.
- All right.
- So the first command we're going to learn is one that will just print the current directory
- then you're in.
- Not instead of who am I, where am I, all right?
- And there's two slightly different ways to do this.
- If you're in a Linux system, you'll simply type the letters P, W, D, and hit enter.
- Should I size this up too?
- Yeah.
- Bigger.
- Right?
- And if you're in Windows, just hit CD and push enter.
- So try doing that.
- And you should see it print something back out at you, right?
- And the way the command line works is when you issue a command that the computer likes
- and it knows how to use, it does the no bells go off.
- It doesn't reward you.
- It's not like there's any magic thing that happens.
- It just sort of quietly finishes its command and prints out the result of what it wanted.
- So you did P, W, D, right?
- And it printed out a little line there or CD and it printed out a little line.
- That line is the location where you currently are on your computer's file system, right?
- And that is going to be the home directory for your user.
- And I bet if you've ever used my computer or the finder or whatever they call it on
- Apple to move around, you kind of recognize that user name and where some of the stuff
- is, right?
- Because you're now browsing and sitting inside of the same file system and folder structure
- that you see when you navigate the folders via the user interface or via the finder and
- the my computer, right?
- You're inside of it operating within that same playing field.
- And this is your current location, which you can move around it.
- And so P, W, D, which stands for, anyone want to guess?
- It's pretty terrible.
- Present working directory, right?
- And CD actually stands for change directory, except you're not changing directories.
- It's just telling you the current directory, which doesn't really make sense either, right?
- And you may say, why do these things have such dumb short names?
- And why don't they really make sense?
- And the reason is, is because they were just invented by nerds in like the late 70s and
- early 80s who were coming up with all this stuff and nobody came up with anything better.
- And they caught on way more than any of them ever thought they would.
- And now we have to live with them because the entire world depends on how they operate.
- Right?
- And so what you'll find as you get inside the internals of your computer and you start
- to do what programmers do and live in programming land, you'll see particularly on the open
- source side of the spectrum, you'll see a lot of it is just kind of haphazard and has
- odd names.
- And it's sort of weirdly just the way it is.
- And you just kind of have to learn the way it is and memorize a bunch of little stuff
- and you'll kind of get it.
- And it sucks, but that's the way it is.
- And it's all the result of some guy, probably Dennis Ritchie in 1977 at Bell Labs in New
- Jersey said, I'm going to call it PWD or whatever, whoever it was, right?
- And so now it's still PWD.
- There's no better reason than that.
- Don't feel stupid because you don't know that stuff.
- It's just some dumb decision, some nerd made, okay?
- And as you write code yourself, you'll see how hard it is the name thinks.
- And they just keep it short because they didn't want to type so much.
- You know what I mean?
- It's a pain in the butt to type present working directory, okay?
- So let's see.
- The next thing we're going to learn is how do we find out what's inside this directory?
- Can I take a look at it and list everything that's in it?
- Yes, you can.
- So if you're on Mac, type L-S and hit enter.
- If you're on Windows, type D-I-R and hit enter, right?
- It's a little big here.
- Let me size that down.
- What's that?
- L-S. You know, I don't even know what L-S stands for.
- You know, there's this other crazy command, man, which will read for you the old manuals
- from like the 70s.
- So if we hit man, it's going to say list directory contents is what it was named originally.
- Why is that L-S?
- I have no idea.
- You know what I mean?
- You could write your doctoral thesis on this stuff.
- And so what you see is a list of the stuff in the directory and, oh man, I am really
- giving you guys a hard time.
- Sorry.
- And, you know, I bet you recognize some of these folder names that come up, right?
- You see like one called desktop or one called documents, right?
- I hate that.
- The documents one I really hate.
- It's just a mess, man, you know?
- And basically you're looking at the command line at the same stuff you see when you move
- around your computer in the interface.
- And we're just looking at it in this way and you can then inspect it.
- So let's say now we see that there's this directory desktop that's inside the directory,
- the PWD, where we're currently sitting, right?
- What if we want to move there?
- So in the same way that when you click on the desktop folder, it opens the desktop folder,
- right?
- You can issue commands that will move from your current directory to another one, right?
- And in this case, it's the same for both.
- It's that CD command from before.
- So what you can do is type CD and then the name of the folder you want to jump into,
- which in this case is desktop, right?
- And again, no, no bone, no bell, no reward.
- The computer just quietly moves on and returns the next line.
- That means it worked, right?
- That's a good sign when you're on the command line.
- Go ahead.
- Yep.
- Oh, hit Q maybe.
- Q. It worked?
- Okay.
- That's just the dumb stuff you memorize.
- So after like years, that's what you can look forward to is that knowledge.
- Hit Q to exit man because in 1977, blah, blah, blah.
- Okay.
- So, so what's happened?
- Now you did CD desktop.
- Now let's type our PWD again, or if you're on Windows, just CD and nothing.
- And guess what?
- My present working directory has changed, right?
- Remember before it was slash home slash Ben.
- Now it's slash home slash Ben slash desktop, right?
- So CD is the tool you use to jump around.
- Let's say I want to go back up to the one that's above me.
- That doesn't really have a name, right?
- That's sort of the parent of the one I'm currently in.
- There's another goofy way to do that, which you'll need to know, which is CD space dot
- dot.
- And dot dot is the magic notation for the parent of the directory you are currently
- in.
- Yeah.
- CD.
- And now I'm going to do PWD again and guess what?
- I'm back where I was before.
- It's in this kind of way that you can jump and move all over the file system, right?
- Let's say you wanted to move to an entirely different, you wanted to go back to the root
- of the file system and see everything from the very beginning.
- CD slash, right?
- L, S. Here's all the goofy stuff that's at the root.
- Then I could do CD slash home slash Ben, right?
- And I could type in the full path of someplace I would like to go, starting with the slash
- at the beginning, and that's going to jump me there.
- And it's with these types of acrobatics and just simple commands that you jump from one
- place to another and move around the file system.
- Make sense?
- Any questions about that?
- No?
- Cool.
- Okay.
- So now let's make some stuff.
- There's also a bunch of commands for creating and editing and deleting files, right?
- Like say a Python script, which we're going to get to, right?
- So let's say we want to make, so in our home directory we have desktop and we have documents
- and maybe you have like videos and music and all that crap that they load you up with,
- right?
- But we're done with that stuff, man.
- We're making code now.
- So we need a new folder called code to stick all our code in, right?
- And it's just as cool as all those other ones, so we're going to make it in the same
- place the same way, right?
- So did everybody CD dot dot back up to their home directory?
- And then do PWD or do a CD and look where you are and see if you're in that home thing.
- Yours won't be home slash whoever, unless you're on Linux like me, it'll be slightly
- different.
- It'll be whatever the Windows home directory is.
- PWD.
- So you might be out of your home directory.
- You know what?
- Let's do a cool shortcut.
- So there's actually a cool shortcut for your home directory that you can use at any time
- like ET to phone home, right?
- Or like Harry Potter to jump in a chimney.
- I use that for the kids now.
- All right, so if you do CD, a tilde, is that what you call in Spanish, too, a tilde?
- No?
- So if you do, it has no name?
- I've seen your keyboard.
- You have some pretty weird shit, too.
- No.
- But so you do CD, a tilde, and a slash, and that is kind of the universal signal for take
- me back to my home directory that you can use pretty much any time.
- That's not in the thing.
- So do that.
- And we can really put this code directory anywhere, so I wouldn't stress about it.
- But do CD, tilde, slash to make sure you're home.
- And now we're going to learn a new command.
- This new command will make directories, right?
- Make this code directory we want to put stuff in.
- All right, so first for the Mac people, and then for Windows.
- So the Mac command is MKDIR, right?
- And some weirdly named little compact thing that guess what stands for make directory.
- And then you follow that with a space and a name of the directory you would like to
- make.
- I'm going to do code with a capital C because it's just as cool as the desktop.
- Right?
- Oh, I already made code because I did this before.
- Hold on.
- This is what I get.
- I'm going to have to fix this real quick.
- Okay, so I'm going to name mine, I'm going to have to name mine code two because that's
- really professional.
- Just to show what you can do, right?
- So you do make dir code.
- Now if you're in Windows, you would do MD, which is a better abbreviation, I would argue.
- Right, guys?
- If we're writing these?
- Maybe not.
- I don't know.
- MD code is what you're going to do on Windows.
- Okay?
- That work for everybody?
- Oh.
- Whoo!
- Impressive.
- Impressive.
- That's a first.
- I've taught many of these classes, that's a first.
- Pretty cool.
- You're on your own.
- That should be Linux, though.
- So make dir should work.
- Though your problem might be that you don't have permissions to edit the file system because
- the phone's all locked down.
- Yeah.
- You might have the same issue.
- Yeah, you could try that.
- You guys can get that machine out if you want to.
- That's got admin on it, though.
- You're screwed.
- You might want to sit with somebody else, I guess.
- Sorry.
- I should have said that before we showed up.
- So did that work for everybody else on laptops?
- Okay, cool.
- So now if you do ls again, you're going to see your directory show up in that list.
- Okay?
- So now what we want to do is we want to move into our new directory.
- Who knows how.
- CD code.
- That's right.
- No, no, dot, dot, because dot, dot is backwards.
- So you just do CD code, and that'll move relative to your current directory.
- It will try to move into a directory named code.
- So everybody try CD code.
- And I have to do CD code 2 because I already have a code directory.
- Okay.
- No, no.
- Code 2 is just me.
- That's because I already had a code, so I had to make another one.
- That's how much I love code.
- All right.
- And, you know, obviously, it doesn't have to be code.
- It could be foo or donkey or whatever.
- You know, you can put whatever names you want in these cases.
- We're just choosing code.
- Okay?
- Awesome.
- Okay, so now we've moved into the directory.
- The next thing we're going to do is make a file there.
- And this is one you can only do on Mac and Linux that Windows doesn't actually let you do.
- So this is just going to be a quick Mac-only lesson.
- Let's say we want to make the Python file where we're going to store our scraper, right?
- There's a funny command named touch.
- I know, kind of creepy, right?
- Nerds.
- There's actually another one named finger.
- I'm not even making this up.
- So touch.
- And then we're going to name our file scrape.py.
- That's where we're going to stick all our Python code at the end of the day, and we're going to hit enter.
- Now, if you're in Windows, you could just open this directory in your My Computer.
- Go to your home directory.
- You're going to see code.
- You can just open it up there, and you can right-click and make a file named scrape.py.
- Have people done that before, just make a new file through My Computer?
- You have to save it yourself.
- You can't do it from the command line.
- Sorry.
- And so now if you do ls in this directory, you're going to see that the file you've made is sitting right there.
- And so touch is the way to make a file.
- Mine's kind of running off the end here because I got it so big.
- No problem.
- So have you cd'd into your code directory?
- Okay, now you need to open up My Computer and navigate to this new folder that we've made.
- You're in Windows.
- Or you're in Apple.
- Never mind.
- You want to do touch?
- You see the touch command right up here?
- Touch scrape.py?
- So what you're going to tell it to do with touch is you're saying, hey, computer, make a new file in this directory and name it scrape.py.
- Okay?
- And then if you ls after that, you're going to see it in the directory.
- And has anyone ever seen a .py file before?
- Do you know what that is?
- It's a Python file.
- And so all it is is it's just like any other file, any text file.
- It's empty.
- There's nothing in it.
- But the fact that we named it .py is when we get to the Python part going to ultimately be the instructions that let the computer know, hey, this is a Python script.
- And we need to, when we run it, we need to use Python.
- And the code in it is going to be Python code.
- And all that is just established by what you put on the end of the file when you name it, right?
- The same way an MP3 file is interpreted by the computer as music.
- And a JPG file, JPEG, is interpreted as an image, right?
- You guys have probably seen those before.
- .py is Python.
- Okay?
- Everybody get their file made?
- If you need help, just say so.
- You got it.
- So it's weird.
- I know.
- I know.
- I'm a little afraid to go into the man page on that one.
- And so that was one last thing we're going to learn.
- And then we'll be done with this, the kindergarten command line.
- Okay?
- And that's the most fun one.
- That's destroy a file.
- Right?
- Yeah.
- Yeah.
- Okay.
- Now, before I teach you how to do this, I'm going to warn you.
- Okay?
- This is not like, are you sure you want to put it in the recycle bin?
- And then even when you say yes, it's still in the recycle bin.
- And then they say, are you sure you want to delete it when you close it?
- You know what I mean?
- You get two chances on that recycle bin stuff, right?
- You get no chances.
- This is the most dangerous thing you do.
- You need to use it really carefully.
- If you were to write this command a certain way, which I will not teach you,
- all right?
- If you were to write this command a certain way, you could delete the entire
- file system of your computer and brick it.
- Right?
- And it's a big joke in computer programming to try to get people to do it,
- to hide it inside other code and screw with your friends.
- Right?
- But we're not going to go there.
- We're going to be really, really careful.
- Right, guys?
- We're just going to delete scrape.py.
- That's all we're going to do.
- We just made it.
- We're going to delete it.
- And the command for that is another dumb little thing.
- It's called rm.
- That's for Windows.
- That's not for Windows.
- That's for Mac only, Mac and Linux.
- And you do rm scrape.py.
- Hit it.
- Now if I do ls, guess what?
- It's gone.
- And you'll note it didn't ask me.
- Are you sure?
- It just did it.
- Okay?
- So if you're in Windows, I believe the command is del.
- Right?
- So you type del scrape.py.
- And then hit dir, right, to list the directory.
- And you'll see it's gone.
- Okay?
- Yeah, the command line.
- So those are the basic stuff.
- We're going to use a lot of these tricks as we get into the next stage.
- But that's basically the fundamentals of kind of basic command line stuff.
- Right?
- And there's one really special command we're going to need to get into Python stuff.
- But you might not have it already installed.
- Okay?
- And so that command is, guess what?
- Python.
- So we need to find out who has Python installed already.
- It probably comes preinstalled on your Apple computers.
- And who doesn't?
- It probably is not on your Windows computers.
- So could everybody please type Python, like the snake, right?
- Though actually it's named after Monty Python, I think.
- I hate to say it.
- Nerds.
- So type Python.
- Hit enter.
- And you see how my thing went into this weird little, like, land down here?
- This is the Python interpreter where you're able to write Python code.
- Really complex, important things that do amazing operations like 2 plus 2.
- Right?
- And if you got that, you've got Python.
- Right?
- And you're good to go on the Python part.
- If you don't have that, we need to install Python on your computer.
- Okay?
- So who doesn't have Python?
- All right.
- It's totally cool.
- It's all Windows people?
- Yes?
- Yes?
- Okay.
- Okay.
- We're going to have now a painful interlude where we try to get Python installed on the
- Windows computers.
- And we just got to do this and get it out of the way.
- It's going to be a pain.
- But once you have it, you'll be set.
- Okay?
- So I'm going to propose, since it's only a small minority of people, that we maybe take
- a 5 or 10 minute break and I just huddle with the Windows people and we work it out.
- And the Mac people don't have to deal with it.
- Does that sound okay?
- Okay.
- Good.
- All right.
- So, hey, guys.
- If you go back to that web page I made, Windows people, I'm talking to you.
- And you go to the top, that prerequisite section.
- Number three says this right here.
- This is going to tell us how we're going to install Python.
- And there's two steps.
- The first will be not so bad.
- The second's going to suck.
- Okay?
- But we're going to do it.
- All right?
- So the first is, is we need to go click on this link I have here where it says here.
- All right?
- You see that?
- And that's going to take you to the Python website where we're going to download something
- that will install Python for you.
- Okay?
- Okay.
- And my mouse has gone haywire so I'm going to restart my computer.
- But you guys all see that link and you're clicking on it?
- Okay.
- So do you see where it has a download?
- Now we need to find Python 2.7.
- We do not want Python 3.0.
- This is a whole lame Python thing that we're just going to dodge.
- Okay?
- 2.7.
- You see that?
- And it says download the Windows installer.
- Do you guys see that?
- The first that Windows x86 MSI installer.
- Does everybody see that right there?
- Okay.
- That's what we're going to download.
- And that's going to be a little click-through thing that's going to try to install Python
- for us.
- Okay?
- So can you just click that and download it?
- Thanks.
- Somehow my mouse has just, like, disappeared.
- It's totally bizarre.
- It must be something to do with the multi-screens.
- Oh, no.
- I got my battery filled up.
- Oh, that's this wireless thing.
- The mouse is back in Los Angeles.
- I'm trying to use the track pad.
- And I'm failing.
- You guys don't see it, do you?
- Totally weird.
- And maybe if I, is that thing downloading?
- So now you're going to have to click through it, okay?
- And let me look over somebody's shoulder as they click through it.
- Did you already do it?
- Oh, you got it?
- Okay, great.
- So we're going to open it up.
- Is it all downloaded for everybody?
- Okay, great.
- And it's going to open up some weird little thing that's like every other program you've
- installed in Windows, I think, right?
- Once it gets fired up.
- Okay.
- And it's going to say, should you allow it to do things to your computer, maybe?
- Right?
- And your answer has to be yes.
- Because we're playing with big boy's toys now.
- Okay.
- And then it's going to maybe ask you to select the destination directory where you're going
- to install Python.
- This is the folder where it's going to put it.
- Don't change this.
- Just say, yeah, I'm good with it.
- Hit next.
- Okay.
- Then it's going to ask you if you want to customize Python.
- You do not want to customize Python.
- Okay.
- Next.
- Okay.
- So it's going to install this.
- It may ask you to type in your administration password.
- If you don't have an administration password, you are screwed.
- To do this type of work, you need to be able to operate like a super user on your system.
- If your job is to try to do these things and your company gives you a computer that won't
- let you, you need to go to your boss and say, I can't do my job.
- Tell the IT idiot to take this off.
- Right?
- Because you won't be able to do the stuff we're going to try to do with this when you
- have those restrictions that are made for people who aren't programmers.
- Right?
- Okay.
- So let me open my terminal.
- Let me move it up over here.
- Okay.
- So again, let's type Python Windows, guys.
- Oops.
- Python.
- Okay.
- Did that happen?
- What happened?
- It doesn't work.
- Okay.
- So this is going to be what I expected and a teachable moment, guys.
- You ready?
- Okay.
- So these little weird things we're typing, LS, CD, PWD, Python, what the hell are they?
- Where are they?
- Right?
- They're little programs that are installed in your computer and they're off in some, like,
- distant hidden system folder that you probably have never been to or even knew existed.
- There's a folder filled with hundreds of these stupid little programs.
- Right?
- But they're not in the directory where you're issuing commands.
- They're off in this distant directory.
- So when you're executing them, you're not giving the full path to where they're at.
- You're just giving their names.
- So how does the computer know to run them?
- It knows to run them because there's this thing called the path.
- And I'm only teaching you this because you have to know it for Windows in this case.
- So the path is this wacky global variable that is on all these computers that's just a list of directories.
- Let me show you.
- This is really exciting stuff.
- And you can call it like that.
- Whoa, what's that, Ben?
- That's a list of directories in these hidden super folders separated by colons, like user local Ben.
- Once you're inside the mind of Dennis Ritchie in 1978, you'll know Ben stands for binary.
- Right?
- And let's say let's look inside that directory and see what's in there, right?
- Hey, it's a bunch of commands, right?
- And what if I grep like there's one that has LS in it, right?
- And if I echo the path again, I can see there's one that's just called Ben, right?
- And that has all kinds of stuff, including LS.
- So there is LS.
- LS itself is actually at slash Ben slash LS, right?
- But the reason why when you type LS, it actually calls that is your computer, every time you issue a command,
- is looking through this path for something that matches, that has the thing you just typed.
- And if it finds it, it executes it, right?
- So it's kind of a shortcut.
- So these things that are in these special folders you can call on without having to type their whole long location
- or to CD into their directory or any of that stuff.
- And Python's the exact same way, except Python and Windows is one we're adding that wasn't there before, right?
- And so Python itself is just a file, a file that was just installed.
- And remember when you were clicking through the goofy installer, it said, where do you want to put this?
- And it said, Python 2.7, it wanted to make that folder.
- You remember that in Windows?
- That's where your Python file is, except that folder is not in your path.
- So even though the file's there, when you type Python, it doesn't find it.
- So I know this is way in the weeds.
- So what you literally have to do to get Python to work on Windows, and remember, you chose Windows, not me, right?
- And this is true for almost all the programming languages.
- You have to install it, and then you have to edit your path so that it can find Python when you type it.
- I know that was a long journey.
- Did that make sense?
- Okay, Windows crew, we're going to edit your path.
- You already did it?
- I found a folder, Python 2.7.
- You already CD'd there?
- Dude, you're a pro.
- Right, so he CD'd into the Python directory, and within it found the Python thing, and then just called it,
- and because he was in the directory, it could find it.
- It didn't have to look on the path, right?
- And it just ran it.
- So okay, how about you Control-C to get out of that, or just type exit.
- Let's leave the Python.
- Right, and so now you're going to be exit, and then open parentheses, or Control-Z.
- Do Control-Z.
- Enter there.
- Now do CD, and let's see what directory you're in where you found the file.
- Just CD nothing, so it'll tell us where you are.
- You're in C colon slash Python 2.7.
- So C colon slash Python 2.7 is what we need to add to your path, right?
- So how do we add things to the Windows path?
- Oh, well, so let me write that down.
- So if I've got this right, tell me if I got this wrong.
- This is the folder where it is.
- Oh, I got Python spelled wrong.
- Write that, capital C maybe.
- Okay, so that's the directory we need to add to your Windows path.
- And so here's how you do it.
- So this could vary depending on your version of Windows.
- Yeah.
- So do you have a My Computer, like a little My Computer icon in your desktop,
- or is that too old for you guys now?
- If you do, could you right-click on it?
- So you're going to go to your My Computer, wherever that is, the little opener.
- And then the actual My Computer icon itself.
- Can you right-click on it?
- Okay, has everybody been able to do that?
- I see you were able to, right?
- This is what we get.
- Okay, so Windows users, right-click on My Computer,
- and then there's going to be something like Systems Properties,
- or what did it say?
- System.
- Okay, so you know where the My Computer is?
- Have you seen that?
- It's probably on your desktop.
- So My Computer itself, right?
- So there's an icon you clicked on to get here.
- That's your user folder.
- How about that guy?
- Right-click on that.
- Right-click on this one.
- Or right-click, so it'll bring up like a menu.
- Yeah, then do proper.ace.
- I get it right now.
- Not even close.
- Yeah, okay.
- Okay, so could you try doing that same thing,
- which is right-click on your computer right there?
- Yeah.
- And then go to Properties.
- I'm sorry.
- It has to be done.
- Okay, we got there?
- You got there?
- Okay, so who's got one in English so I can look over your shoulder?
- Did you right-click on My Computer?
- So you hit that,
- and then there's going to be a little computer icon if you scroll up.
- Scroll up.
- That guy.
- It says equipo on some of these.
- And then go to Properties.
- Okay, so that's going to give us this like crazy computer settings page.
- Does anyone have it in English so I can kind of like peek over your shoulder?
- Oh, thank you.
- So in the left-hand menu,
- I'm going to guess we're going to look in Advanced System Settings.
- And yes.
- So do you guys see something that says like Advanced System Settings?
- You click that, and then that's going to open up a weird gray box, okay?
- I know.
- Blame Bill Gates.
- Can you believe he's this rich?
- Okay, yes.
- Okay, so you guys got it up.
- Okay, and now there's a bunch of weird tabs.
- There's a bunch of weird tabs.
- Come on.
- You've done this before?
- Okay, so then there's an Advanced tab that's in the middle of the new thing that popped up.
- Can you hit that Advanced tab?
- Okay, awesome.
- Now at the very bottom of this dumb thing is something that says Environment Variables.
- Do you see that?
- Environment Variables?
- Let's click on that.
- There it is right there.
- Okay, did you get that up?
- Okay.
- Okay.
- Everybody's got that up.
- Okay, and so you should see in that top box a list of goofy variables, right?
- And so some people will have one that says Path and others won't.
- Is that what's going on?
- I see some people.
- So his says Path.
- Do you have one that says Path in the little list there?
- Maybe in the lower window.
- Let's look on your lower window.
- Oh yeah, so on some machines it might be in the lower window or the upper window,
- but look for the one called Path.
- Do you see that?
- Okay, so that is the same way I was printing out my path up on the screen.
- That's your path that leads to all the folders where the Windows files are.
- And so what we need to do is we need to add your Python directory to that list that's on the right-hand side there.
- And the way we're going to do it is we're going to click on Path in that little list, right?
- So it gets blue.
- And then we're going to hit Edit.
- Do you guys see that Edit button?
- Okay, this is pretty gnarly, I know.
- Okay, so now do not delete what's in there.
- Do not delete what is in there.
- Okay, we're going to add to it.
- Okay, so click into it and say scroll your mouse all the way to the far right-hand side, right?
- And so your cursor is at the end of that long thing that's been written in that box, okay?
- And we need to put in the separator that's going to separate one directory in the list from the next,
- which is a semicolon.
- Is that how you say that in Spanish?
- You guys know what a semicolon is?
- Okay, thank you.
- So at the very end, you're going to put one of those guys,
- and then you're going to put the path to our Python folder where we believe that bin file is, right?
- With the C, yeah.
- So you guys see this last line I have written here?
- I'll blow it up.
- See that line right there?
- You're going to add that to the end on the right-hand side.
- Beautiful.
- Did you add it on the end?
- Oh, it's already working? Great.
- So you must already be in there if you look.
- I bet if you look, it's already in there.
- I might need to do a capital P, I think, on this Python.
- Yep, that's perfect.
- You don't need the space.
- I don't know if it matters, but you don't need it.
- Okay, were you able to get it?
- You're stuck on...
- Okay, so go to...
- There's one called Advanced, maybe?
- Click through these tabs so I can see them.
- No, click right here.
- No, try here.
- Right here.
- Hit this thing.
- No, that's not it.
- Cancel.
- Try this one.
- Click try here again.
- You might have older windows, huh?
- It's not your fault.
- Hit the here.
- My Spanish is so bad.
- No.
- Check this real quick.
- Your hard drive.
- Let's try this one again.
- Just look at it real quick.
- Okay, once you have it in there, just hit okay and just save it.
- Okay, and then close and reopen your DOS prompt.
- So close the command line after you've saved it
- and reopen it by typing CMD into that directory.
- Cancel.
- Hit that.
- Try that again. I'm just confused.
- Let's look through these.
- This is like permission.
- We'll Google it as we get other people caught up here in a minute
- and I'll come back to you. I'm sorry.
- Basically, the Google search we need to do is
- set the path in Windows Vista, which is what you're running.
- Okay, and if you just Google that,
- someone will have the screenshots that will help you real quick.
- I'll look when I have a second, but that's what it's going to be.
- Okay, so everybody that just did that and reopened their terminal
- and typed Python, did it work?
- Yes, so type Python now again.
- Yes.
- What a relief.
- Yes, you have Python. Windows.
- We almost defeated it.
- Okay.
- Right.
- Okay.
- We're ready for the Python segment of our lesson.
- Okay, guys?
- And if you call up, you guys still have that web page up
- where all the lesson is?
- Now we're going to work through, and I know this is a little kindergarten,
- but it's good stuff.
- We're going to work through the fundamentals of Python
- in the same way we had worked through the fundamentals
- of the command line, right?
- So I'm going to help you
- just kind of understand some of the basics,
- show you some simple basic commands,
- and after we kind of got them figured out,
- we're going to write our script, all right?
- But first, we've got to deal with the basics.
- And so, you know, the first thing about...
- Remember our file scrape.py we made?
- Right, so running a Python file,
- which is just in a script like anything else, right?
- God damn it.
- Oh, my God, sorry.
- It's just as simple as taking that Python command.
- Mm-hmm, mm-hmm, mm-hmm.
- All you're doing is...
- Python file name.py
- is then going to execute whatever file you've named
- with the .py and run it, okay?
- And that's basically how you end up running Python files,
- now that we have them at the end of the day.
- But the alternative is just to type in Python itself
- and hit enter and get the Python interpreter, as it's known,
- so you can execute Python code like 2 plus 2 directly, okay?
- Am I cut off at the end there a little bit?
- All right, something like that, okay?
- And so I want everybody to type in Python
- and get down to their interpreter.
- Let's switch, sure.
- Okay, sorry.
- Okay, and so now we're no longer in the command line.
- I mean, we're in the command line,
- but we're not playing by the rules of the command line.
- You know, there is no...
- Right, so there is no more ls here.
- You know what I mean?
- There is no more cd, there is no more rm.
- We're now not writing command line programs and scripts.
- We're writing Python code, which has its own rules,
- its own commands, its own bad names for things,
- and its own kind of world.
- But what's good about it is you can write these scripts
- that from top to bottom do stuff and then rerun them.
- But you got to know the fundamentals of things
- before you can run them.
- So is everybody who's able to get it going
- dropped into Python?
- And you know what you can probably do,
- I just thought of it, it's the fastest family.
- You cd, you do Python directly,
- you do problems in front.
- I just thought of this.
- Sorry, I'm breaking all the rules.
- Let's open up your command line real quick.
- Let's cd into...
- I'm feeling bad.
- Oh, God damn it.
- Ben.
- So...
- cd, this is shift, Python, 2.7, right?
- And then type Python, and it's not on your path,
- but you can just run it right there.
- Okay, so let's just skip the path and you can keep rolling.
- You're with us? Okay.
- So everybody's in.
- So Python, so like the first thing you need to know
- is anybody know what a variable is?
- Have you ever heard of this?
- So a variable is like a little bag or a little name space
- where you name something and then you assign to it a value.
- You know what I mean?
- So you create a label that exists within Python
- that has something stored inside of it.
- And so I'm going to create one called greeting,
- and I'm going to create it by naming it.
- I'm going to name it greeting.
- I'm going to type equal.
- I'm going to open this string and then I'm going to say,
- hola.
- Okay.
- Can everybody try that?
- So you can do greeting equal hola.
- Yes.
- So Python is now the same across all platforms,
- which is a great thing about programming languages too.
- They can be run on different systems and do the same thing.
- And that's good.
- Okay.
- So it just again just kicks down to the next line.
- There's no bells or whistles.
- There's no reward.
- But what you've just done is you've created a thing called greeting
- and you've put inside it the phrase hola.
- Exciting, right?
- So now you can do stuff with it, right?
- Now that you have it.
- So you could do, there's a Python command
- you might be interested in called print.
- Can anyone guess what this one does?
- So one of the great things about Python
- is they kind of did a better job of naming things.
- It has its own problems, but one of the big,
- the emphasis when the language was created in the 90s
- was to make something a little more legible.
- Like let's just name it print instead of echo,
- which is what it's called on the command line, right?
- So if I say print greeting, look what comes out.
- Okay.
- So go ahead and try that.
- So we have, we have this variable,
- which we then could screw around with and do stuff with.
- And so I can do, and I could say greeting, I could change it.
- I could instead say greeting equal hello, right?
- And now I've changed what it is.
- And if I print greeting, it says hello now instead of ola.
- Make it sense?
- Okay.
- So the variable is the thing on the left.
- That's the label or the namespace as it is known.
- Maybe not such a great name either.
- The namespace greeting is where we are storing hello.
- But what is, you know, hello.
- Hello is what is, well, hello is a string.
- Hello is a string of characters grouped together
- and surrounded by quotation marks.
- And the thing about programming languages is they allow things
- only in certain data types.
- Everything has to be very strictly defined as one of a set of data types.
- And so one type of data that you can assign to a variable
- is a string, which is some letters and characters surrounded
- by single or double quotes.
- And you can do double quotes if you prefer that.
- I could do or single quotes greeting equal ola, right?
- So just to make the point, let's make a whole bunch of strings, right?
- So I already have greeting.
- Let's make one.
- Let's call it goodbye.
- And I'm going to say later bro, right?
- Because I'm from America.
- And I'm going to hit enter and I can then print goodbye.
- Right.
- And then I'm going to make a third one and I'm going to call it.
- And, you know, again, you can name these really anything, right?
- So I'm going to name this favorite animal and I'm going to say it's donkey.
- Right.
- And then I can print favorite animal.
- Right.
- And it says donkey.
- And then maybe I just want to make a variable called donkey
- and put donkey inside of it just for fun.
- So I can print donkey and see donkey because I love donkeys.
- But you get the idea that this is just like any kind of string like this
- can just be shoved into a variable and then accessed later.
- And you're going to use this all the time once you get a script.
- And it might not make sense at first, but once we have all these building blocks,
- I promise you we'll make something.
- Okay.
- So there's other kinds of data, you know, it's just like math class, man.
- You got strings with your characters, but you got integers.
- Remember, integers.
- So an integer is a number that doesn't have a decimal point.
- Right.
- And you don't need strings on those.
- So I can make one and like, you know, it's going to be like math class.
- I'm going to name the variable X.
- Right.
- Just like math class and X is going to be 99.
- Right.
- And there's no strings around it.
- And because there's no strings around it and because it's a number,
- it's that X now becomes an integer.
- Right.
- And I can do this thing called type.
- Right.
- And it'll tell me X is an integer.
- Right.
- And then I can do X plus two and get 101.
- Right.
- But if I do greeting plus two, you can't add strings and integers.
- Ints is short for integer.
- Right.
- Another random abbreviation that they've done.
- So is it the word integer in Spanish or what would it be?
- Integre.
- Integre.
- Okay.
- Close.
- Okay.
- And so because X is an integer and Python,
- like all programming languages is picky about data types,
- I was able to add it with another integer,
- the number two and get the combined number X plus two equals 101.
- Right.
- And if I just type X, it'll tell me X is 99 and I can do multiplication.
- I can do X times error.
- Hold on.
- X times X.
- There's all sorts of stuff you can do with integers you can't do with strings.
- Right.
- So that's the second data type we're going to use from time to time as well as integers.
- There's another one called float as in floating point decimal is what that's short for.
- And so that's just the number that has a decimal point and values after it.
- So that would be 99.9.
- Right.
- Would be a float.
- Right.
- And you could do another one and it could be 0.1.
- And you can do Y plus Z then and you get 100 even.
- Right.
- So it's just another data type similar to integer.
- It allows decimals.
- They keep them separate.
- It's a math thing.
- Okay.
- It comes up and when you do really high end math for what we're doing,
- it doesn't matter.
- But if you need the decimal points, you got to make sure things are floats now.
- Now a really fun one that we're going to use all the time, which is called a list.
- So if you guys done a lot of programming before, I don't know if it's okay.
- If you haven't, sounds like not.
- But every programming language has things like this strings, numbers, something similar
- and lists and lists are just collections of variables or values that you can do stuff
- with.
- And the way you make a list is you just make one of these brackets that looks just like
- that.
- Okay.
- It's got to be that rectangular bracket and you just start putting stuff in it.
- So I'm going to make a list and I'm going to say 1, 2, 3, 4 are going to be my list.
- And I hit enter and there's a list.
- 1, 2, 3, 4.
- And I can even put that in a variable.
- I can say list of ints.
- Right.
- So it's a list filled with integers.
- So I can now have a variable that has inside of it a list.
- And you can do stuff besides that.
- You know what I mean?
- You could do 1, 2, A, B. Right.
- That's okay.
- You know what I mean?
- So with a list can contain any of those data types that we've seen before.
- A big list of them.
- And it can even contain other lists.
- You know what I mean?
- I could have in here then 3, 4, 5.
- And I now have a list with a list in it.
- And this sort of nested lists and how we organize lists is a big way that we kind of structure
- and transform data that we're moving around and playing with in our scripts.
- Okay.
- Am I going too fast?
- No.
- This makes sense.
- Okay.
- Good.
- And then lists have cool things you can do with them, man.
- So you remember I made list of ints.
- Right.
- So let's say I want to pull an item out of the list.
- I want the number 1.
- You can do another bracket like that on the end and then put a number in it and it will
- pull the item at that number right out of your list.
- So you're able to access the list, pull items out of it.
- We're going to loop through it pretty soon and interact with it in ways you can't with
- other data types.
- And so why is 0?
- Go ahead.
- I was just about to say that.
- You're kicking butt.
- That's exactly right.
- So remember there's all the weird stuff we have to memorize.
- One of the crazy things about computers, and this is true with almost every programming
- language, is that things are what are known as 0 indexed.
- And so the first of anything is actually the 0.
- So whenever you want to pull, so if I want to pull the first thing out of the list, I
- give it the 0 index that's called.
- If I want the second thing out of the list, I do the 1 index.
- You just subtract 1 from whichever one you want mentally, and you can kind of get where
- you want to go.
- And let's say there is no fifth one, and so if I ask for an index that isn't in the list,
- it just throws this error, right?
- And so that's one of these Python things you just got to memorize, but it's really true
- for all languages.
- 0 comes first.
- Count starting with 0.
- One of these programming things, man.
- There'll be exceptions to it too, which just drive you crazy.
- So then you can do fun things like you can slice a list.
- Let's say I want just the first two things in the list, right?
- You do the same bracket thing.
- You give it the index you want it to start at, a colon, and then the one you want it
- to end at, and it will slice portions of the list in different ways so you could exclude
- values and goof around with it.
- Make sense?
- We'll go over this again when we actually use it.
- Okay.
- Then there's this other one we don't really care about called a tuple, and a tuple is
- a list that you can't add things to.
- It's a permanent list that has to stay the way it is forever.
- So I could take list of ints, for instance, and I could do append, add 5, right?
- And then if I do, you'll see that now 5 is on the back end of the list, right?
- So you're able with lists to edit them and goof around with them after you make them,
- right?
- Tuples you can't, right?
- Who cares?
- Boring, right?
- So you only really need tuples.
- Well, you don't really ever need tuples for what we're doing, but they're kind of there.
- Maybe at some point it'll come up in your life, all right?
- It's a list that you can't edit.
- So for instance, if I take this and then I try to add something to it like I did the list,
- it's going to throw an error.
- And so it's basically a list that can do less than a list.
- We probably shouldn't even cover it.
- It's boring, okay?
- There's a final data type, the dictionary, which is we're not going to use in this lesson,
- but is ultimately very useful.
- And it's like a combination of keys and values that you can store together in something.
- And we won't even go into it in depth because it's kind of complicated.
- But basically, you see how I have something here that's called like the phone book?
- And it has pairs.
- So there's two sets of data.
- There's the name, Chinese takeout, and there's the phone number to call.
- And then there's the other name, mom, and the phone number to call.
- So essentially you create something kind of like a phone book where there's pairs of what are called keys
- and values that you then can look up and goof around with later.
- So I could say, give me mom, right?
- And then out comes the number, that kind of thing.
- But those are all the basic data types.
- That's basically it, guys.
- You've learned every data type and all the stuff that goes not only into Python,
- but pretty much all computer programming.
- And these really boring basic fundamentals that I've just showed you will really quickly
- become things you can do a lot with, okay?
- And so one way you start doing it is by chaining them together into what are called control structures
- or scripts of multiple things on top of each other that negotiate a set of data and do stuff.
- So the one we're going to use in our scraper that we'll cover now is called a for loop.
- Have you guys ever done a loop?
- So we have the list, right, and we want to loop through it,
- and we want to do something to everything in the list, right?
- I'm skipping some stuff in the lesson here too because I want to get to our scraper.
- But you can look at the other stuff if you want to.
- So I'm going to clear this, and I'm going to make a new list, and I'm going to call it letters, right?
- And letters is going to have A, B, and C in it, okay?
- And remember, I could do letters and zero, and I could get back A.
- Makes sense, and I could do letters and two, and I could get back C.
- Okay, so I can access it, but what if I want to loop through it and do something with every letter in the list?
- Like say I might want to loop through a spreadsheet of data, right?
- Huh? That's where we're going to go with that ultimately.
- So what you do is there's another special command called for, spelled for,
- and you do something like four letter in letters, right?
- Hit colon, enter. You should get this dot, dot, dot. You need the colon.
- Four spaces. One, two, three, four. One, two, three, four.
- And then I'm going to do print letter, and I'm going to hit enter twice.
- Look what it did.
- So go ahead and try that. There's a bunch of stuff to unpack here.
- So the magic words here, the magic words here that you will always reuse
- whenever you loop through a list are for and in, right?
- And then these are things you can substitute and do other stuff with.
- So letters, right, is our list that we made up here.
- And we're saying for every letter in the list, right, print the letter.
- And then one, two, three, it prints the letters.
- So what it's doing is it's looping through every item in the list
- or iterating, as they say in programming, right?
- It's looping through every item in the list, and it's doing what you tell it to here, right?
- And if you've done this in other programming languages like JavaScript or Java,
- you may have had to do like a parenthesis like here and here,
- like around the clause that you've written to close off the code
- that's being executed against each letter.
- This is where Python is unique.
- This is the one thing that's totally special about Python and that no other language does.
- It's that whitespace, this space, space, space, space we can't do here,
- basically created a whole other set of indented code
- that only got ran inside the loop for each one, okay?
- And the computer just actually reads your whitespace.
- And so every single space you type in a Python code is meaningful
- and will be interpreted by the computer as it executes the code, right?
- And the indentation is a way that you block something off to say,
- I only want it to run inside of, say, a loop, okay?
- And this is a tricky thing because it means you've got to get your spaces just right.
- You don't use tabs.
- If you use four spaces, you always have to use four spaces for it to work.
- If you have multiple lines in your loop, like I'll show you.
- So let's say I do this again and I do four letter in letters, right?
- Oops, I screwed up.
- See how as soon as I made a typo, I got an error?
- That's how picky this stuff is.
- One, two, three, four.
- Print the number one.
- One, two, three, four.
- Print letter, right?
- And you see how within the loop, for each time through,
- it executed the two things that it had been indented, right?
- Print one, print letter.
- First one's A.
- Print one, print letter.
- Second one's B, okay?
- So we're able, with our indentation, to create kind of a level
- or a bracketed off piece of code that only operates within that world.
- Okay, so like here's another.
- And then once you de-indent, you come out of that world.
- So watch this.
- So I do four letter in letters, right?
- One, two, three, four.
- Print letter, right?
- Then I go down and I do print hello, right?
- That just runs once.
- Because we've come out of our little indented world, right?
- And when I didn't have that space in there,
- when I just tried to print it right after,
- or if I tried to print it at a different indentation,
- like if I tried just one space and I did print, print one,
- it's going to break, right?
- And it's going to say, you have an indentation error.
- This thing you did right here, I don't know what to do with.
- And the computer's so picky, that's how code is,
- it's just going to fall apart.
- And so the real finicky thing at first
- is you have to get the white space just right.
- And some people who don't like Python
- call this white space fascism.
- Yeah, that hurts, right?
- Because it's very, very picky about how you line stuff up.
- And so if you're getting errors,
- that could very well be the cause,
- and that's going to be really important
- as we get more complex and we put more stuff together, right?
- But this is the basics of how you loop through a list.
- And now letters is the name of our list,
- but this letter right here,
- so this is a variable that we create
- that then exists inside the indentation.
- And this doesn't have to be letter,
- we could name this whatever we want, right?
- In the same way we were naming variables
- when we first started.
- So we could do, for instance,
- I could instead say for donkey, right, in letters.
- One, two, three, four.
- Now what do I type?
- Print donkey, that's right.
- And so the way the loop works
- is the things that always have to be the same
- is this has to be a list, right?
- This has to say for and in,
- and then you have to name the variable
- that it's going to be iterating through.
- Okay, and that is how you loop through a list.
- Are we out of time?
- Oh my God, totally?
- Are they going to kick us out?
- Okay, can I just show you the scraper script really quick
- even though we won't have time to write it together?
- Guys, I'm sorry.
- Okay, so, Boone County Jail, who's in lockup, right?
- So they have this goofy website.
- I'm going to go super duper fast.
- Get excited, right?
- So it is this webpage right here.
- Oh, hold on, right here.
- And so this is a dumb webpage
- that they do in Columbia, Missouri every day
- telling you who is currently in jail.
- And it updates and looks the same every day.
- It has their name and as their gender,
- their race, their age, and where they're from,
- mostly from Columbia, Missouri, right?
- And so let's say in all this page,
- I want this as a spreadsheet.
- I want a scraper to go,
- and every day while I sleep even, right,
- download this spreadsheet and save it to a file.
- So over time, I can build a database
- of everybody who gets locked up, right?
- This is a table that's inside this HTML thing,
- and I want to scrape it out.
- If I inspect it using this magic tool,
- I can see that here it is.
- This is the table that's inside the page
- I want to pull the data from, right?
- And I can see that it has a unique goofy name
- called results table.
- That's the name of this table.
- And so what I need to do is write Python code
- that's going to pull out results table,
- loop through all the rows, use our looping skills,
- grab the data, and print it out to the file, okay?
- And so then the way that happens is you have to install
- a couple other Python libraries,
- and this was going to be the other hellish install
- we don't have time for,
- that allow you to download websites
- and parse them using Python and make a script.
- And so the script is right here, right here, right here, right here.
- Oh, too big.
- Right?
- And so this is a Python code we'd write in that script
- that does a lot of the basic things we talked about.
- What we didn't get into is you can import these other libraries
- that do all kinds of magic things so you don't have to write code.
- One will download websites, the other will parse them.
- This one will write them out to a spreadsheet, right?
- And then what you do is you just start using them.
- You make a variable, right, called URL,
- which includes a string,
- which is the URL of the web page you want to visit.
- You then use the fancy library we imported right here
- to get the URL and save it
- and pull the HTML from the web thing magically in the script.
- We then pass that into this other library that can parse the HTML, right?
- We give it the special instructions and we say,
- hey, find in that the table called results table.
- Remember how we saw that in the page?
- And that's the detective work of scraping,
- is that you have to find in the page the unique thing
- that allows you to pull the string and extract the data,
- which we did really quickly inspecting it.
- There it is.
- That table is now a thing that we can find stuff in using beautiful soup.
- So I say, hey, I want you to take the table
- and I want you to find every row in the table,
- which happens to have an HTML tag called TR, which stands for row.
- And guess what, I want you to loop through it
- just like we just did, right, guys?
- So for every row in the HTML rows, right,
- I want you to then do a second loop.
- I want you to find all the cells in the table loop through the cells, right?
- I want you to grab the text out of them
- and then I want you to add them to a list, right?
- So as we're looping through the cells,
- we're adding them to the list and that's a row
- and then add the row to the big list of all the rows
- and then we create a list of lists
- that has all the data pulled from the HTML via the loop, right?
- And then write it out to a spreadsheet using the simple spreadsheet tool.
- I know that was really fast, but that's basically where we're going.
- If you follow on the web page, which again is firstwebscraper.rtfd.org, right?
- You guys remember that one?
- That's this one.
- Every single step is explained and written out
- and it will show you how to work through the skills you have now
- to write the script and do exactly what I just showed you.
- So we didn't have time to get through all this stuff tonight.
- We only got you set up and running, but you can go home,
- click on Act 3 and you can do all that stuff
- and end up with the script that works
- and there's step-by-step instructions.
- So the class can continue even though they may kick us out of here.
- Okay, anybody have any questions or you want to yell at me
- for not going fast enough?
- Yeah.
- So there are utilities in Python to interact with all sorts of things.
- You know, so there's millions of open source libraries
- or thousands that will make it easy to access different things
- like requests and beautiful super we just saw.
- And there are tools for accessing Twitter and Facebook
- that allow you to, just in the same way we import requests,
- you sort of import Twitter or whatever it's called.
- And these things, you just Google them on the web,
- you look for them, Python and Twitter,
- and you'll see that there's libraries for interacting
- with all sorts of different stuff that have in them functions
- and tools that allow you just to stuff in the data you want
- and kind of make something simple by smashing them together.
- Yes.
- There's not necessarily a best one.
- I like Python just because I like Python because I like the simplicity.
- I think that the white space and a lot of the stuff makes it easy to read
- and that's why Python is often taught to beginners,
- especially even children that taught Python as a first programming language
- because a lot of the stuff we were just doing is really simple
- and you'd be surprised how kind of weird it gets in other languages.
- But to be honest, you can do this with any of the really popular languages.
- I can't say it's necessarily better.
- I just think it's readable.
- Oh yeah, it's all free.
- I mean, remember we downloaded and installed it.
- Everything's free and open source.
- I mean, all the major programming languages are,
- with the exception of Java and the Windows stuff.
- But unless you need to make a Windows app,
- there's no reason to bother with that stuff.
- So it's usually best just to, if you can,
- to just code your own web scraper rather than using, you know,
- kimono labs or import.
- Like the point and click tools?
- Oh, those are great.
- You know what I mean?
- I like them and I think that if it's like a browser plug-in,
- they can do it and you can just grip and rip the data,
- like I like to say, just go for it.
- I mean, you don't need to write code when you don't need to.
- But this is the foundation from which you'll be able to execute
- much more complex scrapes.
- So for instance, you could take this and then automate it
- and set it to run every day at 5 a.m.
- and gradually build a list over a year,
- where if you use a browser plug-in,
- you'd have to go and click it every day at 5 a.m., right?
- So you're able to make it repeatable more easily.
- And then also there's some things you want to scrape,
- and these are the ones that are the most fun,
- that are like paginated.
- So let's say you want to scrape a web page
- that has 5,000 pages of data that you have to go through, right?
- You'd have to go with your browser plug-in,
- you'd have to go and click and click and click,
- which what we could do with this,
- and there's a lot of more advanced scrapes you can look on online
- and if you email me, I can send you some,
- that basically will then click through all the pages
- and repeat the same scrape for every single page
- and then you're able to download hundreds of thousands of records
- by basically just visiting every page on the website
- and extracting the data you want.
- But before you can get there, before you can run,
- you've got to learn to walk.
- But if you just need something one time,
- smash and grab, I just want it, yeah, the plug-ins are great.
- Another just short question.
- In any case, is web scraping illegal?
- Yes.
- If it's a government website, almost certainly no,
- because it's public information.
- If it's a private website that has a terms of service or whatever,
- they oftentimes don't want you to do it.
- Now, there's a difference between collecting it for your own private use
- and republishing any of it legally, and I'm not a lawyer,
- but I think you can scrape things that are private data
- and I think maybe legally argue in some cases
- you're not breaking the law if you just store it
- and don't do anything with it, but if you say republish it, you issue...
- Do you guys have Craigslist here?
- Yeah, so it's classified ads for apartments.
- They don't want anyone scraping them
- because people could scrape them and make a competitor
- or do game the system or something.
- And so they want to make it against the rules of their website,
- which gives them the right to ban you or potentially sue you or whatever,
- and also will develop software that will try to stop you from doing it.
- So they'll see that you're visiting every page
- and they'll kill your request or ban you kind of automatically.
- Also, the same technology that issues this request...
- So hackers are pretty famous now, right? They're everywhere.
- Have you ever heard the phrase denial of service attack?
- Right, so that's basically when you...
- It's the most basic hacker attack.
- It's what Anonymous does to MasterCard or whatever.
- And all it really is is requesting as many web pages as you can
- as quickly as possible so that the web server
- that passes the page gets overwhelmed and crashes.
- So it's just request, request, request, request, request, request,
- request as many times as you want.
- The technology that does that is no more complicated than this.
- It's literally code this simple that makes those requests.
- It just does it many, many times in a row
- and distributed across many different machines,
- which is what a distributed denial of service attack is.
- So if you were to run this Python script over and over again,
- or let's say you were to make a loop,
- you were to make a list that had the number 1 through 1 million,
- 1 million numbers in the list,
- and you were to say, 4 number in, 1 through 1 million, colon,
- space, space, space, this,
- you would then issue 1 million web requests in a row
- for the exact same URL over and over again.
- And you would essentially be running a low-grade denial of service attack.
- Maybe not even intentionally, but you would.
- And so sometimes if it's a crappy government website,
- they'll crash just because they're poorly built
- and they can't handle it and you don't mean it.
- And so what you can do is you can put in delays.
- There's a library called time,
- and you can say inside that loop, 1 through 1 million,
- you could put time.sleep 2 seconds.
- Every time it went through the loop, it would make the request,
- wait 2 seconds, start the loop again.
- Make the request, wait 2 seconds, start the loop again.
- And so with tricks like that, you can kind of make it easier
- on the people you're scraping, and it's kind of polite scraping,
- which I would recommend.
- So don't use this.
- So what I've just told you,
- you could basically be a crappy member of Anonymous at this point
- with that basic technology.
- And as you get more into programming,
- you'll learn a lot of this stuff once you understand it.
- It's not that complicated.
- It's just all these basic data types and loops
- and things that we're going through.
- The last question.
- Thank you.
- If we want to learn more Python, what would you recommend?
- Well, there's a million things on the Internet.
- You know what I mean?
- Google is great for this.
- Well, Google will lead you to many things that are great for this,
- which is you just type in...
- So there's free books that will teach you Python.
- There's one called Dive into Python.
- There's another called How to Think Like a Computer Scientist or something.
- But what they really are is kind of kindergarten Python.
- And they'll teach you actually a lot of the stuff that we just did.
- You know, I'm a member of IRE-NICAR,
- which is a group at the University of Missouri where that jail is.
- And they have a lot of training materials for Python and other languages
- that have kind of journalism examples.
- So if you just type in Python and journalism,
- and you'll find probably a bunch of stuff.
- I can't think of a ton off the top of my head,
- but there are literally hundreds of thousands of things to look at.
- I would say like a beginning Python book would be good.
- But the reality is that computer science,
- you could study it for years and years and years.
- It's very deep.
- What you really need is a combination of the fundamentals
- and then kind of a practical project,
- like a web scrape that you want to work on
- and to partner the two things, and you'll get there.
- Okay?
- Thanks, guys.
- I know that was the crash course.
- That was a crash course, but we survived it, huh?
- Thank you very much.
- Okay.
First Web Scraper
By Ben Welsh • • CCCB Lab in Barcelona