6 Lesson 06: Webscraping

6.1 Getting to Know WGET

Here we will discuss webscraping — the main process of efficiently collecting large volumes of information from the Internet.

6.2 Software

  • wget (https://www.gnu.org/software/wget/), a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS the most widely-used Internet protocols. It is a non-interactive command line tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

  • NB: on installing wget:

    • On Windows (the easiest): download from https://eternallybored.org/misc/wget/ > choose the latest 64-bit ZIP file (EXE will most likely be blocked by your browser as a potentially dangerous file).
      • Unzip the file and copy wget.exe to the folder where you are planning to scrape data; NB: the easiest approach on Windows is to move/copy this file into relevant folders.
    • On Mac (and, possibly, Linux): brew install wget

6.3 Class

  • practical examples of working with wget
  • single link download
  • batch download
    • web-page analysis
    • extraction of links with regular expressions
    • modification of links with regular expressions

6.4 Sample commands

wget link
wget -i file_with_links.txt
wget -i file_with_links.txt -P ./folderYouWantToSaveTo/ -nc

Where:

  • -P is a folder parameter, which instructs wget where you want to store downloaded files (optional).
  • -nc is a no-clobber parameter, which instructs wget to skips files, if they already exist (optional)

Link examples:

https://maximromanov.github.io/dh_in_mes/files/articles/1860-11-12_article_01.txt
https://maximromanov.github.io/dh_in_mes/files/articles/1860-11-12_article_02.txt
https://maximromanov.github.io/dh_in_mes/files/articles/1860-11-12_article_03.txt

NB: there are many other parameters with which you can adjust wget to your needs.

6.4.1 Issues with WGET on Windows

As we have run into a number of issues with trying to run WGET on Windows, here are some steps that will help to run it smoothly:

  • first of all, WGET does not seem to play well with Powershell; it does work without any problems via Command Prompt (if your Windows “speaks” German, it is called Eingabeaufforderung; but if you search for Command Prompt, you should still be able to find it.). It also works without any issues via Git-Bash. The following steps will make it easier to use with both Command Prompt and Git-Bash.
  • download the wget.exe file (from here: https://eternallybored.org/misc/wget/);
  • copy/paste it into the C:\Windows\System32 folder (C:\Windows\System32 is a part of the so-called PATH — a series of paths which all Windows command line tools check);
  • now, all the commands should be working as expected;

NB: There is an alternative to WGET on Windows Powershell. Here is a detailed tutorial: https://adamtheautomator.com/powershell-download-file/.

6.4.2 Issues with WGET on Mac

It may so happen that WGET will not work as intended with the homework assignment (specifically, you may not be able to download all the issues of Dispatch). From what I understand this is specifically a Mac issue. Last year the MacOS changed their main command line program from bash to zsh, and in some cases WGET may not work as intended under zsh. The solution is, luckily, rather simple: we just need to install bash and run WGET from bash.

  • to install bash, run brew install bash;
  • after that you can start bash by running bash on the command line; the prompt (the beginning of the command line where you type in your commands) should change into something like: bash-5.1$
  • now you can go to a folder where you want to save your downloaded results and run WGET;
  • NB: you will need bash only for this step; after you restart the Terminal, you will be back to the default zsh (you can also run exit command to quit bash and return to zsh).

6.5 A sidenote on issues in general

Keep in mind that one of the most important things that you need to learn in this course is that there are multiple solutions to most of the problems and tasks that you may face and the most common way to solve your problem is to break is down into smaller tasks (remember, there was a detailed discussion of this in Zelle’s book), and then look for efficient solutions to each step. No matter how advanced you are, “googling” will be the major way of finding suitable solutions.

6.6 Practicing Scraping

The following sections give you some examples of links that you are most likely to encounter in real life. Your task is to figure out how to prepare lists of links (URLs) for downloading with WGET. Your first step will be to look under the hood of the current page, which you can do right clicking on the page and selecting something that looks like “View page source” (in Chrome or Edge) or “Show Page Source” (in Safari; you will need to enable “Show Develop menu in menu bar” in Preferences > Advanced). Now, looking at the HTML code of the page you can find the actual URLs, which you can then extract from the HTML code with regular expressions (one thing you can do is to copy/paste the entire code of the page into a text editor that supports regular expressions—like Sublime Text).

6.9 Practice 3 (aka Homework): a tiny-bit tricky

6.10 Reference Materials

6.11 Homework

  1. Scraping the “Dispatch”: download issues of “Richmond Times Dispatch” (Years 1860-1865, only!), which are available at: http://www.perseus.tufts.edu/hopper/collection?collection=Perseus:collection:RichTimes)
  2. In a separate markdown file, describe your steps of how you completed this task (to be uloaded with the rest of your homework).

Python

  • Work through Chapters 8 and 11 of Zelle’s book; read chapters carefully; work through the chapter summaries and exercises; complete the following programming exercises: 1-8 in Chapter 8 and 1-11 in Chapter 11;
  • Watch Dr. Vierthaler’s videos:
    • Episode 12: Functions
    • Episode 13: Libraries and NLTK
    • Episode 14: Regular Expressions
  • Note: the sequences are somewhat different in Zelle’s textbook and Vierthaler’s videos. I would recommend you to always check Vierthaler’s videos and also check videos which cover topics that you read about in Zelle’s book.

6.12 Submitting homework

  • Homework assignment must be submitted by the beginning of the next class;
  • Email your homework to the instructor.
    • if your homework is to create a file, email it as an attachment
    • if your homework is a blogpost on your website, email the link to your website and to the blogpost with your homework.
    • In the subject of your email, please, add the following: 070112-LXX-HW-YourLastName-YourMatriculationNumber, where LXX is the lesson for which the homework is submitted, YourLastName is your last name, and YourMatriculationNumber is your matriculation number.