Python – Code Notes

No symlink in zsh with venv

The problem:

Unable to symlink '/Library/Frameworks/Python.framework/Versions/3.8/bin/python3' to '/Users/mcadams/Documents/ ... /env/bin/python3'

This is, I think, an issue with the new Z shell (zsh) in MacOS Catalina. What I had attempted was my usual virtualenv command, at the bash prompt:

python3 -m venv env

That gave me the “Unable to symlink” pain. So I opened my .zshrc file with nano and added this line:

export PATH="/usr/local/opt/python/libexec/bin:/usr/local/bin:$PATH"

THIS FIXED NOTHING.

So then I changed the command to:

python3.8 -m venv env

And it worked. source env/bin/activate and I was in my new env, just like normal.

If you aren’t sure which Python 3 version you’ve got, type: python3 --version and you’ll find out. Note again, this is MacOS, not Windows.

Data science

Data science is recognized as a field distinct from computer science; it has been called “the child of statistics and computer science” (Blei & Smyth, 2017, p. 8689 doi: 10.1073/pnas.1702076114).

Many universities have whipped up degree programs in data science. I’ve searched and examined a lot of the curricula, and my favorite is the master’s program at the University of San Francisco, because it seems very comprehensive, and the faculty have solid credentials.

There’s a block of “foundation courses” from which students must complete two:

MSDS 501 – Computation for Analytics
MSDS 502 – Review of Linear Algebra
MSDS 504 – Review Probability and Stats

Then there are 33 units of required courses:

MSDS 593 – EDA and Visualization
MSDS 601 – Linear Regression Analysis
MSDS 603 – Product Analytics
MSDS 604 – Time Series Analysis
MSDS 605 – Practicum I
MSDS 610 – Communications for Analytics
MSDS 621 – Intro to Machine Learning
MSDS 625 – Practicum II
MSDS 626 – Case Studies in Data Science
MSDS 627 – Practicum III
MSDS 629 – Experiments in Data Science
MSDS 630 – Advanced Machine Learning
MSDS 631 – Special Topics in Analytics
MSDS 632 – Practicum IV
MSDS 633 – Ethics in Data Science
MSDS 689 – Data Structures and Algorithms
MSDS 691 – Relational Databases
MSDS 692 – Data Acquisition
MSDS 694 – Distributed Computing
MSDS 697 – Distributed Data Systems
MSDS 699 – Machine Learning Laboratory

In addition, students must attend seminars and take 10 hours of interview skills training.

This is a one-year full-time residential program that includes 15 hours/week of practicum for nine months of the program.

Some of the things that most impress me about the curriculum:

Three courses on machine learning
A course devoted to ethics
A course on exploratory data analysis and visualization
The 2-unit course on data acquisition focuses on web scraping with Python (check out the course description for this!)
A communications course for learning how to present data to clients and stakeholders
Use of both R and Python; omission of unnecessary programming languages
A course on SQL databases and a separate course on MongoDB
A course on conducting experiments

I have no stake in this master’s degree program (in fact I work at a different university in another state), but when I’ve looked at other programs with “data science” in the title, I’ve concluded that most do not compare favorably with this one.

Mainly I am interested in the intersection of journalism and data science, so I’m continually making comparisons between data-focused journalism projects and the work of data scientists.

Python videos that don’t suck

Note to self: The educational video studio Socratica has some of the best (and most amusing, in a wry way) videos about Python.

> YouTube playlist

Exporting from Bokeh

Bokeh is a Python library for creating interactive data visualizations. I just started learning about it, and I immediately wanted to export either static images or HTML/JavaScript — or both! However, at first it seemed I would need to install extra libraries to make it happen.

Persistently, I kept searching, and I found that there are export options that do not require any extra libraries. Hooray!

So say you have already created and displayed a chart assigned to the variable chart1, using a Jupyter Notebook. This is all you’ll need to export a complete, fully functioning HTML file with included JavaScript:

Lordy, it was torture to embed that freaking gist using the WP Gutenberg editor. New post to follow.

The default toolset in Bokeh includes a “Save” icon. This outputs a PNG image of the chart.

Installing Python for beginners

Students really struggle with setup. By the time they’ve finished setting up Python, Jupyter Notebooks, etc., they’re ready to quit the course and not even learn Python at all — especially students using Windows.

I think with Miniconda I’ve finally tamed that beast. Here are my instructions for students, in one Google doc. Feel free to copy and edit it for your own use.

http://bit.ly/mm-conda

Magic commands in Jupyter

Here’s a Jupyter Notebook I exported:

https://weimergeeks.com/examples/jup/JupyterHowTo.html

That’s not a live notebook, but you can see what it does. Magic commands are pretty awesome. I learned about them in the video that’s linked at the end of that page.

I also found out how to export a notebook as HTML. 😃

Python, data work, and O’Reilly books

I own many O’Reilly books about code. I’m kind of mad that they quit selling PDFs, because I loved those PDFs for searchability, and the Kindle editions are nowhere near as good (they have layout issues that don’t occur in PDFs).

Recently, though, I bought a hardcopy of Python Data Science Handbook, and this inspired me to examine my O’Reilly Python library.

First, a bit about Python Data Science Handbook: It’s a large book, 530 pages, but it has only five chapters:

“iPython: Beyond Normal Python” (all the stuff you can do with the iPython shell, which is different from Jupyter Notebooks)
Intro to NumPy
Pandas
Matplotlib
Machine learning

That list is exactly why I bought this book, even though I already owned others. (See the whole book online.) I especially want to learn more about using Matplotlib in a Jupyter Notebook.

After reading chapters 1 and 2, I went into my older O’Reilly PDFs to see what other Python books I have in that collection. I opened Data Wrangling with Python and ended up spending more time in it than I’d expected, because — surprise! — not only is it completely different from Python Data Science Handbook; it is all about the kinds of things journalists use Python for the most: web scraping, document management, data cleaning. I don’t know why I’ve never spent more time with that book! (See the table of contents.) The first two chapters explain the Python language for beginners, and then it goes on to data types (CSV, JSON, XML) that you need to know about when dealing with data provided by government agencies and the like. There’s a whole chapter on working with PDFs.

The big downside to Data Wrangling with Python is that the examples and code are Python 2.7. I understand why the authors made that choice in 2015, but now it’s a detriment, as those old 2.7 libraries are no longer being maintained. You can still learn from this book, and if you’re a bit experienced with Python and the differences between 2.x and 3.x, it should be easy to work around any issues caused by the 2.7 code.

Another criticism I’d offer about Wrangling is that the chapter “Data Exploration and Analysis” uses agate, a Python library designed for journalists, but in 2019 Pandas (another Python library) would be a much better choice.

I’ve been teaching web scraping with Python to journalism students for four years now, and I’ve used a different O’Reilly book, Web Scraping with Python, by Ryan Mitchell, since the beginning. An updated second edition of Mitchell’s book came out last year, updating from 2.x to 3.x, which is good. (See the table of contents.)

I have several other Python books (including some not from O’Reilly), but as I’m focused here on dealing with data issues (analysis and charts as well as scraping and documents), there’s only one other book I’d like to include in this post. It’s actually not a Python book, but it is from O’Reilly: Doing Data Science, by Schutt and O’Neil. (See the table of contents.) It’s older (published in 2013), but I think it holds up as an introduction to data analysis, algorithms, etc. It even has a chapter titled “Social Networks and Data Journalism.” Charts are in color, which I like very much. There’s not a lot of code in the book — it’s not about showing us how to write the code — and examples are in several languages, including Python, R, and Go.

All four books referenced here are distinctly different from one another. Although there is some overlap, it’s minimal.

(This post was edited in November 2019. After a recent closer reading of several chapters in the first edition of Data Wrangling with Python, I have concluded that it really needs an update, and much of it cannot be comfortably used with today’s libraries.)

Scraping details

I’ve been scraping websites with BeautifulSoup for several years, but not always using the Requests library.

Old way:

from urllib.request import urlopen
from bs4 import BeautifulSoup
url = "https://weimergeeks.com/index.html"
html = urlopen(url)
soup = BeautifulSoup(html, "html.parser")

New way:

import requests
from bs4 import BeautifulSoup
url = "https://weimergeeks.com/index.html"
html = requests.get(url)
soup = BeautifulSoup(html.text, 'html.parser')

So they are really similar, but it turns out that the Requests library offers us two choices for html.text — instead, we could use html.content — so what’s the diff, and does it matter?

As usual, it’s Stack Overflow to the rescue. html.text will be the normal, usual choice. It gives us the content of the HTTP response in unicode, which will suit probably 99.9 percent of all requests. html.content would give us the content of the HTTP response in bytes — meaning raw. We would choose that for a non-HTML file, such as a PDF or an image.

Could regex’s days be numbered?

Regex lets us find anything, check for patterns, format accurately — and drives us crazy with anxiety and deepest discomfort. Regex stands for regular expressions, and my fave regex editor for Python is Pythex (I could not possibly write regex without it).

But there’s hope for the future! Check out Rosie and the Rosie Pattern Language (RPL) for Python!

I watched this video (the whole thing) and it just made me feel so happy. Oh, and Rosie works with other languages too!

This post will help me remember “What was that thing I heard about that can replace regex?” when I need to.