Python

The best programming language

Thursday, Apr 9, 2020

Python

Getting started

Installing Python on macOS via Homebrew, at the time of writing, will not install the most current version of Python available, despite it being released 5 months ago.

brew install python@3.8

# Add to .zshenv
path=(/usr/local/opt/python@3.8/libexec/bin ${path})

Silencing the python Console Welcome Message

Normally, when you open the python console, the following welcome message will appear when you enter.

Python 3.7.3 (default, Jun 19 2019, 07:38:49)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

To disable it, use the following command to enter the python console:

python -q

set()

A set in Python is implemented as a hash table. It has O(1) lookup and insertion time.

Adding items to a set:

bag = {1, 2}
array = [1, 2, 3, 4, 4, 5]

# Adding 1 item with add()
set.add(3)

# Adding a list of items using update()
bag.update(array)

# Adding a set of items using |=
bag |= set(array)

os.path()

from os.path import *
from glob import glob

home = expanduser('~')
# => /Users/tommy

sym_folder = join(home, 'tmp')
# => /Users/tommy/tmp

abs_folder = realpath(sym_folder)
# => /Users/tommy/real/location/folder

files = glob(expanduser('~/notes/*.md'))
# => ['/Users/tommy/notes/file1.md', '/Users/tommy/notes/file2.md']

Jupyter Notebook

A Jupyter Notebook brew install jupyter is an open source web application that allows you to create and share documents containing snippets of pre-executed code.

The Jupyter Notebook supports the following languages, among others:

The Jupyter Notebook will also render LaTeX and Markdown, allowing for flexible formatting of data.

jupyter notebook --generate-config
pass="letmein"
python -c "from notebook.auth import passwd; print(passwd('${pass}'))"

Setting Up pylint

pylint --generate-rcfile > ~/.pylintrc

If pylint notifies you about a linting error that you don’t like, add it as an entry, seperated by a comma, to disable. e.g.

disable=missing-docstring,
        invalid-name,
        bare-except

Plotly

Getting Started

Plotly is a visualization tool you can install with pip3 install plotly. You can create an account at plot.ly and generate an API key. Once you have, edit the ~/.plotly/.credentials and insert the following information.

{
  "username": "tommytrojan",
  "api_key": "Fou4dE18o4TUtCz91n6O"
}

Plotting Data

Plotting Online

import pandas as pd
import plotly.plotly as py
import plotly.figure_factory as ff

df = pd.read_csv("earnings.csv")
table = ff.create_table(df)
py.iplot(table, filename='table1')

Tip: When using a jupyter notebook, use the iplot() method, which will plot the data inline. If you’d like the graph to be shared with others, use the plot() method.

Plotting Offline

import plotly.offline as py
import plotly.graph_objs as go
from plotly.offline import init_notebook_mode, plot, iplot
init_notebook_mode()

data = go.Bar(x=df.height, y=df.weight)
figure = [data]
py.iplot(data, filename='height_weight')

Warning: Your graph will not appear if you don’t call the init_notebook_mode() method first!

The only big difference between plotting online and plotting offline is whether to use plotly.plotly or plotly.offline for your import. Both will include the plot() and iplot() methods.

Regular Expressions

Use the re library for regular expressions in python. Regular expression patterns are created by calling re.compile() which accepts two arguments, a raw string, and flags. To specify multiple flags, specify each with the bitwise OR operator |

Regular Expression Flags

Flag Function
re.A Make the pattern match only ASCII characters
re.I Make the pattern case insensitive
re.M The ^ & $ special characters match the start/end of each line in a string, instead of the start/end of the string itself
re.S Allow the . character to match newline characters
re.X Ignore whitespace in the pattern definition, and allow for comments
Special Character Function
+ Match 1 or more
* Match 0 or more
? 0 or 1
{k} Match k consecutive occurances of the preceeding pattern
{m,n} Match from m to n consecutive occurences (inclusive) of the preceeding pattern (as many as possible)
{m,n}? Match from m to n consecutive occurrences (inclusive) of the preceeding pattern (as few as possible)
. Match any character except a newline \n
^ Match the start of the string
$ Match the end of string
( Specify the start of capture group
) Specify the end of capture group
Escaped Character Matches
\t Horizontal tab
\v Vertical tab
\n Newline
\f Form feed
\r Carriage return
\w [a-zA-Z0–9_]
\d [0–9]
\s [\t\n\r\f\v]
\b Specify a word boundary
# Import the RegEx library
import re

# Create a raw-string pattern
pattern = re.compile(r'http[s]?://([^/?:]+)', re.A|re.I)
# Search for the text matching the pattern in "sample"
text = 'https://helpful.wiki/python'

match = pattern.search(text)
print(match.group(0))
# => https://helpful.wiki
print(match.group(1))
# => helpful.wiki

Dates, Times, Timestamps

ISO8601 and RFC3339 are the documents that outline how date and time should be denoted on computers. 1996-12-20T00:39:57Z is an example of an ISO8601 timestamp. The Z denotes that this timestamp is in zulu time, the UTC timezone. The UTC timezone is Coordinated Universal Time and, depending on the time of year, is either 7 or 8 hours ahead of the time in California. The equivalent time in California would be represented as 1996-12-19T16:39:57-08:00

The datetime library has a few packages:

import datetime
import time

event = datetime.date.fromisoformat("2018-12-31")
# -00:00 is an illegal format but Python will interpret it successfully
event = datetime.datetime.fromisoformat("2018-12-31T12:31:58-08:00")
# Using whitespace seperator instead of 'T' character
event = datetime.datetime.fromisoformat("2018-12-31 04:31:58+00:00")

# If it's currently daylight savings
if time.daylight:
  tz = time.altzone
  # => 25200 (# of seconds offset from UTC)

else:
  tz = time.timezone
  # => 28800 (# of seconds offset from UTC)

print(time.tzname)
# => ('PST', 'PDT')

print(time.localtime)
# => time.struct_time(tm_year=2019, tm_mon=7, tm_mday=17, tm_hour=11, tm_min=42, tm_sec=15, tm_wday=2, tm_yday=198, tm_isdst=1)

moment = datetime.datetime.utcnow()

print(moment.isoformat(timespec='seconds'))
# => 2019-07-17T11:25:07

print(moment.isoformat(timespec='milliseconds'))
# => 2019-07-31T02:21:15.125

print(moment.isoformat(sep=' ', timespec='microseconds'))
# => 2019-07-3102:21:15.125991

Setting the timezone

import time
import os
os.environ['TZ'] = 'US/Eastern'
time.tzset()
print(time.tzname)
# => ('EST', 'EDT')
from datetime import datetime
from pendulum import timezone, timezones

for zone in timezones:
  print(zone)
[
  timezone("Pacific/Honolulu"), # -10:00
  timezone("America/Juneau"), # -09:00
  timezone("America/Los_Angeles"), # -08:00
  timezone("America/Denver"), # -07:00
  timezone("America/Chicago"), # -06:00
  timezone("America/New_York"), # -05:00
  timezone("Europe/London"), # +00:00
  timezone("Europe/Paris"), # +01:00
  timezone("Europe/Athens"), # +02:00
  timezone("Europe/Moscow"), # +03:00
  timezone("Asia/Tehran"), # +03:30
  timezone("Asia/Dubai"), # +04:00
  timezone("Asia/Kabul"), # +04:30 (capitol of Afghanistan)
  timezone("Asia/Dushanbe"), # +05:00 (capitol of Tajikstan)
  timezone("Asia/Kathmandu"), # +05:45 (capitol of Nepal)
  timezone("Asia/Dhaka"), # +06:00 (capitol of Bangladesh)
  timezone("Asia/Bangkok"), # +07:00
  timezone("Asia/Shanghai"), # +08:00
  timezone("Asia/Tokyo"), # +09:00
  timezone("Australia/Sydney"), # +10:00
  timezone("Asia/Noumea"), # +11:00 (capitol of New Caledonia)
  timezone("Pacific/Fiji") # +12:00
]
moment = datetime.now()

print(moment.astimezone(zone).isoformat())

Custom Module Locations

If you have your own modules that you want to use, there’s a way to tell python where to look for a module that are added by the import keyword.

To do this, set the environment variable PYTHONPATH in your shell, and export that variable. For example, export PYTHONPATH=~/example/python/modules. Now, all of the folders within this directory will be considered a module. For instance, if ~/example/python/modules/ex was a folder containing python code, now you’d be able to type import ex in future python programs.

Warning: If you forget to call export, python will be unable to reference the PYTHONPATH variable.

Sockets

Server-Side TCP Socket

import socket
    # Create a TCP socket
    mysocket = socket.socket(
        type=socket.SOCK_STREAM
    )

    # Create an address tuple, the interface to bind to, and a port number
    address = ('0.0.0.0', 1234)

    # Bind the socket to the address
    mysocket.bind(address)

    # Listen, allowing 1 pending connection
    mysocket.listen(1)

    # Listen until the process is killed
    while True:

        # Save the accepted connection
        connection, address = mysocket.accept()

        print(f"Accepted connection from {address}")

        # Continue until the transmission has no more data
        while True:
            data, orig_address = connection.recvfrom(4096)

            # If there's no data being transmitted, exit
            if not data:
                break
            else:
                # For now, reply to the same connection, echoing the message
                reply = f"echo \'{data.decode()}\'".encode()
                connection.sendto(reply, address)

        # Close the connection now that the message has been replied to
        connection.close()

print("exit")
from socket import socket

    mysocket = socket.socket()

    address = ('127.0.0.1', 1234)

    # Connect the socket to the server
    mysocket.connect(address)

    # Send a message to the server, registering the name of the client
    message = f'register {args.name}'

    # Send the message to the connection
    mysocket.sendto(message.encode(), address)

    # Save the reply from the response, as well as the address
    reply, address = mysocket.recvfrom(4096)

    # Decode the reply's binary encoding, store as UTF-8 string
    reply = reply.decode()

    print(reply)

    mysocket.close()

    ofile = open(args.logfile, 'w')
    ofile.write('connected to server and registered\n')
    ofile.write('waiting for messages...\n')
    ofile.write('exit')

Pandas

from pandas import DataFrame, Series, read_json
barset["EMA12"] = barset["o"].ewm(span=12).mean()
barset.to_json('ofile.json', date_format='iso', date_unit='s', orient='index')
df = read_json('ofile1.json', orient='index', date_unit='s')
print(df.head())

Type Hints

from typing import List, Set, Dict, Tuple, Optional, Callable, Iterator, Union

# For simple built-in types, just use the name of the type
x: int = 1
x: float = 1.0
x: bool = True
x: str = "test"
x: bytes = b"test"

# For collections, the name of the type is capitalized, and the
# name of the type inside the collection is in brackets
x: List[int] = [1]
x: Set[int] = {6, 7}

# Same as above, but with type comment syntax
x = [1]  # type: List[int]

# For mappings, we need the types of both keys and values
x: Dict[str, float] = {'field': 2.0}

# For tuples, we specify the types of all the elements
x: Tuple[int, str, float] = (3, "yes", 7.5)

# Use Optional[] for values that could be None
x: Optional[str] = some_function()
# Mypy understands a value can't be None in an if-statement
if x is not None:
    print(x.upper())
# If a value can never be None due to some invariants, use an assert
assert x is not None
print(x.upper())
# This is how you annotate a function definition
def stringify(num: int) -> str:
    return str(num)

# And here's how you specify multiple arguments
def plus(num1: int, num2: int) -> int:
    return num1 + num2

# Add default value for an argument after the type annotation
def f(num1: int, my_float: float = 3.5) -> float:
    return num1 + my_float

# This is how you annotate a callable (function) value
x: Callable[[int, float], float] = f

# A generator function that yields ints is secretly just a function that
# returns an iterator of ints, so that's how we annotate it
def g(n: int) -> Iterator[int]:
    i = 0
    while i < n:
        yield i
        i += 1

# You can of course split a function annotation over multiple lines
def send_email(address: Union[str, List[str]],
               sender: str,
               cc: Optional[List[str]],
               bcc: Optional[List[str]],
               subject='',
               body: Optional[List[str]] = None
               ) -> bool:

Subprocesses

Using the subprocess library, you can execute other commands from within your script, and capture the standard input and standard output of those commands

Filepaths

from pathlib import Path

filepath = Path.home() / 'Downloads' / 'meme.jpg'

Function Parameters

There is a new function parameter syntax / to indicate that some function parameters must be specified positionally and cannot be used as keyword arguments

def f(a, b, /, c, d, *, e, f):
    print(a, b, c, d, e, f)

One use case for this notation is that it allows pure Python functions to fully emulate behaviors of existing C coded functions. For example, the built-in divmod() function does not accept keyword arguments:

def divmod(a, b, /):
    "Emulate the built in divmod() function"
    return (a // b, a % b)

Python Image Library

pip install pillow