King Coding
Python File Handling Get the PDF

Chapter 10

Binary and Temporary Files

Every file so far has been text. Plenty aren't: images, audio, compiled programs, the compressed formats in the next chapter. This chapter is reading and writing raw bytes, and a tool that comes up alongside it: the temporary file, for when you need a file that isn't meant to stick around.

Binary mode

Add b to the mode and Python does no decoding at all:

with open("reading-log.csv", "rb") as f:
    head = f.read(30)

print(head)
print(type(head))
Output
b'title,author,rating,finished\nT'
<class 'bytes'>

You get a bytes literal, the b'...' object from Chapter 4. There's no encoding= argument in binary mode, because there's no text step to encode. Writing is "wb", and you hand it bytes, not str.

Reading in chunks

For a text file you loop over lines. A binary file has no lines, so you read a fixed number of bytes at a time, a chunk, until there are none left:

import hashlib

digest = hashlib.sha256()
with open("reading-log.csv", "rb") as f:
    while True:
        chunk = f.read(4096)
        if not chunk:
            break
        digest.update(chunk)

print(digest.hexdigest()[:16])
Output
2718c5ab0ac6530b

f.read(4096) returns up to 4096 bytes, and an empty b"" once the file runs out, which is falsy and ends the loop. This is how you checksum, hash, or copy a file of any size without loading the whole thing into memory. The chunk size is not magic; a few kilobytes is fine.

What kind of file is this?

Many binary formats begin with a fixed sequence of bytes, a magic number, that says what the file is:

signatures = {
    b"\x89PNG": "PNG image",
    b"%PDF": "PDF document",
    b"PK\x03\x04": "ZIP archive",
}

def sniff(path):
    with open(path, "rb") as f:
        start = f.read(8)
    for sig, name in signatures.items():
        if start.startswith(sig):
            return name
    return "unknown"

with open("fake.png", "wb") as f:
    f.write(b"\x89PNG\r\n\x1a\n")

print(sniff("fake.png"))
print(sniff("reading-log.csv"))
Output
PNG image
unknown

This is more reliable than trusting the file extension, which anyone can rename.

struct: bytes as numbers

Sometimes bytes are packed numbers rather than characters. struct reads and writes them with a format string:

import struct

raw = struct.pack(">HH", 1920, 1080)
print(raw)

width, height = struct.unpack(">HH", raw)
print(width, height)
Output
b'\x07\x80\x048'
1920 1080

> means big-endian byte order and H means a 16-bit unsigned integer, so ">HH" is two of them in a row. The struct documentation has the full table of format codes. You reach for this when you're picking apart a binary file format by hand, which is uncommon but does happen.

Temporary files

When you need a file that shouldn't outlive the program, use tempfile. It finds a unique name in the system's temp folder, and TemporaryDirectory cleans up everything inside it when you're done:

import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmp:
    scratch = Path(tmp) / "working.txt"
    scratch.write_text("intermediate results\n", encoding="utf-8")
    print("exists inside the block: ", scratch.exists())

print("exists after the block:   ", scratch.exists())
Output
exists inside the block:  True
exists after the block:    False

The folder and its contents are deleted when the with block ends, whether it finished normally or raised. This is the clean way to do multi-step work that produces files you don't want to keep.

tempfile.NamedTemporaryFile gives you a single temp file that has a real path on disk, which matters when you need to hand that path to another program. By default it's deleted when you close it.

A word on pickle

pickle turns almost any Python object into bytes and back, which looks like a tidy way to save program state. The catch is that loading a pickle can execute code stored inside it.

Never pickle.load a file you didn't write yourself. A pickle from an untrusted source can run anything on your machine. For data that will be shared, or that crosses any kind of trust boundary, use JSON or CSV.

Common mistakes

Mixing str and bytes. open(path, "wb").write("text") raises TypeError. Binary mode wants bytes: call "text".encode("utf-8"), or open in text mode instead.

Reading a binary file in text mode. On Windows the line-ending translation corrupts the bytes, and a \x1a byte can be read as end-of-file. Use "rb" for anything that isn't text.

A sizeless f.read() on a big binary file. Same memory problem as text. Loop with a chunk size.

Holding a Path from a TemporaryDirectory after the block. The folder is gone; the Path object still points at nothing.

Practice

Try each of these before you read the solution under it.

  1. Write file_sha256(path) that returns a file's full hex digest, reading it in chunks.
  2. Write same_contents(a, b) that returns whether two files hold identical bytes, without loading either one whole.
  3. Write hash_in_temp(path) that copies a file into a TemporaryDirectory, hashes the copy with file_sha256, and returns the digest. Nothing is left behind.

Solutions

1.

import hashlib

def file_sha256(path):
    digest = hashlib.sha256()
    with open(path, "rb") as f:
        while True:
            chunk = f.read(4096)
            if not chunk:
                break
            digest.update(chunk)
    return digest.hexdigest()

print(file_sha256("reading-log.csv")[:16])
Output
2718c5ab0ac6530b

2. Read both in step, chunk by chunk, and stop at the first difference.

def same_contents(a, b):
    with open(a, "rb") as fa, open(b, "rb") as fb:
        while True:
            ca = fa.read(4096)
            cb = fb.read(4096)
            if ca != cb:
                return False
            if not ca:
                return True

with open("copy.csv", "wb") as f:
    f.write(open("reading-log.csv", "rb").read())

print(same_contents("reading-log.csv", "copy.csv"))
print(same_contents("reading-log.csv", "settings.json"))
Output
True
False

3. Copy in, hash, let the block clean up.

import hashlib
import shutil
import tempfile
from pathlib import Path

def file_sha256(path):
    digest = hashlib.sha256()
    with open(path, "rb") as f:
        while True:
            chunk = f.read(4096)
            if not chunk:
                break
            digest.update(chunk)
    return digest.hexdigest()

def hash_in_temp(path):
    with tempfile.TemporaryDirectory() as tmp:
        copy = Path(tmp) / Path(path).name
        shutil.copy(path, copy)
        return file_sha256(copy)

print(hash_in_temp("reading-log.csv")[:16])
Output
2718c5ab0ac6530b

Where this leaves you

You can read and write raw bytes, work through a file in chunks so size stops mattering, recognize a file from its first bytes, and use a temporary file or folder for work that shouldn't leave a trace. Chapter 11 is one specific kind of binary file you'll actually meet: the zip archive.