Chapter 8

JSON Files

settings.json is written in JSON, a format for nested data: dicts, lists, strings, numbers, and not much else. It's what most config files and almost every web API use. Python's json module converts between JSON text and Python values, and most of the time it just works. This chapter is mostly about the times it doesn't.

Reading JSON

import json

with open("settings.json", encoding="utf-8") as f:
    settings = json.load(f)

print(settings)
print(settings["books_per_year_goal"])

{'reader': 'Sam', 'books_per_year_goal': 24, 'formats': ['paper', 'ebook', 'audio']}
24

json.load(f) reads the file and hands back plain Python values. The outer {} became a dict, the [] became a list, 24 became an int, and the strings stayed strings. From there it's just data you already know how to work with.

There's also json.loads(s), with an s for "string", which parses JSON you already have in memory, like the body of an API response.

The type mapping

That's the entire vocabulary. JSON has no dates, no tuples, no sets, no custom objects. Everything you load is built from those seven types.

Writing JSON

json.dump is the mirror of json.load:

import json

data = {"reader": "Sam", "books_per_year_goal": 24}

with open("out.json", "w", encoding="utf-8") as f:
    json.dump(data, f)

with open("out.json", encoding="utf-8") as f:
    print(f.read())

{"reader": "Sam", "books_per_year_goal": 24}

By default it writes everything on one line. That's fine for a machine and hard for a person, so for a file you'll open in an editor, pass indent:

import json

data = {"reader": "Sam", "goal": 24, "formats": ["paper", "ebook"]}

print(json.dumps(data, indent=2))

{
  "reader": "Sam",
  "goal": 24,
  "formats": [
    "paper",
    "ebook"
  ]
}

indent=2 puts each key on its own line, indented two spaces. sort_keys=True also orders the keys alphabetically, which makes changes to a config file show up as clean diffs in version control. Without it, the keys keep the order they were loaded or created in.

Updating a config file

The running-example task: raise the yearly reading goal. Load it, change the one value, write the result with the safe-replace pattern from Chapter 3. (The example writes to a new file so the original stays untouched for the rest of the chapter; in real code you'd write back over settings.json itself.)

import json
import os

def write_atomically(path, text):
    tmp = path + ".tmp"
    with open(tmp, "w", encoding="utf-8") as f:
        f.write(text)
    os.replace(tmp, path)

with open("settings.json", encoding="utf-8") as f:
    settings = json.load(f)

settings["books_per_year_goal"] += 6

write_atomically("goal-raised.json", json.dumps(settings, indent=2) + "\n")

with open("goal-raised.json", encoding="utf-8") as f:
    print(f.read(), end="")

{
  "reader": "Sam",
  "books_per_year_goal": 30,
  "formats": [
    "paper",
    "ebook",
    "audio"
  ]
}

json.dumps doesn't add a trailing newline, so the + "\n" puts one on, which most tools and editors expect. And there's the encoding="utf-8" that Chapter 4 promised would join write_atomically.

What doesn't round-trip

A round-trip is writing a value to JSON and reading it back. Most values come back unchanged. A few don't.

Tuples go out as arrays and come back as lists:

import json

sent = {"formats": ("paper", "ebook")}
back = json.loads(json.dumps(sent))
print(back)

{'formats': ['paper', 'ebook']}

Sets and datetimes stop json.dumps with an error:

import json
json.dumps({"tags": {"scifi", "classic"}})

That's TypeError: Object of type set is not JSON serializable. Convert before you dump: list(the_set) for a set, some_date.isoformat() for a date.

Dictionary keys always become strings:

import json

back = json.loads(json.dumps({1: "a", 2: "b"}))
print(back)

{'1': 'a', '2': 'b'}

The integer keys came back as the strings "1" and "2", because JSON object keys can only be strings.

When the file is broken: JSONDecodeError

Config files get hand-edited, and hand-edited JSON gets a stray comma:

import json
json.loads('{"goal": 24,}')

json.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 13 (char 12). The message gives you the line, the column, and the character offset. A file that was cut off mid-download raises the same kind of error with "Expecting value" or "Unterminated string".

If you're reading a file a person might have touched, catch it and say which file:

import json

def load_config(path):
    try:
        with open(path, encoding="utf-8") as f:
            return json.load(f)
    except FileNotFoundError:
        print(f"{path}: not found, using defaults")
        return {}
    except json.JSONDecodeError as e:
        print(f"{path}: invalid JSON ({e})")
        return {}

print(load_config("settings.json")["reader"])
print(load_config("missing.json"))

Sam
missing.json: not found, using defaults
{}

Common mistakes

dump versus dumps. dump writes to a file, dumps returns a string. Same split for load and loads. The s is for "string".

Single quotes. {'a': 1} is valid Python and invalid JSON. JSON requires double quotes on both keys and strings.

Comments. JSON has none. If a config file needs explaining, that's a hint to use a format that allows comments, like TOML, which is out of scope here but worth knowing about.

Expecting a trailing newline. json.dump and json.dumps don't add one.

Practice

Try each of these before you read the solution under it.

1. Write load_settings() and save_settings(settings), where save_settings writes atomically with indent=2 and a trailing newline.
2. Write add_format(name) that adds a format to settings.json if it isn't already listed.
3. Write safe_load(path) that returns the parsed data, or None with a printed warning if the file is missing or not valid JSON.

Solutions

1.

import json
import os

def load_settings():
    with open("settings.json", encoding="utf-8") as f:
        return json.load(f)

def save_settings(settings):
    text = json.dumps(settings, indent=2) + "\n"
    tmp = "settings.json.tmp"
    with open(tmp, "w", encoding="utf-8") as f:
        f.write(text)
    os.replace(tmp, "settings.json")

s = load_settings()
s["reader"] = "Sam Ellison"
save_settings(s)
print(load_settings()["reader"])

Sam Ellison

2. Load, check, append, save. Reuse the two functions from problem 1.

import json
import os

def load_settings():
    with open("settings.json", encoding="utf-8") as f:
        return json.load(f)

def save_settings(settings):
    tmp = "settings.json.tmp"
    with open(tmp, "w", encoding="utf-8") as f:
        f.write(json.dumps(settings, indent=2) + "\n")
    os.replace(tmp, "settings.json")

def add_format(name):
    settings = load_settings()
    if name not in settings["formats"]:
        settings["formats"].append(name)
        save_settings(settings)

add_format("audio")   # already there
add_format("comic")   # new
print(load_settings()["formats"])

['paper', 'ebook', 'audio', 'comic']

3. Two failure modes, one function.

import json

def safe_load(path):
    try:
        with open(path, encoding="utf-8") as f:
            return json.load(f)
    except FileNotFoundError:
        print(f"{path}: missing")
        return None
    except json.JSONDecodeError:
        print(f"{path}: not valid JSON")
        return None

print(safe_load("settings.json")["books_per_year_goal"])
print(safe_load("nope.json"))

24
nope.json: missing
None

Where this leaves you

You can read and write JSON, you know the seven types it's built from and which Python values won't survive the trip, and you can update a config file without risking the old one. Chapter 9 goes back to the files themselves: creating folders, and copying, moving, and deleting.