hayaliv / Hi! Python / Level 1
JSONL-style records: one JSON object per line
Store one JSON object per line, then read it back line by line and report bad lines by number.
Think of it like this
Think of a stack of index cards where each card holds one complete note. You can read the stack one card at a time, and a smudged card does not stop you reading the next one.
Where the analogy stops being true. Index cards have no strict rules, while each line here must be valid JSON on its own, and one bad line still needs your code to catch it.
What to know
- In a fine-tuning project we often store one example per line like this. It is a working habit, so treat it as practice rather than as part of the json module.
- Build each line with
json.dumps(record). The default output has no indentation, so it fits on one line. - A newline inside a text value is escaped as
\n, so it does not break the line (see predict item 1). - Read with a loop over the lines and call
json.loadson each one. - Skip blank lines before parsing. An empty string is not valid JSON and raises
json.JSONDecodeError. JSONDecodeErroris a subclass ofValueError, so you can catch it and carry on.- Keep your own line counter with
enumerate(..., start=1). The error'slinenocounts lines inside the one string you parsed.
Example
import io
import json
records = [
{"input": "Can we talk Friday?", "label": "meeting_request"},
{"input": "Please remove me.", "label": "not_interested"},
]
text = "\n".join(json.dumps(r) for r in records)
text += "\n\n" + '{"input": "oops"' + "\n"
for number, line in enumerate(io.StringIO(text), start=1):
line = line.strip()
if not line:
continue
try:
record = json.loads(line)
except json.JSONDecodeError as err:
print("bad line", number)
continue
print(number, record["label"])
It prints:
1 meeting_request
2 not_interested
bad line 4
Predict, then check
1. What does this print?
import json
print(json.dumps({"t": "a\nb"}))Show the answer
{"t": "a\nb"}The newline inside the text is written as the two characters backslash and n, so the whole record stays on one line.
2. What does this print?
import json
try:
json.loads("")
except json.JSONDecodeError as e:
print(e.lineno)Show the answer
1The empty string is not a JSON document, so JSONDecodeError is raised. The lineno is 1 because it counts inside the string we parsed.
3. What does this print?
import json
print(issubclass(json.JSONDecodeError, ValueError))Show the answer
TrueJSONDecodeError is a subclass of ValueError, so except ValueError would catch it too.
4. What does this print?
import json
try:
json.loads('{"a": 1,\n"b" 2}')
except json.JSONDecodeError as e:
print(e.lineno)Show the answer
2The mistake is on the second line of the string, so lineno is 2. It counts lines inside the string, not inside a bigger file.
Try it
Write parse_lines(lines). It gets a list of strings, skips blank lines, and returns a pair (records, bad_numbers): the parsed dicts, and the 1-based numbers of lines that are not valid JSON.
def parse_lines(lines):
return [], []
Show a solution
import json
def parse_lines(lines):
records, bad = [], []
for number, line in enumerate(lines, start=1):
line = line.strip()
if not line:
continue
try:
records.append(json.loads(line))
except json.JSONDecodeError:
bad.append(number)
return records, badCommon mistake
Do not report err.lineno as the file line. It counts inside the string you parsed, so keep your own counter (see predict item 2).
In a small-language-model project
A fine-tuning data file with thousands of {"input": ..., "label": ...} lines may well have a few broken ones. Reporting "bad line 412" lets you fix the file instead of losing the whole run.
Sources
Each quote below appears word for word on the cited page of the official Python documentation (version 3.12).
None (the default) selects the most compact representation.
docs.python.org/3.12/library/json.html — indent=None gives the most compact output.If the data being deserialized is not a valid JSON document, a JSONDecodeError will be raised.
docs.python.org/3.12/library/json.html — Invalid input raises JSONDecodeError.Subclass of ValueError with the following additional attributes
docs.python.org/3.12/library/json.html — JSONDecodeError is a ValueError subclass with a lineno attribute.lineno The line corresponding to pos .
docs.python.org/3.12/library/json.html — lineno is the line corresponding to pos.