I came across an interview with John
Carmack where he talked about debuggers, a
tool near and dear to my heart. Here are a choice few quotes from Carmack:
It still boggles my mind how hostile to debuggers and IDEs that so much of the
big money-get-billions-of-dollars venture backed companies are.
A debugger is how you get a view into a system that it so complicated to
understand. Anybody that thinks “just read the code and think about” – that’s
an insane statement. You can’t even read all the code on a big system.
While I agree with his statements, I found them strange.
I used debuggers since my early days programming in ruby. One of the first
things I did when I joined my current company was figure out how to get the
python debugger to run in the development environment. The debugger was a natural extension to the REPL that it made so much sense.
Everyone I worked with used the debugger, so I didn’t quite understand why the
big companies weren’t using it as much.
Maybe it’s the fact that they don’t use IDE-based debuggers, for which I have no
experience with.
Either case, I love debuggers. And since I’ve been learning python, let’s see how the python debugger works.
Python’s debugger is apart of the standard library:
pdb. It works well in a pinch, but
I find it primitive.
Instead, I rely on ipdb. ipdb has the same
interface, but enables tab completion and syntax highlighting; I’m all for
looking at pretty terminals.
Let’s set it up.
Install ipdb
To install ipdb, we use pip:
Setting Breakpoints
When you need to invoke the debugger, add the following line to where you want
to debug code.
# pdb
import pdb; pdb.set_trace()
# ipdb
import ipdb; ipdb.set_trace()
If you find this verbose like I do, and you’re using python 3.7+, then PEP
553
has a nice addition for you:
the breakpoint() statement.
breakpoint removes the need to import the debugger package and call
set_trace().
Additionally, breakpoint is a single statement rather
than two separated by a semi-colon – ensuring black leaves my line alone when
I save the file.
And finally, PEP 553 allows configuring which debugger to use with the built-in
breakpoint statement. By default, it uses pdb from the standard library.
But, you can use ipdb by setting the PYTHONBREAKPOINT environment
variable:
PYTHONBREAKPOINT="ipdb.set_trace" python myscript.py
For the rest of this post, I use breakpoint.
Line by Line
Let’s walk through the following python script.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
# filename: debugger.py
def say(msg: str) -> None:
print({msg})
def main() -> None:
breakpoint()
print("I'm inside main.")
prefix = "Loop:"
for i in range(5):
say(f"{prefix} {i}")
print("I'm exiting")
if __name__ == "__main__":
main()
|
From the terminal, execute the script:
PYTHONBREAKPOINT="ipdb.set_trace" python debugger.py
The execution should output our source code with line numbers, an arrow, and a prompt.
Welcome to the debugger!
1
2
3
4
5
6
|
> /Users/peter/code/python-debugger/debugger.py(10)main()
9 breakpoint()
---> 10 print("I'm inside main.")
11 prefix = "Loop:"
ipdb>
|
The execution stopped at line 10.
Let’s break down the output.
- Line 1: Shows the file the debugger is paused in, the line it stopped on
(10), and the function name it is paused in.
- Line 2-4: Shows the line number in the file (not the output) and the code
on that line. On Line 3, the debugger placed an arrow (
--->)
indicating the next line to execute.
- Line 6: The debugger prompt. This is where you enter commands.
Now, type “next” and press Enter to submit the command.
The terminal displays:
1
2
3
4
5
6
7
8
|
>ipdb> next
I'm inside main.
> /Users/peter/code/python-debugger/debugger.py(11)main()
10 print("I'm inside main.")
---> 11 prefix = "Loop:"
12 for i in range(10):
ipdb>
|
The debugger executed the code print("I'm inside main.") and printed I'm inside main. to the terminal. This is shown on line 1 of the output above. The
debugger now points to line 11, the next statement in the program. Finally, the
debugger is now waiting for you to submit the next command.
So what happened?
The next command continues the program’s execution until it reaches the next
line. We’ll use this command often to move through the debugger. Luckily,
there’s also a shortcut for next: the letter n.
Actually, many of the commands in this post have shorter forms. I’ll stick with
the long form, but you can find the short versions in pdb’s
documentation.
Back to the execution.
The program has executed line 10 and stopped. The next
line to execute is line 11 which creates a variable.
Type “next” and press Enter.
The output now displays:
ipdb> next
> /Users/peter/code/python-debugger/debugger.py(12)main()
11 prefix = "Loop:"
-> 12 for i in range(5):
13 say(f"{prefix} {i}")
ipdb>
Now we’re ready to look into inspection.
What’s that variable set to?
The program executed line 11 and now the variable prefix exists. Let’s find
out what it is – forget the fact that you know what the source code is for a
moment.
We can inspect prefix with several commands.
Type “p prefix” and press Enter.
The output now displays:
ipdb> p prefix
'Loop:'
ipdb>
The p command evaluates the expression provided and prints the result. In our
case, we evaluated the prefix variable and printed the value: 'Loop:'.
There is related command, pp that does the same thing as p, but uses
pprint
to print the output. I like to use this when printing nested dicts or lists.
Before we continue, let’s type in one more inspection command.
Type “whatis prefix” and press Enter.
The output now displays:
(Pdb) whatis prefix
<class 'str'>
(Pdb)
The command whatis prints the type of the expression, in this case the type of
the variable prefix. whatis effectively executes the following:
Hold on, where am I?
Before we move on, let’s refresh ourselves with where the python executor is
paused.
Type “list” and press Enter.
You the output now displays:
(Pdb) list
7
8 def main() -> None:
9 breakpoint()
10 print("I'm inside main.")
11 prefix = "Loop:"
12 -> for i in range(5):
13 say(f"{prefix} {i}")
14
15 print("I'm exiting")
16
17
(Pdb)
The list command shows the familar -> arrow where the execution is paused.
In addition, list shows the five lines above and five lines below the ->.
If you typed list and pressed Enter again, list shows the next set of lines.
You can repeat the command until the end of the file.
If you decided to list the rest of the file, then to show the current line
again, you can type “list 11”.
The related command longlist, when submitted, shows the code for the
current function you’re paused in.
Type “longlist” and press Enter.
The output now displays:
(Pdb) longlist
8 def main() -> None:
9 breakpoint()
10 print("I'm inside main.")
11 prefix = "Loop:"
12 for i in range(5):
13 -> say(f"{prefix} {i}")
14
15 print("I'm exiting")
(Pdb)
Yup, that’s the full definition for our main function.
Up and Down the Stack
Let’s return to moving around. We are currently paused right before the say()
function call inside our for loop. Let’s learn how to step into functions.
Type “step” and press Enter.
The output now displays:
(Pdb) step
--Call--
> /Users/peter/code/python-debugger/debugger.py(4)say()
-> def say(msg: str) -> None:
(Pdb)
Execution has now paused at the definition of say(). And at this location, we
can print out the arguments by using our inspection commands. In otherwords, the
execution has moved into say(), before executing the body of the function. At
this location, the arguments are available for inspection.
Since we covered inspection, let’s move to the next line and list the source
code.
Type “next” and press Enter.
Type “list” and press Enter.
The output now displays:
(Pdb) next
> /Users/peter/code/python-debugger/debugger.py(5)say()
-> print({msg})
(Pdb) list
1 # filename: debugger.py
2
3
4 def say(msg: str) -> None:
5 -> print({msg})
6
7
8 def main() -> None:
9 breakpoint()
10 print("I'm inside main.")
11 prefix = "Loop:"
(Pdb)
Now let’s learn about moving around the call stack since the execution is paused in say().
Type “where” and press Enter.
The output now displays:
(Pdb) where
/Users/peter/code/python-debugger/debugger.py(19)<module>()
-> main()
/Users/peter/code/python-debugger/debugger.py(13)main()
-> say(f"{prefix} {i}")
> /Users/peter/code/python-debugger/debugger.py(5)say()
-> print({msg})
(Pdb)
The where command prints the stacktrace where the execution is paused. The
stacktrace is printed sequentially with the entrypoint of the script at the top
and the location we’re paused at at the bottom. This output displays exactly
where we are in the stack trace and provides an anchor for the next two
commands: up and down.
Let’s move up to main().
Type “up” and press Enter.
The output is now:
(Pdb) up
> /Users/peter/code/python-debugger/debugger.py(13)main()
-> say(f"{prefix} {i}")
(Pdb)
We are now back at the callsite for say() within our main() function. To
move back down, well …
Type “down” and press Enter.
The output is now:
(Pdb) down
> /Users/peter/code/python-debugger/debugger.py(5)say()
-> print({msg})
(Pdb)
Now we’re back.
I like to use up and down to peek into a function callstack as I debug code.
They also come in handy when looking into open source libraries!
Let’s continue moving through the say() function and see what happens when it
returns.
Type “next” and press Enter.
The output is now:
1
2
3
4
5
6
|
(Pdb) next
{'Loop: 0'}
--Return--
> /Users/peter/code/python-debugger/debugger.py(5)say()->None
-> print({msg})
(Pdb)
|
The output here looks intense. Let’s review line by line.
- Line 1: The command we typed.
- Line 2: The output from executing the
print() function
- Line 3: Indicates that the function has completed and is
returning.
- Line 4: Shows the filepath, linenumber, function and the return value
(
None).
- Line 5: The last line that was executed before the return. Since the
execution has actually moved back to the upper stack, the debugger repeats
the line to let you see the return value.
Now that execution has paused after returning, we can move forward to the next
line and expect to pause at the for loop in main().
Type “next” and press Enter:
The output is now:
(Pdb) next
> /Users/peter/code/python-debugger/debugger.py(12)main()
-> for i in range(5):
(Pdb)
Indeed, we are back in main().
Finishing Up
Since the rest of the program is finishing up the loop and another print statement, let’s finish up.
Type “continue” and press Enter.
The output is now:
(Pdb) continue
{'Loop: 1'}
{'Loop: 2'}
{'Loop: 3'}
{'Loop: 4'}
I'm exiting
(venv) peter@doomslug ~/code/python-debugger %
The continue command resumes the execution. If there are no other
breakpoint() function calls in the code path, the execution completes and
exits successfully like above.
And we’re done!
There are times when continue isn’t exactly what you want to do. For instance,
maybe you found the bug you were looking for and you want to quit.
Luckily, there’s the quit command. It terminates the execution and returns you
to the shell.
Help!
One last tip. If you’re in the debugger and forget what the commands do, the
help command is handy. Without any arguments, it lists all the available
commands. With a command name as the argument, help prints out the
documentation for the command.
Until Next Time
With next, step, p, pp, list, longlist, where, up, and down
commands, you’ll have plenty to use in your next debugging session.
Enjoy!
