Python Debugger

Wed, Feb 8, 2023 | python, and programming

I came across an interview with John Carmack where he talked about debuggers, a tool near and dear to my heart. Here are a choice few quotes from Carmack:

It still boggles my mind how hostile to debuggers and IDEs that so much of the big money-get-billions-of-dollars venture backed companies are.

A debugger is how you get a view into a system that it so complicated to understand. Anybody that thinks “just read the code and think about” – that’s an insane statement. You can’t even read all the code on a big system.

While I agree with his statements, I found them strange.

I used debuggers since my early days programming in ruby. One of the first things I did when I joined my current company was figure out how to get the python debugger to run in the development environment. The debugger was a natural extension to the REPL that it made so much sense. Everyone I worked with used the debugger, so I didn’t quite understand why the big companies weren’t using it as much.

Maybe it’s the fact that they don’t use IDE-based debuggers, for which I have no experience with.

Either case, I love debuggers. And since I’ve been learning python, let’s see how the python debugger works.

Python’s debugger is apart of the standard library: pdb. It works well in a pinch, but I find it primitive.

Instead, I rely on ipdb. ipdb has the same interface, but enables tab completion and syntax highlighting; I’m all for looking at pretty terminals.

Let’s set it up.

Install ipdb

To install ipdb, we use pip:

pip install ipdb

Setting Breakpoints

When you need to invoke the debugger, add the following line to where you want to debug code.

# pdb
import pdb; pdb.set_trace()

# ipdb
import ipdb; ipdb.set_trace()

If you find this verbose like I do, and you’re using python 3.7+, then PEP 553 has a nice addition for you: the breakpoint() statement.

breakpoint removes the need to import the debugger package and call set_trace().

Additionally, breakpoint is a single statement rather than two separated by a semi-colon – ensuring black leaves my line alone when I save the file.

And finally, PEP 553 allows configuring which debugger to use with the built-in breakpoint statement. By default, it uses pdb from the standard library. But, you can use ipdb by setting the PYTHONBREAKPOINT environment variable:

PYTHONBREAKPOINT="ipdb.set_trace" python myscript.py

For the rest of this post, I use breakpoint.

Line by Line

Let’s walk through the following python script.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# filename: debugger.py


def say(msg: str) -> None:
    print({msg})


def main() -> None:
    breakpoint()
    print("I'm inside main.")
    prefix = "Loop:"
    for i in range(5):
        say(f"{prefix} {i}")

    print("I'm exiting")


if __name__ == "__main__":
    main()

From the terminal, execute the script:

PYTHONBREAKPOINT="ipdb.set_trace" python debugger.py

The execution should output our source code with line numbers, an arrow, and a prompt.

Welcome to the debugger!

1
2
3
4
5
6

> /Users/peter/code/python-debugger/debugger.py(10)main()
      9     breakpoint()
---> 10     print("I'm inside main.")
     11     prefix = "Loop:"

ipdb>

The execution stopped at line 10.

Let’s break down the output.

  • Line 1: Shows the file the debugger is paused in, the line it stopped on (10), and the function name it is paused in.
  • Line 2-4: Shows the line number in the file (not the output) and the code on that line. On Line 3, the debugger placed an arrow (--->) indicating the next line to execute.
  • Line 6: The debugger prompt. This is where you enter commands.

Now, type “next” and press Enter to submit the command.

The terminal displays:

1
2
3
4
5
6
7
8

>ipdb> next
I'm inside main.
> /Users/peter/code/python-debugger/debugger.py(11)main()
     10     print("I'm inside main.")
---> 11     prefix = "Loop:"
     12     for i in range(10):

ipdb>

The debugger executed the code print("I'm inside main.") and printed I'm inside main. to the terminal. This is shown on line 1 of the output above. The debugger now points to line 11, the next statement in the program. Finally, the debugger is now waiting for you to submit the next command.

So what happened?

The next command continues the program’s execution until it reaches the next line. We’ll use this command often to move through the debugger. Luckily, there’s also a shortcut for next: the letter n.

Actually, many of the commands in this post have shorter forms. I’ll stick with the long form, but you can find the short versions in pdb’s documentation.

Back to the execution.

The program has executed line 10 and stopped. The next line to execute is line 11 which creates a variable.

Type “next” and press Enter.

The output now displays:

ipdb> next
> /Users/peter/code/python-debugger/debugger.py(12)main()
     11     prefix = "Loop:"
  -> 12     for i in range(5):
     13         say(f"{prefix} {i}")

ipdb>

Now we’re ready to look into inspection.

What’s that variable set to?

The program executed line 11 and now the variable prefix exists. Let’s find out what it is – forget the fact that you know what the source code is for a moment.

We can inspect prefix with several commands.

Type “p prefix” and press Enter.

The output now displays:

ipdb> p prefix
'Loop:'
ipdb>

The p command evaluates the expression provided and prints the result. In our case, we evaluated the prefix variable and printed the value: 'Loop:'.

There is related command, pp that does the same thing as p, but uses pprint to print the output. I like to use this when printing nested dicts or lists.

Before we continue, let’s type in one more inspection command.

Type “whatis prefix” and press Enter.

The output now displays:

(Pdb) whatis prefix
<class 'str'>
(Pdb)

The command whatis prints the type of the expression, in this case the type of the variable prefix. whatis effectively executes the following:

print(type(prefix))

Hold on, where am I?

Before we move on, let’s refresh ourselves with where the python executor is paused.

Type “list” and press Enter.

You the output now displays:

(Pdb) list
  7
  8  	def main() -> None:
  9  	    breakpoint()
 10  	    print("I'm inside main.")
 11  	    prefix = "Loop:"
 12  ->	    for i in range(5):
 13  	        say(f"{prefix} {i}")
 14
 15  	    print("I'm exiting")
 16
 17
(Pdb)

The list command shows the familar -> arrow where the execution is paused. In addition, list shows the five lines above and five lines below the ->.

If you typed list and pressed Enter again, list shows the next set of lines. You can repeat the command until the end of the file.

If you decided to list the rest of the file, then to show the current line again, you can type “list 11”.

The related command longlist, when submitted, shows the code for the current function you’re paused in.

Type “longlist” and press Enter.

The output now displays:

(Pdb) longlist
  8  	def main() -> None:
  9  	    breakpoint()
 10  	    print("I'm inside main.")
 11  	    prefix = "Loop:"
 12  	    for i in range(5):
 13  ->	        say(f"{prefix} {i}")
 14
 15  	    print("I'm exiting")
(Pdb)

Yup, that’s the full definition for our main function.

Up and Down the Stack

Let’s return to moving around. We are currently paused right before the say() function call inside our for loop. Let’s learn how to step into functions.

Type “step” and press Enter.

The output now displays:

(Pdb) step
--Call--
> /Users/peter/code/python-debugger/debugger.py(4)say()
-> def say(msg: str) -> None:
(Pdb)

Execution has now paused at the definition of say(). And at this location, we can print out the arguments by using our inspection commands. In otherwords, the execution has moved into say(), before executing the body of the function. At this location, the arguments are available for inspection.

Since we covered inspection, let’s move to the next line and list the source code.

Type “next” and press Enter. Type “list” and press Enter.

The output now displays:

(Pdb) next
> /Users/peter/code/python-debugger/debugger.py(5)say()
-> print({msg})
(Pdb) list
  1  	# filename: debugger.py
  2
  3
  4  	def say(msg: str) -> None:
  5  ->	    print({msg})
  6
  7
  8  	def main() -> None:
  9  	    breakpoint()
 10  	    print("I'm inside main.")
 11  	    prefix = "Loop:"
(Pdb)

Now let’s learn about moving around the call stack since the execution is paused in say().

Type “where” and press Enter.

The output now displays:

(Pdb) where
  /Users/peter/code/python-debugger/debugger.py(19)<module>()
-> main()
  /Users/peter/code/python-debugger/debugger.py(13)main()
-> say(f"{prefix} {i}")
> /Users/peter/code/python-debugger/debugger.py(5)say()
-> print({msg})
(Pdb)

The where command prints the stacktrace where the execution is paused. The stacktrace is printed sequentially with the entrypoint of the script at the top and the location we’re paused at at the bottom. This output displays exactly where we are in the stack trace and provides an anchor for the next two commands: up and down.

Let’s move up to main().

Type “up” and press Enter.

The output is now:

(Pdb) up
> /Users/peter/code/python-debugger/debugger.py(13)main()
-> say(f"{prefix} {i}")
(Pdb)

We are now back at the callsite for say() within our main() function. To move back down, well …

Type “down” and press Enter.

The output is now:

(Pdb) down
> /Users/peter/code/python-debugger/debugger.py(5)say()
-> print({msg})
(Pdb)

Now we’re back.

I like to use up and down to peek into a function callstack as I debug code. They also come in handy when looking into open source libraries!

Let’s continue moving through the say() function and see what happens when it returns.

Type “next” and press Enter.

The output is now:

1
2
3
4
5
6

(Pdb) next
{'Loop: 0'}
--Return--
> /Users/peter/code/python-debugger/debugger.py(5)say()->None
-> print({msg})
(Pdb)

The output here looks intense. Let’s review line by line.

  • Line 1: The command we typed.
  • Line 2: The output from executing the print() function
  • Line 3: Indicates that the function has completed and is returning.
  • Line 4: Shows the filepath, linenumber, function and the return value (None).
  • Line 5: The last line that was executed before the return. Since the execution has actually moved back to the upper stack, the debugger repeats the line to let you see the return value.

Now that execution has paused after returning, we can move forward to the next line and expect to pause at the for loop in main().

Type “next” and press Enter:

The output is now:

(Pdb) next
> /Users/peter/code/python-debugger/debugger.py(12)main()
-> for i in range(5):
(Pdb)

Indeed, we are back in main().

Finishing Up

Since the rest of the program is finishing up the loop and another print statement, let’s finish up.

Type “continue” and press Enter.

The output is now:

(Pdb) continue
{'Loop: 1'}
{'Loop: 2'}
{'Loop: 3'}
{'Loop: 4'}
I'm exiting
(venv) peter@doomslug ~/code/python-debugger %

The continue command resumes the execution. If there are no other breakpoint() function calls in the code path, the execution completes and exits successfully like above.

And we’re done!

There are times when continue isn’t exactly what you want to do. For instance, maybe you found the bug you were looking for and you want to quit.

Luckily, there’s the quit command. It terminates the execution and returns you to the shell.

Help!

One last tip. If you’re in the debugger and forget what the commands do, the help command is handy. Without any arguments, it lists all the available commands. With a command name as the argument, help prints out the documentation for the command.

Until Next Time

With next, step, p, pp, list, longlist, where, up, and down commands, you’ll have plenty to use in your next debugging session.

Enjoy!