Martin clarified in the podcast that refactoring (the verb/process) is a sequence of very small refactorings, while you keep making sure that you can run the test suite always. A refactoring (noun) is a change where you change the structure of the code without any externally observable changes, with the intent of making it easier to understand and cheaper to change in future.
I also really liked the metaphor of a 'healthy code base'. Refactoring is, then, the process of keeping healthy – exercise, speaking metaphorically. You can stack up all the exercise you need to do, until you get really unfit. Refactoring, similarly, needs to be done regularly, to keep the code base healthy. This lets you go faster, in the future.
I also really liked the advise about trying to push back the mental contexts you build, while trying to debug/understand some code that is not very clear, by refactoring the code to make it clearer. Code needn't be one big chunk of cryptographic text. Code is writing. It should be clear and understandable. Or, at least we should strive to make it so!
The discussion, as always on this podcast, was very lively, pleasant and enjoyable! Enjoy!