Jurigged – Hot code reloading for Python(github.com)
github.com
Jurigged – Hot code reloading for Python
https://github.com/breuleux/jurigged
32 comments
You can also run `jurigged -vm IPython`, for what it's worth (or put `import jurigged; jurigged.watch()` in the config). There are a few edge cases where jurigged will work and not IPython's autoreload, and vice versa. Notably, jurigged updates decorated functions a bit more aggressively and only runs the code diff so it may keep some state that IPython won't.
I do this too and it's amazing.
I also import common packages and classes that I use regularly, such as numpy, os, math, collections...
I also import common packages and classes that I use regularly, such as numpy, os, math, collections...
Neat!
Attempted to run on Windows and it almost works on my machine.
- Watchdog's win32 implementation watches directories rather than files. If Jurigged is patched to watch the directory instead, this part works.
- When the toplevel code is patched, the thread dispatching Watchdog events stops and won't pick up further filesystem changes. This can happen when changing code at toplevel or sometimes a file update gets picked up as Delete/Add instead which causes Jurigged to patch toplevel.
Still trying to track down why that happens.
Attempted to run on Windows and it almost works on my machine.
- Watchdog's win32 implementation watches directories rather than files. If Jurigged is patched to watch the directory instead, this part works.
- When the toplevel code is patched, the thread dispatching Watchdog events stops and won't pick up further filesystem changes. This can happen when changing code at toplevel or sometimes a file update gets picked up as Delete/Add instead which causes Jurigged to patch toplevel.
Still trying to track down why that happens.
And the reason Jurigged breaks on changes to toplevel is it's re-running the top level in its own thread... including the infinite loop that is your application.
This makes sense: it has to re-run the module's code. As long as you don't touch the module that kicks off your application it'll work fine. How can we make it work in the same module that runs the app?
One approach that seems to work is to use the: if __name__ == "__main__" guard on your app code, then modify Jurigged to change __name__ to something else upon reloading. I'm not familiar enough with Python internals to tell if changing __name__ is kosher.
Another possibility is to examine the stack and stop executing code when reaching the root of the current stack... although there's no guarantee the code has been running long enough to reach the program's infinite loop such as when the file was modified a split second after the program started.
This makes sense: it has to re-run the module's code. As long as you don't touch the module that kicks off your application it'll work fine. How can we make it work in the same module that runs the app?
One approach that seems to work is to use the: if __name__ == "__main__" guard on your app code, then modify Jurigged to change __name__ to something else upon reloading. I'm not familiar enough with Python internals to tell if changing __name__ is kosher.
Another possibility is to examine the stack and stop executing code when reaching the root of the current stack... although there's no guarantee the code has been running long enough to reach the program's infinite loop such as when the file was modified a split second after the program started.
I mean, if you watch the example gif in the README, it shows the toplevel module being changed (you can see the if __name__ guard on the screen). It should work.
Basically, Jurigged only re-runs the diff: if you don't change the `if __name__ ...`, it shouldn't run it again. If you change a function in the toplevel module (outside of the guard) it should figure that out by comparing the old AST to the new one and only re-execute that specific function.
Can I see the code you're using for your tests? There might be some other issue.
Basically, Jurigged only re-runs the diff: if you don't change the `if __name__ ...`, it shouldn't run it again. If you change a function in the toplevel module (outside of the guard) it should figure that out by comparing the old AST to the new one and only re-execute that specific function.
Can I see the code you're using for your tests? There might be some other issue.
Here's the test code:
https://pastebin.com/Ye8qecgW
The original issue I had is that under Windows sometimes Watchdog picks up the entire file as being Deleted/Added. Seems to be a race condition of some sort because adding "print" statements after picking up a change makes that less likely.
That causes the part guarded by __name__=="__main__" to be re-run under the Jurigged thread: https://pastebin.com/p68NBxmE
Or to get the same results without the Watchdog issue I can change time.sleep(1) to some other number.
It makes sense that if you change the program top-level Jurigged will try to re-run it but at the same time it never makes sense to re-run infinite loops. I've tested changing __name__ when re-running and that works although requires a guard and I also think it might be helpful not to re-run module code that is already running as determined by the thread's call stack.
--- (edited since can't reply) ---
Good call on de-bouncing. I tested with both "notepad" and "notepad++" and get the same behavior. The Delete/Run happens within a millisecond so 0.1s filtering should work. Perhaps that would also allow you to monitor for file Delete/Add to work with editors that use atomic replacement.
With that fixed, avoiding re-executing the main loop isn't as important but I think it's worth doing for robustness. Detecting this also gives you the option of restarting the process instead.
The original issue I had is that under Windows sometimes Watchdog picks up the entire file as being Deleted/Added. Seems to be a race condition of some sort because adding "print" statements after picking up a change makes that less likely.
That causes the part guarded by __name__=="__main__" to be re-run under the Jurigged thread: https://pastebin.com/p68NBxmE
Or to get the same results without the Watchdog issue I can change time.sleep(1) to some other number.
It makes sense that if you change the program top-level Jurigged will try to re-run it but at the same time it never makes sense to re-run infinite loops. I've tested changing __name__ when re-running and that works although requires a guard and I also think it might be helpful not to re-run module code that is already running as determined by the thread's call stack.
--- (edited since can't reply) ---
Good call on de-bouncing. I tested with both "notepad" and "notepad++" and get the same behavior. The Delete/Run happens within a millisecond so 0.1s filtering should work. Perhaps that would also allow you to monitor for file Delete/Add to work with editors that use atomic replacement.
With that fixed, avoiding re-executing the main loop isn't as important but I think it's worth doing for robustness. Detecting this also gives you the option of restarting the process instead.
Oh, I see. Jurigged doesn't watch for add/delete of files, just modifications (which means it'll miss changes from certain editors like vi that use a move instead of modifying an existing file). I think the write's not atomic so the file's flashing empty. It might depend on the editor?
Either way it's a problem. What's the time gap between the deletes and adds? I think debouncing the event so it only executes after e.g. 0.1s with no changes might fix it. I suppose I could also just flat out ignore empty files but that wouldn't be as robust.
Either way it's a problem. What's the time gap between the deletes and adds? I think debouncing the event so it only executes after e.g. 0.1s with no changes might fix it. I suppose I could also just flat out ignore empty files but that wouldn't be as robust.
I made an issue to track this so I don't forget and where it can be discussed further if needed: https://github.com/breuleux/jurigged/issues/2
Explicitly detecting changes to if __name__ == "__main__" is a good idea as well, I think. It's an easy pitfall and I can't really think of any situation where it would work, so I might as well warn the user.
Thanks for the useful feedback :D
Explicitly detecting changes to if __name__ == "__main__" is a good idea as well, I think. It's an easy pitfall and I can't really think of any situation where it would work, so I might as well warn the user.
Thanks for the useful feedback :D
I've been using watchgod [0], which seems to work similarly—not sure if jurigged also monitors all included modules, or is cross-platform-friendly. Anyone know the similarities/differences?
[0] https://github.com/samuelcolvin/watchgod
[0] https://github.com/samuelcolvin/watchgod
I believe watchgod basically restarts the process whenever changes happen. Jurigged does not restart anything: it computes a diff between the old and new code, executes that diff, fudges the code object pointers inside the existing functions, and prays that doesn't break it.
The advantage is that it keeps the program state, so you can edit a program that has a slow startup or salvage a process that has been running for hours. It's especially nice with a repl, if you run jurigged with no arguments or jurigged -m IPython, your changes will be automatically included in your interactive session.
The disadvantage is that it keeps the program state, so if you make changes to init functions for example, it'll most likely just crash. So it depends on your use case. If your process is lightweight and/or you don't care about keeping the current program state, watchgod is probably better for you (I believe there's a similar alternative called hupper that you might want to check out).
The advantage is that it keeps the program state, so you can edit a program that has a slow startup or salvage a process that has been running for hours. It's especially nice with a repl, if you run jurigged with no arguments or jurigged -m IPython, your changes will be automatically included in your interactive session.
The disadvantage is that it keeps the program state, so if you make changes to init functions for example, it'll most likely just crash. So it depends on your use case. If your process is lightweight and/or you don't care about keeping the current program state, watchgod is probably better for you (I believe there's a similar alternative called hupper that you might want to check out).
It seems like this is just watching a file for changes and then calling reload on it? What am I missing?
See the How it works and the following Comparision sections of the readme. It's a bit more advanced than simply reloading.
Reimporting a module would likely work most of the times and be a simpler solution but i think it does not affect already created instances of classes, which this solution seem to do.
Reimporting a module would likely work most of the times and be a simpler solution but i think it does not affect already created instances of classes, which this solution seem to do.
This looks impressive, and equally a recipe for utter confusion. I've always thought that it's a brilliant feature of Python that you can edit code while it's running and not affect the process; unlike, say, bash scripts.
In a proper managed environment (whatever that may mean...), this could be useful as a means for continuous integration. Maybe not worth all the hassle, but maybe there's a usecase where this would be a beloved silver bullet.
Hot code reloading (ala Erlang) is often used in embedded devices for control code where restarting is very expensive. Mostly telco and networking devices.
we use it in prod to instrument code to check up on some internal data, say, from the database, when some forensic analysis is needed
We use something similar in some of the web crawlers in my company, we dynamically load the module that selects the data from a web page so that if something (website and/or requirement) changes we can just swap the module without downtime.
Wait what? I thought Python was an interpreted language. Did I miss something?
Python doesn't re-read a function's source code from the filesystem every time it calls it, and that would be a bad idea in general since you'd inadvertently crash your running programs during development.
This watches for changes on the filesystem and injects them into the running program as well as it can, basically keeping the current program state but with new behaviour.
This is kind of a footgun and there are many changes you can make that will crash or ruin the process, but if you're aware of the limitations it can be pretty useful.
This watches for changes on the filesystem and injects them into the running program as well as it can, basically keeping the current program state but with new behaviour.
This is kind of a footgun and there are many changes you can make that will crash or ruin the process, but if you're aware of the limitations it can be pretty useful.
Imported modules are compiled to bytecode and frozen on import, so changing the code in-place won't change what actually get run when the code is called unless you clear the bytecode cache and re-import.
importlib provide a reload method for this purpose: https://docs.python.org/3.8/library/importlib.html#importlib...
importlib provide a reload method for this purpose: https://docs.python.org/3.8/library/importlib.html#importlib...
Interesting. I've done it with no issues on both 27 and 36. Just edit the code directly in site-packages.
So? Doesn't change the fact that once you've intepreted a function in some file, you've interpreted it and now it's in memory, and you're going to use the same as long as your program runs (even if you try to reload the module).
(I mean, this is the default case without using something like this project, or other tricks).
(I mean, this is the default case without using something like this project, or other tricks).
gloryless(2)
Looks more usable than the reloading library which does a similar thing, going to be interesting to use them together. I found Reloading when it was discussed in 2019:
https://news.ycombinator.com/item?id=21268324
https://news.ycombinator.com/item?id=21268324
Hey author of reloading here - cool to see this mentioned. Would love to know how reloading could be more usable for you.
Since the post on HN in 2019, reloading can also be used as a decorator to load functions from source before execution. For some use cases that may work better than the reloading loop.
Since the post on HN in 2019, reloading can also be used as a decorator to load functions from source before execution. For some use cases that may work better than the reloading loop.
I still use the version from 2019, and I usually do not have much code in tight loops, but rather need to reload different parts of a much bigger application so I tend to decorate the wrong parts of the code. I have no complaints about Reloading, just admiration using the AST module like that was genious.
Make sure to turn off auto-save on your editor :)
To enable it add the following to your ipython_config.py: