Detect memory leaks of C extensions with psutil and psleak(gmpy.dev)

72 pointsby grodolahace 7 meses10 comments

Detect memory leaks of C extensions with psutil and psleak

https://gmpy.dev/blog/2025/psutil-heap-introspection-apis

10 comments

paradoshace 7 meses

Hey, thank you so much for all your hard work on psutil, it is my goto import! And this looks like a really interesting development you have created to add to the available tools for detecting those tricky memory leaks.

I have done some work in this space as well such as how to write a Python C extension without leaks in the first place [0]. This also contains a section on memory leaks, using tools like your psutil [1] This also shows where the Python documentation for C extensions contains errors, omissions or is misleading. These errors can trap the unwary.

Also I developed a Python memory tracer, pymemtrace, that uses a variety of techniques (including using your psutil!) to track memory usage at different costs and levels of granularity [2].

Now psutil 7.2.0 is out I'll update both those projects.

Thanks again for all your hard work.

[0] Project: https://github.com/paulross/PythonExtensionPatterns Docs: https://pythonextensionpatterns.readthedocs.io/en/latest/ind...

[1] https://pythonextensionpatterns.readthedocs.io/en/latest/mem...

[2] Project: https://github.com/paulross/pymemtrace Docs: https://pymemtrace.readthedocs.io/

ofekhace 7 meses

Your C extension guide looks very useful and I quite like the foreword/history behind it. Have you considered updating the resource to account for the freethreaded mode (which will eventually become the default) on 3.14+?

paradoshace 7 meses

For sure, it is on my TODO list. It takes a lot of work to explore each new Python C API and I'll get around to it when I can.

grodolahace 7 meses

I have released new psutil 7.2.0, which includes 2 new APIs to inspect C heap memory allocations.

I have also released a new tool called psleak, which detects memory leaks in C extension modules.

https://gmpy.dev/blog/2025/psutil-heap-introspection-apis

https://github.com/giampaolo/psleak/

ofekhace 7 meses

This is awesome, thanks a lot! I'm going to introduce this in the test suites of the extension modules I maintain [0][1] and, if all goes well, eventually at work [2].

I really appreciate the support for Windows as that platform is currently underserved [3] when it comes to such memory tooling.

[0]: https://github.com/jcrist/msgspec

[1]: https://github.com/ofek/coincurve

[2]: https://github.com/DataDog/integrations-core

[3]: https://github.com/bloomberg/memray

dsp_personhace 7 meses

In a pinch one way to deal with memory leaks is to wrap the offending work in a multiprocessing pool, forcing it to live in an ephemeral process

ofekhace 6 meses

Indeed, I introduced this [0] as an option for every Python integration that runs on the Datadog Agent.

[0]: https://github.com/DataDog/integrations-core/pull/12986

gortokhace 7 meses

The attached post feels AI-generated/AI-edited. I don’t know why that detracts from the usefulness of this utility, but it — at least how I’m reacting to it — seems to.

For me, it reads as not having any soul to it, as well as changing who its target reader is throughout the post. The choice of emphasis comes across weirdly too.

I think this is a wonderful tool, but whether it’s because the author uses AI to help them write this post, or that there’s AI edited content everywhere now, I’m weary of reading and trusting something that reads like this post does.

I don’t know if this is overblown or meta-commentary on the state of online posts these days, but it’s how I’m feeling, more and more, when I see online writings.

petcathace 7 meses

These kinds of comments (accusing OC of being ai gen) seem like they're showing up in every thread and they're always useless and insulting.

HN needs a rule to ban these

ofekhace 7 meses

The post is entirely authentic; it matches the writing style of the author from before the LLM boom and discusses work that the human author recently released. Can you pinpoint what makes you feel that way?

edit: I asked for explanation before the post was edited to expand on that. I disagree but am sympathetic to the weariness of reading content now that AI use is widespread.