WinDirStat – Windows Directory Statistics(windirstat.net)
windirstat.net
WinDirStat – Windows Directory Statistics
https://windirstat.net/
200 comments
What's the downside of just reading the MFT? Why doesn't Microsoft do it in file explorer, and why wouldn't every tool use it instead of walking through the file system? Maybe there's no downside but it's such a huge speed boost that it would be weird to not use it otherwise, right?
>What's the downside of just reading the MFT? Why doesn't Microsoft do it in file explorer, and why wouldn't every tool use it instead of walking through the file system?
One disadvantage is that you can't read the MFT of network shares or device emulators presenting "virtual drive letters" to the OS.
The typical (and slower) Win32 API functions FindFirstFile()/FindNextFile() used to iterate through the files structure work at a higher level of abstraction so they work on more targets that don't have an NTFS MFT. Indeed, if you point WizTree to a SMB network share, it will be a lot slower because it can't directly read the MFT.
It's conceivable that Microsoft developers could have programmed Windows Explorer differently to have an optimized code path of reading MFT for local disks and then fall back to slower FindFirstFile()/FindNextFile() for non-MFT disks. Maybe that adds too much complexity and weird bugs. I notice that most of the 3rd-party "Win Explorer replacement" utilities also don't read MFT.
One disadvantage is that you can't read the MFT of network shares or device emulators presenting "virtual drive letters" to the OS.
The typical (and slower) Win32 API functions FindFirstFile()/FindNextFile() used to iterate through the files structure work at a higher level of abstraction so they work on more targets that don't have an NTFS MFT. Indeed, if you point WizTree to a SMB network share, it will be a lot slower because it can't directly read the MFT.
It's conceivable that Microsoft developers could have programmed Windows Explorer differently to have an optimized code path of reading MFT for local disks and then fall back to slower FindFirstFile()/FindNextFile() for non-MFT disks. Maybe that adds too much complexity and weird bugs. I notice that most of the 3rd-party "Win Explorer replacement" utilities also don't read MFT.
> It's conceivable that Microsoft developers could have programmed Windows Explorer differently to have an optimized code path of reading MFT for local disks and then fall back to slower FindFirstFile()/FindNextFile() for non-MFT disks
Surely this would have been worth doing, even if it meant flushing out bugs elsewhere.
Surely this would have been worth doing, even if it meant flushing out bugs elsewhere.
[deleted]
Along with the reasons others have mentioned, it would also bypass any filter driver in the file system stack (Windows has the concept of a stack of filter drivers that can sit in front of the file system or hardware) and would also ignore any permissions (ACLs) on who can see those files. There’s no way they can credibly use this technique outside of say something from SysInternals: it violates the security and layering of the operating system and its APIs.
Is there a Linux equivalent for those "filters"? I'm a bit clueless about win32 and NT sadly enough...
Would that mean that there's no way to "scope" the MFTs?
Edit: That also makes sense, since if I got it right they aren't necessarily supposed to be consumed by userspace programs?
I guess that's why those tools always ask for admin access and basically all perms to the FS.
It's a bit sad that the user gets exposed to a much slower search and FS experience even if the system underneath has the potential to be as fast as it gets. And I don't think ReFS is intended to replace NTFS (not that it's necessarily more performant anyways)
Would that mean that there's no way to "scope" the MFTs?
Edit: That also makes sense, since if I got it right they aren't necessarily supposed to be consumed by userspace programs?
I guess that's why those tools always ask for admin access and basically all perms to the FS.
It's a bit sad that the user gets exposed to a much slower search and FS experience even if the system underneath has the potential to be as fast as it gets. And I don't think ReFS is intended to replace NTFS (not that it's necessarily more performant anyways)
There is no equivalent on Linux. That's why linux has no online antivirus scanners (scanners that scan the file as it's opened) while this is a basic feature of every antivirus program on Windows.
Linux has device mappers (dm-crypt, dm-raid and friends). But those sit below the file system, emulating a device. Window's file system filter drivers sit above the file system, intercepting API calls to and from the file system. That's super useful if you want to check file contents on access, track where files are going, keep an audit log of who accessed a file, transparently encrypt single files instead of whole volumes, etc. But you pay the price for all that flexibility in performance.
Linux has device mappers (dm-crypt, dm-raid and friends). But those sit below the file system, emulating a device. Window's file system filter drivers sit above the file system, intercepting API calls to and from the file system. That's super useful if you want to check file contents on access, track where files are going, keep an audit log of who accessed a file, transparently encrypt single files instead of whole volumes, etc. But you pay the price for all that flexibility in performance.
Sure there is, you're talking about fanotify.
https://man7.org/linux/man-pages/man7/fanotify.7.html
https://lwn.net/Articles/339399/
It even lets you block the access until the scan/decision is made.
https://man7.org/linux/man-pages/man7/fanotify.7.html
https://lwn.net/Articles/339399/
It even lets you block the access until the scan/decision is made.
> That's super useful if you want to check file contents on access, track where files are going, keep an audit log of who accessed a file, transparently encrypt single files instead of whole volumes
Or if you just want to generally make the filesystem so slow that everyone has to invent their own pack files just to avoid file system api calls as much as possible.
Or if you just want to generally make the filesystem so slow that everyone has to invent their own pack files just to avoid file system api calls as much as possible.
What are the APIs related to this named?
IO Minifilter drivers are the modern version: https://learn.microsoft.com/en-us/windows-hardware/drivers/i...
Filters are vaguely similar to things like mountpoints overlaying portions of the filesystem. E.g. in Linux you might have files in /d1/d2/{f1,f2,f3} in the root filesystem but you also have a mountpoint of a 2nd filesystem on /d1/d2 that completely changes the visibility / contents of d2. Filter drivers can do similar things (although they are not actually independent mountpoints).
I believe they're approximately equivalent to FUSE
> it violates the security ... of the operating system
Maybe stating the obvious, but if the security can be violated that easily, it's not very secure.
Maybe stating the obvious, but if the security can be violated that easily, it's not very secure.
You need admin permissions to read the MFT on Windows. The traditional security model of both Windows and Linux assumes that the kernel is a security barrier between system and unprivileged user, and between different unprivileged users. An admin being able to bypass security restrictions isn't traditionally seen as a problem.
Indeed, only in very recent history has the admin/root user/owner been seen as a threat to the system and the system employs defenses against them. I'm hoping that trend reverses because I really hate the direction things are going.
There are pretty good reasons to do that. We've been really lax in what is allowed to run as root/admin when in reality, those permissions should only be used when doing things like reading the MFT or snooping on all the network traffic with Wireshark. It should not be required to run as root/admin in order to install most software because installing software is a very common thing to do.
Even if you want more control over your system, I still think technically capable people would be better served by having a separate administrator account from your normal day-to-day account which you have to explicitly log into (so no UAC prompts, you need to go onto that other account and then you get the UAC prompt). Unfortunately, I think most Desktop OSes are still too unusable with this sort of workflow due to how much software insists on admin for installation.
Even if you want more control over your system, I still think technically capable people would be better served by having a separate administrator account from your normal day-to-day account which you have to explicitly log into (so no UAC prompts, you need to go onto that other account and then you get the UAC prompt). Unfortunately, I think most Desktop OSes are still too unusable with this sort of workflow due to how much software insists on admin for installation.
I largely agree. I think what makes the "the user is a threat" model so difficult to me is that there is a lot of truth to it. Users often don't know enough to make good decisions.
I really like your idea of logging in separately, such that is isn't something you're going to do cavalierly. That seems like a great compromise to me! I fully agree that we way overuse admin and really don't need it for the majority of things.
I really like your idea of logging in separately, such that is isn't something you're going to do cavalierly. That seems like a great compromise to me! I fully agree that we way overuse admin and really don't need it for the majority of things.
Then it doesn't violate the security of the OS, if you need to be an admin to do it.
> it would also bypass any filter driver in the file system stack
The main use case for filter drivers is antivirus, and that is primarily about file contents not file metadata - so if MFT access bypassed filter drivers, that might not be a major issue. I think most non-antivirus use cases are also primarily about data not metadata.
If necessary, one could even devise a design in which MFT access is combined with filter drivers - MFT scanning to find matching files, then for each matched file access its metadata via standard APIs (to ensure filter drivers are invoked) before returning to client. That would be slower than a pure MFT scan but still faster than a scan done purely with standard APIs. A registry key could turn this on/off so sites can decide for themselves where to place the performance versus security tradeoff
> and would also ignore any permissions (ACLs) on who can see those files
They could expose an API which enables MFT scanning with some degree of ACL checking added.
If you do the ACL check as late as possible in processing the query, it would give much better performance than standard APIs that evaluate ACLs on every access. For example, suppose I want to scan a volume for all files with the extension ‘*.exe’. The API would only have to do an ACL check on each matching entry, not on every entry it considers.
There also might be reasonable situations in which ACL checking could be bypassed. For example, if I am requesting a search for files of which I am the owner, just assume the owner should have the right to read the file’s metadata. Or, if I have read permission on a directory, assume I am allowed aggregate information on the count and total size of files in that directory and its recursive subdirectories. These “bypasses” could be controlled by system settings (registry entries / group policy), so customers with higher security needs could disable them at the cost of reduced performance.
Rather than putting this in the OS kernel, it could be a privileged system service which exports an API over LPC/COM/etc. Actually with that design it isn’t even necessary to wait for Microsoft to implement this, it could always be implemented as an open source project, if someone felt sufficiently motivated to do so. (Or even as a proprietary product, although I suspect that would limit its adoption, and the risk is if it takes off, Microsoft would just implement the same thing as a standard part of Windows.)
The main use case for filter drivers is antivirus, and that is primarily about file contents not file metadata - so if MFT access bypassed filter drivers, that might not be a major issue. I think most non-antivirus use cases are also primarily about data not metadata.
If necessary, one could even devise a design in which MFT access is combined with filter drivers - MFT scanning to find matching files, then for each matched file access its metadata via standard APIs (to ensure filter drivers are invoked) before returning to client. That would be slower than a pure MFT scan but still faster than a scan done purely with standard APIs. A registry key could turn this on/off so sites can decide for themselves where to place the performance versus security tradeoff
> and would also ignore any permissions (ACLs) on who can see those files
They could expose an API which enables MFT scanning with some degree of ACL checking added.
If you do the ACL check as late as possible in processing the query, it would give much better performance than standard APIs that evaluate ACLs on every access. For example, suppose I want to scan a volume for all files with the extension ‘*.exe’. The API would only have to do an ACL check on each matching entry, not on every entry it considers.
There also might be reasonable situations in which ACL checking could be bypassed. For example, if I am requesting a search for files of which I am the owner, just assume the owner should have the right to read the file’s metadata. Or, if I have read permission on a directory, assume I am allowed aggregate information on the count and total size of files in that directory and its recursive subdirectories. These “bypasses” could be controlled by system settings (registry entries / group policy), so customers with higher security needs could disable them at the cost of reduced performance.
Rather than putting this in the OS kernel, it could be a privileged system service which exports an API over LPC/COM/etc. Actually with that design it isn’t even necessary to wait for Microsoft to implement this, it could always be implemented as an open source project, if someone felt sufficiently motivated to do so. (Or even as a proprietary product, although I suspect that would limit its adoption, and the risk is if it takes off, Microsoft would just implement the same thing as a standard part of Windows.)
Reading the MFT directly requires Administrator permissions, and doing it correctly means reimplementing support for every nook and cranny of NTFS including things like hard links, junction/reparse/mount points, sparse files, etc.
Spacemonger uses the MFT and doesn't require Administrator privileges
AFAIR MFT access requires Administrator/SYSTEM rights and there is absolutely no way to read it as a regular user.
The only workaround (used by Everything by VoidTools) is to install a service which would run with a needed rights and communicate with it in the GUI.
The only workaround (used by Everything by VoidTools) is to install a service which would run with a needed rights and communicate with it in the GUI.
You call that a workaround but it’s basically the best possible situation security-wise. If this didn’t work securely then it wouldn’t be possible to implement disk defragmenter or even explorer. It’s so core to Windows NT’s security model that I wouldn’t call it a workaround.
You do similar things even with more modern stacks - assign a permission to an application and grant permissions to the application to the user.
The only real concern is that Windows NT permissions are not as granular as they could be.
You do similar things even with more modern stacks - assign a permission to an application and grant permissions to the application to the user.
The only real concern is that Windows NT permissions are not as granular as they could be.
> Windows NT permissions are not as granular as they could be.
For objects, Windows NT permissions are ridiculously granular; e.g. GENERIC_WRITE can be mapped to a half-dozen separately settable type-specific flags, depending on the object type (file, named pipe, etc.). It’s too granular for even an administrator to make sense of, arguably, and the documentation is somewhere between bad and nonexistent. (The UI varies from decent, like the ACL editor you can access from e.g. Explorer, to “you can’t make this shit up”, like SDDL[1].)
For subjects, the situation is not good, like on every other conventional OS. You could deal with that by introducing a “user” for each app, as on Android. But I’m not aware of any attempts to do that (that would expose this mechanism in a user-visible way).
(Then there’s the UWP sandbox, which as far as I tell is build with complete disregard of the fundamental concepts above. I don’t think it’s worth taking seriously at this time.)
[1] https://learn.microsoft.com/en-us/windows/win32/secauthz/sec...
For objects, Windows NT permissions are ridiculously granular; e.g. GENERIC_WRITE can be mapped to a half-dozen separately settable type-specific flags, depending on the object type (file, named pipe, etc.). It’s too granular for even an administrator to make sense of, arguably, and the documentation is somewhere between bad and nonexistent. (The UI varies from decent, like the ACL editor you can access from e.g. Explorer, to “you can’t make this shit up”, like SDDL[1].)
For subjects, the situation is not good, like on every other conventional OS. You could deal with that by introducing a “user” for each app, as on Android. But I’m not aware of any attempts to do that (that would expose this mechanism in a user-visible way).
(Then there’s the UWP sandbox, which as far as I tell is build with complete disregard of the fundamental concepts above. I don’t think it’s worth taking seriously at this time.)
[1] https://learn.microsoft.com/en-us/windows/win32/secauthz/sec...
I have no idea if there’s a granular object permission that could give access to the MBR of a disk. I’ve thankfully never had to dig that deep into Windows internals.
I’ve had to work with SDDL before to setup granular permissions for WMI monitoring on a whole lot of computers and my god, did it make me love the Cloud and Linux. I can’t emphasize enough how unintuitive setting these permissions is creates systemic over privileging.
I’ve had to work with SDDL before to setup granular permissions for WMI monitoring on a whole lot of computers and my god, did it make me love the Cloud and Linux. I can’t emphasize enough how unintuitive setting these permissions is creates systemic over privileging.
Is this the Spacemonger you are talking about https://web.archive.org/web/20121126062443/http://www.sixty-...
It does not say anything like that in FAQ and i don't remember it being fast.
It does not say anything like that in FAQ and i don't remember it being fast.
Yes that one. Just use it and see. It's blazing fast.
It uses FindFirstFile etc https://github.com/seanofw/spacemonger1/blob/6a41c012534b170...
I thought you meant the $15 utility from Stardock, but if not then I'm fairly confident it's not reading the MFT.
https://github.com/seanofw/spacemonger1/blob/6a41c012534b170...
https://github.com/seanofw/spacemonger1/blob/6a41c012534b170...
It's still interesting that they got it to work as fast and precise as they did.
Just learned that its open source now https://github.com/seanofw/spacemonger1
Been using the portable version of 1.4 for decades after first coming across it in some PC magazine or something like that many years ago. Not terribly pretty, but it does what I need and it still works.
> What's the downside of just reading the MFT?
One possible reason is that it isn't a published part of the filesystem's external interface, and the format is not guaranteed to be static between versions or even point releases (though in reality, while the behaviours may be officially undefined that are unlikely to change significantly).
Also, it requires admin elevation to access. Anything running elevated is a potential security concern as it can access much else too.
> Why doesn't Microsoft do it in file explorer
Not sure, but it could be because that would be seen as an unfair advantage so to avoid anti-trust allegations they would have to publish the format and make stability guarantees for it, so others could use it as easily/safely. That, and the reasons above & below too.
> and why wouldn't every tool use it instead of walking through the file system?
Largely because walking the filesystem works for all filesystems, local and remote, so you cover everything with one tree walk implementation. Implementing a tree-walk over the MFT data where available is extra work to implement and support for one filesystem, and not many care enough, or are not aware of the potential speed benefit at all, for it to be a huge selling point such that all toolmakers feel compelled to bother.
One possible reason is that it isn't a published part of the filesystem's external interface, and the format is not guaranteed to be static between versions or even point releases (though in reality, while the behaviours may be officially undefined that are unlikely to change significantly).
Also, it requires admin elevation to access. Anything running elevated is a potential security concern as it can access much else too.
> Why doesn't Microsoft do it in file explorer
Not sure, but it could be because that would be seen as an unfair advantage so to avoid anti-trust allegations they would have to publish the format and make stability guarantees for it, so others could use it as easily/safely. That, and the reasons above & below too.
> and why wouldn't every tool use it instead of walking through the file system?
Largely because walking the filesystem works for all filesystems, local and remote, so you cover everything with one tree walk implementation. Implementing a tree-walk over the MFT data where available is extra work to implement and support for one filesystem, and not many care enough, or are not aware of the potential speed benefit at all, for it to be a huge selling point such that all toolmakers feel compelled to bother.
> One possible reason is that it isn't a published part of the filesystem's external interface, and the format is not guaranteed to be static between versions or even point releases (though in reality, while the behaviours may be officially undefined that are unlikely to change significantly).
I am not going to pull every document, but the MFT structure is documented and published. I am uncertain what you mean by "external interface".
"About 9,810 results (0.04 sec)"
https://scholar.google.com/scholar?hl=en&as_sdt=0%2C11&q=mft...
I am not going to pull every document, but the MFT structure is documented and published. I am uncertain what you mean by "external interface".
"About 9,810 results (0.04 sec)"
https://scholar.google.com/scholar?hl=en&as_sdt=0%2C11&q=mft...
Moreover it is documented by Microsoft itself: https://learn.microsoft.com/en-us/windows/win32/devnotes/mas...
Though all the sub-pages of that state things like “[This structure is valid only for version 3 of NTFS volumes; it may be altered in future versions.]” — while it is true that any API could see breaking changes in future, this suggests that you should expect them, so I'd not call it supported in the same sense of the main file/directory access APIs which I would not expect to see breaking changes in (additional properties & functionality yes, but not existing things changing behaviour).
A lot of people talking about the details, does not constitute official documentation though.
You can find a lot of articles talking about SQL Server's DBCC IND and DBCC PAGE, but that isn't official documentation – they are essentially internal functions and not supported and could change or go away entirely despite having been around for many versions, as they have in Azure). Similarly there articles talking about sys.dm_db_database_page_allocations which sort-of does the job of DBCC IND, but again this is not officially documented & supported.
> I am uncertain what you mean by "external interface".
I meant the published interface. Maybe "supported API" would have been a better phrase to use?
Though as pointed out below, there is at least some official documentation on the MFT structure.
You can find a lot of articles talking about SQL Server's DBCC IND and DBCC PAGE, but that isn't official documentation – they are essentially internal functions and not supported and could change or go away entirely despite having been around for many versions, as they have in Azure). Similarly there articles talking about sys.dm_db_database_page_allocations which sort-of does the job of DBCC IND, but again this is not officially documented & supported.
> I am uncertain what you mean by "external interface".
I meant the published interface. Maybe "supported API" would have been a better phrase to use?
Though as pointed out below, there is at least some official documentation on the MFT structure.
It's probably also racy to access the raw MFT while there are concurrent programs creating new files (or deleting files). That complication can be avoided by using the ordinary OS directory iteration primitives.
Yep but then the tradeoff of performance gains are completely discarded. The easiest solution is to take a snapshot with VSS, which is both fast and makes a quiesced copy of $MFT. From there, one could monitor FS changes if they wanted to have live updates.
With RAM sizes now, it's curious why any OS wouldn't just cache some or all of metadata for some local volumes on a block basis rather incur the greater resource usage of transforming disk into different structures, and then caching and track individual entries.
[deleted]
[deleted]
More MFT goodness: the file search tool Everything (https://www.voidtools.com/)
It's crazy how the Windows Search Indexer still doesn't use MFT.
It doesn't even bloody support network drives so there's no such reason.
It doesn't even bloody support network drives so there's no such reason.
I am building an advanced filemanager (FileNinja) for Windows with full integrated everything search & query. you have the option of saving bookmarks to virtual folders that consist of everything searches. Instant directory sizes, tags, custom file descriptions for ntfs. Anyone interested?
https://youtu.be/JREufgkf5pk?si=sP05UCOrskpX8OTq
I'm interested! Great marketing video by the way, a good example of using AI-powered voiceovers to level up the one-man-marketing polish capabilities.
Cool, but what is the biggest feature compared to Everything and Listary?
Do you have a git repo to follow?
You can check out the following sites https://github.com/sandeberger, http:\\thefile.ninja or my homepage at https://kodar.ninja. The project is not opensource.
haha I like the voiceover, the video is fun
So that's why Everything is so fast. Nice.
I want to like Everything but every time I start it up it takes 30 sec to 1 minute to update it's index
Try Everything 1.5a - an "alpha" version with many improvements, in development for years but inexplicably hidden away on their website. Never experienced any instability.
https://www.voidtools.com/forum/viewtopic.php?t=9787
https://www.voidtools.com/forum/viewtopic.php?t=9787
You can set it to run on startup or as a service so it updates the index in the background.
You should not be starting it when you want to search. You should open it when you log in, and leave it in the tray. It will do a full index on launch then subscribe to filesystem notifications to keep itself up to date for as long as it’s open.
Do that and it’s alarmingly fast and responsive except for the minute or two right after launch.
Do that and it’s alarmingly fast and responsive except for the minute or two right after launch.
Contrasting seemingly all the other responses to this, I use it the same way you do (only opening it when needed) and I'm fine with the delay: even at its slowest rebuilding the index and searching is faster than the in-built windows Search.
It should be starting at boot if you installed it as a service, so the indexing will be done then. After that opening and searching is instant.
essential tool
WizTree also understands things like OneDrive and Dropbox, and know that files "stored in the cloud" aren't taking up any disc space -- WinDirStat thinks my drive is 140% full.
What about hard links?
Wiztree and WinDirStat will both double count hard links. I have a 12TB hard drive holding "17TB" because of sparse files and hard links. Windows file manager properties agree with Wiztree and WinDirStat as far as space used. I think the file manager looks for free space and calculates that separately, while Wiztree and WinDirStat are just adding up used space.
WizTree is no longer free for commercial use.
I believe version 3.38 was the last version that is completely "free as in beer" with optional donations.
I believe version 3.38 was the last version that is completely "free as in beer" with optional donations.
> WizTree isn't open-source like WinDirStat but "free as in beer" with optional donations.
Which is enough for me to not use it because WinDirStat still only takes a minute. Cool software though.
Which is enough for me to not use it because WinDirStat still only takes a minute. Cool software though.
Wiztree takes like 3 seconds where WDS takes 30. In realy big analysis and cleanup scenarios with rescans, it's enough to let you do your job faster. In every day scenarios, it removes any hesitation to visualize a system. It's basically free and near instant.
Fact is, WDS community must be kind of abandoned, or else it would be doing the same trick. It's SO much faster that it becomes a genuine quality of life improvement. I need it, and don't mind using a non free tool until the OSS solution has the capability.
Fact is, WDS community must be kind of abandoned, or else it would be doing the same trick. It's SO much faster that it becomes a genuine quality of life improvement. I need it, and don't mind using a non free tool until the OSS solution has the capability.
Exactly this.
Didn't try AltWinDirStat, but did try FastWinDirStat.
The thing is, FastWinDirStat uses a licensed propietary component. No problem for me, but the author did have some back and forth with another user on GitHub.
Seems FastWinDirStat license don't match with using a closed source library, or something...
As for its actual functioning, it does as it says. Works much faster than WinDirStat
The thing is, FastWinDirStat uses a licensed propietary component. No problem for me, but the author did have some back and forth with another user on GitHub.
Seems FastWinDirStat license don't match with using a closed source library, or something...
As for its actual functioning, it does as it says. Works much faster than WinDirStat
Looks like a pretty clear violation of the WinDirStat license. They took WinDirStat which is GPL, linked it with some other proprieraty code and distributed the result.
(They could have been clear-ish (with caveats) by distributing only the source code and let the users do the compiling and linking, similarly to how you could download ZFS and build it into Linux. But you mustn't distribute the result further.)
(They could have been clear-ish (with caveats) by distributing only the source code and let the users do the compiling and linking, similarly to how you could download ZFS and build it into Linux. But you mustn't distribute the result further.)
I'm a big Filelight fan. It used to not work well on NTFS volumes, it would miss files flagged Archived, has that been solved?
I wish there was a duplicate file finder that used the MFT scan to pre-process the data instead of the FS tree walk
Do you use czkawka?
https://github.com/qarmin/czkawka
I don't think it uses the MFT, but it's the fastest and most flexible open source dup finder as of last year or so
https://github.com/qarmin/czkawka
I don't think it uses the MFT, but it's the fastest and most flexible open source dup finder as of last year or so
You’ve got me interested but I’m finding it quite annoying that WizTree doesn’t actually have pictures of the software UI on the website. At least not under any of the obvious links I’ve checked.
If you want to see screenshots of any piece of software, just search the name of the software on your favorite search engine and go to ‘images’.
(This might seem obvious, but it took me a long time to realize, hence why I’m passing the tip on.)
(This might seem obvious, but it took me a long time to realize, hence why I’m passing the tip on.)
Just try it, if you hoard data or clean up client systems. It's the best Windows file size checker in 21 years, since windirstat appeared.
It's not just an improvement, it's freaking astonishing.
I sincerely hope that the author open sources it one day, or that mtf-based solutions come to open source.
It's just life changing and will change when you want to do a scan. It removed any sense of hesitation or time waste.
It's not just an improvement, it's freaking astonishing.
I sincerely hope that the author open sources it one day, or that mtf-based solutions come to open source.
It's just life changing and will change when you want to do a scan. It removed any sense of hesitation or time waste.
having used both on my pc, can attest to the speed claims. wiztree has yet to demonstrate annoying freeware/donationware pop ups during daily use.
Seeing a description directly in the README for the folders in the repo and their contents makes me really happy. I wish more projects would do that.
Thanks for this; it is incredibly faster. Never heard of it before.
FileLocator Pro is a good search tool that also uses the MFT.
SpaceSniffer is a much easier to use tool.
SpaceSniffer's UI is less clunky, but Wiztree's scan is an order of magnitude faster. That kind of speed difference can affect when you're willing to use the tool.
I find myself much more willing to pop open wiz tree to get a quick view of my system or a particular storage folder.
I find myself much more willing to pop open wiz tree to get a quick view of my system or a particular storage folder.
They are getting very close to releasing windirstat-next [0] and already have some betas out, you can learn more about it in the subreddit. [1]
>WinDirStat fans,
>As a new pet project, I recently started some substantial revisions to WinDirStat in my GitHub branch and will work with current maintainer (Oliver Schneider) to eventually publish a new release hopefully in the next few months. The current changes on deck speed up performance drastically (seconds compared to minutes in some cases). It uses less memory compared to recent alternatives (WizTree), is faster as scanning network paths, and obviously isn't pushy about donating (although I certainly would not discourage folks from donating to their favorite opensource projects).
>Oliver recently opened up the GitHub Issues trackers, and I would love to hear suggestions or known bugs for the existing version:
>For the nerds interested in the changes I have queued up, you can visit the GitHub page
[0] https://github.com/windirstat/windirstat-next
[1] https://www.reddit.com/r/WinDirStat/
>WinDirStat fans,
>As a new pet project, I recently started some substantial revisions to WinDirStat in my GitHub branch and will work with current maintainer (Oliver Schneider) to eventually publish a new release hopefully in the next few months. The current changes on deck speed up performance drastically (seconds compared to minutes in some cases). It uses less memory compared to recent alternatives (WizTree), is faster as scanning network paths, and obviously isn't pushy about donating (although I certainly would not discourage folks from donating to their favorite opensource projects).
>Oliver recently opened up the GitHub Issues trackers, and I would love to hear suggestions or known bugs for the existing version:
>For the nerds interested in the changes I have queued up, you can visit the GitHub page
[0] https://github.com/windirstat/windirstat-next
[1] https://www.reddit.com/r/WinDirStat/
It really frustrates me when a project has a GUI and there are no screenshots of it on their github.
I know exactly what Windirstat looks like, I have no idea what this will/does look like and shouldn't have to install it to find out. I don't like Windirstats UI.
I know exactly what Windirstat looks like, I have no idea what this will/does look like and shouldn't have to install it to find out. I don't like Windirstats UI.
It is a directory view coupled to a tree map view of folders where the area is proportional to their size.
Navigating the directory view highlights the corresponding area on the tree map.
Files are color coded according to their type.
Navigating the directory view highlights the corresponding area on the tree map.
Files are color coded according to their type.
[deleted]
Wow, that's beautifully fast and opposed to WizTree it doesn't nag you to donate literally every 6.5 seconds (I timed it).
Hijacking this comment...
Does anyone know anything close to WinDirStat for google drive? I really need something like this.
Does anyone know anything close to WinDirStat for google drive? I really need something like this.
You could install the Google Drive client and then run WinDirStat against the virtual drive letter that it creates:
https://support.google.com/drive/answer/10838124
https://support.google.com/drive/answer/10838124
I have done this before, and it works, but the client downloads every single file locally in the background before it's able to do the comparison. I suspect this is because WinDirStat requests metadata about the file itself, and the sync client needs to download the files in order to serve that metadata and/or file contents information to the application.
You can use `rclone ncdu` on Google Drive
https://rclone.org/commands/rclone_ncdu/
It is great at answering the question - where is all my storage? Also has a handy interface for deleting stuff.
https://rclone.org/commands/rclone_ncdu/
It is great at answering the question - where is all my storage? Also has a handy interface for deleting stuff.
this has been my go-to. great tool all around
About a decade ago when I worked in IT I used Treesize Pro to scan SMB volumes and delete the MOVIES that employees had downloaded on their computers, which meant they were downloading MOVIES to our SAN because we used roaming profiles. I'll never get over that these people downloaded literal pirated movies to our servers. It was constant. I ended up putting file size limits on their directories, or a file extension block, I forget.
The free version can't scan network drives if the servers on a domain IIRC.
https://www.jam-software.com/treesize_free
edit: Oh I thought I read SMB drive not google drive for some reason. I'm not sure if this does that or not. It's my favorite GUI so worth checking out.
The free version can't scan network drives if the servers on a domain IIRC.
https://www.jam-software.com/treesize_free
edit: Oh I thought I read SMB drive not google drive for some reason. I'm not sure if this does that or not. It's my favorite GUI so worth checking out.
Mount Google Drive as a folder (rclone) and run WinDirStat?
I like "ncdu", a TUI equivalent for Unixy systems.
Although I learned the hard way that if you run it on a Mac home folder, and have iCloud's "optimize Mac storage" turned on, macOS will suddenly try to download literally everything in your iCloud storage to try to count the size of it, probably filling your disk. Oops.
Although I learned the hard way that if you run it on a Mac home folder, and have iCloud's "optimize Mac storage" turned on, macOS will suddenly try to download literally everything in your iCloud storage to try to count the size of it, probably filling your disk. Oops.
There's also "diskonaut", a TUI which displays the output like the treemap of WinDirStat. Bonus is that the display is incremental and updates as it scans everything, so you don't need to wait for the complete scan to see how everything looks like.
Written in Rust, and it's a `cargo install diskonaut` away if you have the rust toolchain installed.
Written in Rust, and it's a `cargo install diskonaut` away if you have the rust toolchain installed.
You can use `rclone ncdu` to work in a very similar way on any cloud storage that rclone supports
https://rclone.org/commands/rclone_ncdu/
I love ncdu and this is my homage to it!
https://rclone.org/commands/rclone_ncdu/
I love ncdu and this is my homage to it!
`ncdu` has the `-x` flag which means that "Only count files and directories on the same filesystem as the specified dir.".
Maybe that could help.
Maybe that could help.
Sadly it does not help, the files appear to be normal files in the same filesystem, macOS simply does magic behind the scenes.
You have to manually tell ncdu not to scan the ~/Library/Mobile Documents directory.
You have to manually tell ncdu not to scan the ~/Library/Mobile Documents directory.
DaisyDisk is an awesome macOS GUI version of this app
I switched to gdu a couple years back for reasons that I do not remember. I think it is faster or something.
If you want it RIIR there dua: https://github.com/Byron/dua-cli
I'm a big fan of godu: https://github.com/viktomas/godu
don't execute a (space taking) write operation when the user requests a read seems to be a reasonable rule for most things.
I find Space Sniffer http://www.uderzo.it/main_products/space_sniffer/ to be a much better visualisation. It updates in near real-time. If you have lots of copy/move ops going in background, you will see those dirs blinking rectangles growing/shirking in Space Sniffer.
To me, this is a clone of (now dead) Space Monger https://www.portablefreeware.com/index.php?id=150
To me, this is a clone of (now dead) Space Monger https://www.portablefreeware.com/index.php?id=150
Just learned that Space Monger 1.4 has been open sourced for a while now https://github.com/seanofw/spacemonger1
Accompanying blog post https://www.werkema.com/programming/the-spacemonger-1-x-post...
Accompanying blog post https://www.werkema.com/programming/the-spacemonger-1-x-post...
I used Space Sniffer for a very long time, but looked for alternatives because it crashes somewhat regularly and is generally a bit of a resource hog.
Since trying WizTree I don't think I'll ever go back, it's so much faster, hasn't crashed once on me, and the visualizations are completely adequate for me to be able to see where space is being used.
Since trying WizTree I don't think I'll ever go back, it's so much faster, hasn't crashed once on me, and the visualizations are completely adequate for me to be able to see where space is being used.
Also, SpaceSniffer is the only tool of all the popular ones that correctly accounts the size of files with NTFS compression, accurately accounts for directory junctions and links (by not counting them) and NTFS alternate data streams.
I really like the visualization, the functionality to watch live file access is so freaking cool. It's a bit slow though, no? After using something like WizTree, it's hard to go to something that needs minutes to finish scanning. I do wonder what kind of a performance impact it has.
It takes time for first indexing, yes. But once its done, any copy/move you do afterwards shows up in the UI at the same time.
Not open source (freeware), but much faster than WinDirStat for NTFS - WizTree https://diskanalyzer.com/. In short - it scans the actual filesystem metadata directly instead of enumerating files through the OS APIs, which makes it extremely fast.
Nice, so a bit like Everything but for disk space!
The same dev has an Everything alternative: WizFile [1]
[1]: https://antibody-software.com/wizfile/
[1]: https://antibody-software.com/wizfile/
I like how in the Year of our Lord 2024 this is not some basic functionality provided with the operating system.
It still shocks me that windows can’t sort folders by size
[deleted]
I'm just grateful we're not counting file shit in "blocks" anymore.
Up until very recently we couldn't even open zip files without a helper software. Unsure why microsoft doesn't add things like powertoys by default.
Zip file support has been in Windows since 98 (via the Plus pack) or Me (2000). https://en.m.wikipedia.org/wiki/ZIP_(file_format)
Think they just added 7z and rar support very recently though. But zip has been in there for ages... I use it all the time.
I meant rar. Sorry heh
Your first statement is already refuted by other commenters.
For the second, the reason is offical software support, internationalization and accessibility. Microsoft provides certain guarantees for its officially released software. They don't want to provide those for Powertoys.
For the second, the reason is offical software support, internationalization and accessibility. Microsoft provides certain guarantees for its officially released software. They don't want to provide those for Powertoys.
25 years ago is not "recently".
In technology years it's getting longer every year.
Windows could open zip files all the way back to the XP era....
Windows could open zip files all the way back to the XP era....
or reproduce something like ZipMagic (iirc, so defunct that its website is gone)
For Linux there's QDirStat: https://github.com/shundhammer/qdirstat
I really like the fsview plugin that comes with Konqueror. It can be run independently: `fsview /some/dir`.
IIRC WinDirStat is inspired by it.
I have used WinDirStat for years. It's not perfect, but it solves my use case very well.
My use case is just: my disk is full, I don't know why. This happens on one of my computers like once a year, so the fact that it's slow is fine. It usually helps me spot some folder set that is taking up a lot of space that I don't need on that PC, or something large that is duplicated.
My personal favorite example is wedding photos and videos. Turns out: those are huge, I am not going to delete them, but they don't need to be backed up on every computer I own.
My use case is just: my disk is full, I don't know why. This happens on one of my computers like once a year, so the fact that it's slow is fine. It usually helps me spot some folder set that is taking up a lot of space that I don't need on that PC, or something large that is duplicated.
My personal favorite example is wedding photos and videos. Turns out: those are huge, I am not going to delete them, but they don't need to be backed up on every computer I own.
Probably the one thing you should over do the backups for
Treemaps are generally a really cool way to visualise hierarchical data. See also the Observatory of Economic Complexity, which has treemaps of international trade and economic statistics.
https://oec.world/profile/country/gbr#yearly-trade
https://oec.world/profile/country/gbr#yearly-trade
fancy and interesting, thx for sharing.
There are quite many apps for visualizing disk folder structure for almost any OS. Any flavor you like: lists (diskwave, omnidisk), treemaps (GrandPerspective), pie charts (DiskSavvy), sunburst (baobab), icicle, etc.
The two winning visualization types are sunburst and treemaps. Both have their own cons and pros, but in our tests user sunburst performed slightly better for regular users. My personal bet is that no disk space analyzing tool's developer took it seriously or tried to actually advance the algorithms. Most of the apps I know use quite straightforward implementation and haven't been touched for years. Guess a little bit of filtering, grouping and changing coloring algorithms could significantly improve the treemap's perception, but someone has to do the job.
disclaimer: I'm the original designer of DaisyDisk.
The two winning visualization types are sunburst and treemaps. Both have their own cons and pros, but in our tests user sunburst performed slightly better for regular users. My personal bet is that no disk space analyzing tool's developer took it seriously or tried to actually advance the algorithms. Most of the apps I know use quite straightforward implementation and haven't been touched for years. Guess a little bit of filtering, grouping and changing coloring algorithms could significantly improve the treemap's perception, but someone has to do the job.
disclaimer: I'm the original designer of DaisyDisk.
Hey, your DaisyDisk UI gets stolen for Movies and stuff. Here, TV Series, Person of Interest used the interface to make a fancy scene!
https://x.com/brajeshwar/status/1733390176112800037
https://x.com/brajeshwar/status/1733390176112800037
I'm surprised they photo-manipulated that much. The makers of Unfriended: Dark Web just used the actual app :)
Thank you for DaisyDisk, it's my go-to when I need to free up some space and it's a delight to use!
For anyone who wants to find more information regarding treemaps, which are used in this application, this link https://www.cs.umd.edu/hcil/treemap-history/ has some history about treemaps and here https://www.cs.umd.edu/hcil/treemap/demos/ you can find a Java implementation where you can play
https://www.jam-software.com/treesize_free has been my companion for years, fast and clean
Same for me. Before using it I was using WinDirStat and before that SequoiaView, but I've now stuck to using only TreeSize Free for years now. It's good enough and the user interface is very clean.
Another oss alternative that a like is SpaceSniffer[1]. It doesn't have the color coding but it is much faster and updates live.
[1]: http://www.uderzo.it/main_products/space_sniffer/
[1]: http://www.uderzo.it/main_products/space_sniffer/
It still tickles me that many Linux distros come with a tool like this preinstalled (https://wiki.gnome.org/action/show/Apps/DiskUsageAnalyzer). Ah, the good ol' Windows days of having to hunt down tools that were actually useful. Far in the rearview mirror for me.
It tickles me that I still can't get a proper console program (like Conemu) on most Linux distros. Or anything resembling AutoHotKey or Notepad++.
Like it may be fine for you, but it's definitely not a flex for some of us. And I use Linux as my daily driver.
Like it may be fine for you, but it's definitely not a flex for some of us. And I use Linux as my daily driver.
Notepad++ will work with wine, and there's https://github.com/dail8859/NotepadNext
Google says there are options for autohotkey too, although some are works in progress:
https://github.com/Paris/IronAHK / https://github.com/poa00/ahk.keysharp/blob/master/README.md
https://github.com/autokey/autokey
https://linux.die.net/man/7/xautomation
http://sikulix.com/
https://github.com/phil294/AHK_X11
https://github.com/jordansissel/xdotool
https://github.com/baskerville/sxhkd
Google says there are options for autohotkey too, although some are works in progress:
https://github.com/Paris/IronAHK / https://github.com/poa00/ahk.keysharp/blob/master/README.md
https://github.com/autokey/autokey
https://linux.die.net/man/7/xautomation
http://sikulix.com/
https://github.com/phil294/AHK_X11
https://github.com/jordansissel/xdotool
https://github.com/baskerville/sxhkd
Or the old classic du -h / | sort -h
Note: There’s a gotcha when using it in multi-user environments (like a server).
Users with Administrator access do not have permission to enumerate directories / files inside other Admin users home directories. So any per-user files are not counted in this scenario.
Source: ‘The mysterious case of the Windows server with a full disk but WinDirStat shows it as only half-full’ :-)
Users with Administrator access do not have permission to enumerate directories / files inside other Admin users home directories. So any per-user files are not counted in this scenario.
Source: ‘The mysterious case of the Windows server with a full disk but WinDirStat shows it as only half-full’ :-)
Yes, that's why WinDirStat is a no go at all.
Use WizTree, or any other util what reads NTFS MFT directly.
Use WizTree, or any other util what reads NTFS MFT directly.
It can't get to all the files even for a single admin user, you need to do something like run it as system with psexec.
I use WinDirStat. At one point a number of years ago, I became curious about the parent program KDirStat. So I actually got that installed on some Ubuntu or something. It was interesting. Like a prototype for WinDirStat or something.
I recently observed, much to my delight, that WinDirStat runs flawlessly on Android under Winlator (https://www.winlator.org/).
Wow. A system utility that reads the Windows MFT is about the last thing (after system drivers maybe) that I would expect to work under an emulator on Android.
Many good alternative listed already.
But I have quite liked FileLight which is cross platform https://apps.kde.org/filelight/
Likely not as fast as WizTree though.
Edit 1: source at https://invent.kde.org/utilities/filelight with GPL 2.0 licence
But I have quite liked FileLight which is cross platform https://apps.kde.org/filelight/
Likely not as fast as WizTree though.
Edit 1: source at https://invent.kde.org/utilities/filelight with GPL 2.0 licence
For my MacOS friends, it might be a little outdated but I prefer GrandPerspective: https://grandperspectiv.sourceforge.net/
Haven't tried it in years. Have they ever fixed it not accurately accounting for directory junctions and links?
Edit: it seems they have! But I don't know about NTFS compression and alternate data streams.
Edit: it seems they have! But I don't know about NTFS compression and alternate data streams.
It's a longtime friend of mine who has kept my computer from wasting space.
But it's pretty outdated, and I think there are better programs out there now.
Such as?
WizTree, TreeSize or Fast WinDirStat (just fork).
But I still like to turn on WinDirStat and take a moment to relax.
But I still like to turn on WinDirStat and take a moment to relax.
based.
Can WinDirStat deal with hard links nowadays? Windows uses them extensively in the WinSxX folder and possibly elsewhere.
I tried many other so called "successors" to WinDirStat. I didn't like a single one of them, each of them had some significant problem.
This is a case of "If it ain't broken, don't fix it". This executable is from 2005 and still works flawlessly.
If you haven't tried it yet, I encourage you to!
This is a case of "If it ain't broken, don't fix it". This executable is from 2005 and still works flawlessly.
If you haven't tried it yet, I encourage you to!
I used to have similar views, but last year found out about WizTree and was instantly converted. Way faster, just as functional.
I like Diskitude by Evan Wallace the most for this kind of quick and easy drive content sizes inspection.
It does full scan, but is pretty fast and easy to use. And is super tiny.
https://madebyevan.com/diskitude/
https://madebyevan.com/diskitude/
I was about to share this, I just told someone about it the other night, been using it well over a decade now without any issue, the same exe I downloaded in 2011 or so.
Other Linux implementations:
- https://github.com/bootandy/dust (command line, extremely fast) - mate-disk-usage-analyzer (gui/gtk, a bit more intuitive and allows operating on files too).
- https://github.com/bootandy/dust (command line, extremely fast) - mate-disk-usage-analyzer (gui/gtk, a bit more intuitive and allows operating on files too).
I love WinDirStat and was pleasantly surprised to learn there is also a Linux variant that is really good. https://github.com/shundhammer/qdirstat
I'm a fan of WinDirStat. Yes it can be slow, but it runs on darn near every Win OS, hasn't changed in years, is a small executable, and the site always seems available. I used it many times to solve disk space issues on the job.
It's likely the product is now abandoned-ware or no longer developed. I'm sure this is the same version (or just a point different) that I had on my PC about a decade or so go. That said, that version worked OK.
That might just mean that the product is good and finished.
This!
I've used windirstat recently and it worked fine. Valuable to find the wasted space...
Still used in Windows IT environments as it's portable, even if it is a bit slow and there is no console version for generating tree-maps for remote viewing.
I feel like I’ve been using this tool for 20 years. So useful!
For macOS there is Disk Inventory X: https://www.derlien.com/
From the download page:
It is known to run on Windows 95 (IE5), Windows 98 SE, Windows ME, Windows NT4 (SP5), Windows 2000, Windows XP, Windows Vista, Windows 7, 8 and 8.1.
...it is also known to run on Windows 10 and 11, and likely any newer versions too. IMHO this is a great example of how software should be. One tiny binary that is very widely compatible, doesn't have any user-hostile "features", and remains stable and bug-free. It's a contrast from an industry that largely can't produce such achievements, is becoming increasingly hostile, and quite telling when there are already comments here complaining about its age.
It is known to run on Windows 95 (IE5), Windows 98 SE, Windows ME, Windows NT4 (SP5), Windows 2000, Windows XP, Windows Vista, Windows 7, 8 and 8.1.
...it is also known to run on Windows 10 and 11, and likely any newer versions too. IMHO this is a great example of how software should be. One tiny binary that is very widely compatible, doesn't have any user-hostile "features", and remains stable and bug-free. It's a contrast from an industry that largely can't produce such achievements, is becoming increasingly hostile, and quite telling when there are already comments here complaining about its age.
Disk Inventory X is the closest to as pretty as that I've found. I wish it was maintained
WizTree directly reads the MFT, and is a LOT faster, but it is not a free program.
TreeSize is far better IMHO
I've used this in the past, it's really great.
TreeSizeView also works perfectly.
been using GrandPerspective on Mac for the past 15 years and still v. happy with it
WizTree isn't open-source like WinDirStat but "free as in beer" with optional donations.
There's also a fork of WinDirStat patched to read the MFT but I don't know anyone who's tried it: https://github.com/ariccio/altWinDirStat