Windows AI Studio Preview(github.com)
github.com
Windows AI Studio Preview
https://github.com/microsoft/windows-ai-studio
68 comments
Windows AI Studio is in real a Linux AI Studio as it needs WSL to run. Little funny
But can you get this to run in Linux? From the article I can see it's a VSCode extension or something like that but the name kind of implies it's Windows only.
The link takes me to marketplace.visualstudio.com which makes it seem like a Visual Studio extension, not a VSCode extension? Not sure, it's as confusing as the live/hotmail/outlook/authentication parts of Microsoft tends to be.
Besides, it says the following:
> Windows AI Studio will run only on NVIDIA GPUs for the preview, so please make sure to check your device spec. WSL Ubuntu distro 18.4 or greater should be installed and is set to default prior to using Windows AI Studio.
So seems it'll only run in Linux, as it's a requirement to have Ubuntu WSL running beforehand.
Besides, it says the following:
> Windows AI Studio will run only on NVIDIA GPUs for the preview, so please make sure to check your device spec. WSL Ubuntu distro 18.4 or greater should be installed and is set to default prior to using Windows AI Studio.
So seems it'll only run in Linux, as it's a requirement to have Ubuntu WSL running beforehand.
> So seems it'll only run in Linux, as it's a requirement to have Ubuntu WSL running beforehand.
Yes, it'll only run in Linux, but only when that Linux is also running inside Windows.
Yes, it'll only run in Linux, but only when that Linux is also running inside Windows.
> https://github.com/microsoft/windows-ai-studio/blob/main/QA....
For now, but seems to be planned in the future.
For now, but seems to be planned in the future.
Yep, didn't read that. In the meantime, kudos to Canonical (and other parties) who helped Microsoft with WSL. Now they can also use Linux to produce Windows-only software! :D
that probably explain why nvidia only. Amd don't really have rocm for linux on latest consumer cards.
The Nvidia container toolkit works pretty much out of the box on WSL these days as well.
Funny that some cuda stuff works better through Windows virtualizing Linux than Windows natively, but if we're being honest even as a native Linux user, WSL probably provides a better user experience (vs having to use Nvidia drivers on Linux anyways)
Funny that some cuda stuff works better through Windows virtualizing Linux than Windows natively, but if we're being honest even as a native Linux user, WSL probably provides a better user experience (vs having to use Nvidia drivers on Linux anyways)
Having had the displeasure of trying to get CUDA running under WSL2, I can tell you it is most definitely not a better user experience :-P
Weird it was pretty much turn key here. Maybe that was the early days when it was in beta?
Is it a better user experience than trying to get CUDA natively? :P
My personal experience:
Fedora 38- just needed to compile gcc-12 (took very long),
Windows- Installer failed
This clearly shows that GNU/Linux must be superior
This clearly shows that GNU/Linux must be superior
Is the container toolkit even necessary anymore in WSL2? I did a fresh install of windows 10 LTSC a few weeks ago and installed WSL2 and docker with WSL2 integration and I was able to use my nvidia rtx card (checked through nvidia-smi) without any issues.
It registered the card, as in you can query the device id using nvidia-smi. But CUDA/cuDNN requires further work.
Interesting. I'll have to check to be sure, but I think maybe something is happening automagically if you have reasonably up to date nvidia drivers on the host OS, because I was able to run the EmotiVoice TTS docker (which requires nvidia gpu) from WSL2.
https://github.com/netease-youdao/EmotiVoice
https://github.com/netease-youdao/EmotiVoice
What? No latest windows drivers give full cuda support in wsl out of the box. I am running all kinds of models without any issues at all.
Hasn't been my experience. Windows keeps a few hundred mb of vram tied up for the gui, Linux only holds 14mb per GPU
Nvidia drivers work just fine on Linux... unless you are clueless at following instructions
Counterpoint:
https://forums.developer.nvidia.com/t/545-drivers-have-bad-f...
https://forums.developer.nvidia.com/t/wayland-native-wayland...
https://gitlab.freedesktop.org/xorg/xserver/-/issues/1317
and literally me being a Linux Nvidia 1080ti user for years and having plenty of issues
https://i.imgur.com/lwHDZoq.png
https://forums.developer.nvidia.com/t/545-drivers-have-bad-f...
https://forums.developer.nvidia.com/t/wayland-native-wayland...
https://gitlab.freedesktop.org/xorg/xserver/-/issues/1317
and literally me being a Linux Nvidia 1080ti user for years and having plenty of issues
https://i.imgur.com/lwHDZoq.png
I thought we were talking about CUDA here?
As someone who has had a home ML server since 2016 with two TitanX GPUs, and has worked on and maintained numerous servers since then I can definitely echo that maintaining Nvidia drivers, along with CUDA, CUDNN, etc has always been a hassle. It's certainly gotten better over time, but it's still quite fragile.
Automatic kernel update? https://forums.developer.nvidia.com/t/nvidia-smi-not-working...
What about upgrading to a new version of CUDA? https://stackoverflow.com/questions/43022843/nvidia-nvml-dri...
What about trying something like enabling forward compatibility for CUDA using an older driver? https://discuss.pytorch.org/t/torch-is-unable-to-detect-cuda... (This issue was actually just posted within the last day, so clearly people still have problems.)
If you haven't run into any issues, then I'd say you're very lucky. Just don't pretend lots of others haven't run into issues.
Automatic kernel update? https://forums.developer.nvidia.com/t/nvidia-smi-not-working...
What about upgrading to a new version of CUDA? https://stackoverflow.com/questions/43022843/nvidia-nvml-dri...
What about trying something like enabling forward compatibility for CUDA using an older driver? https://discuss.pytorch.org/t/torch-is-unable-to-detect-cuda... (This issue was actually just posted within the last day, so clearly people still have problems.)
If you haven't run into any issues, then I'd say you're very lucky. Just don't pretend lots of others haven't run into issues.
ROCM support for windows is also just a few months old.
Support is not nearly as ubiquitous as for CUDA, but it's there for some current and last Gen cards.
They did add support for the whole rdna2/3 series to windows. But only support for 7900xt/xtx were added to linux for some reason. And softwere in title seems actually ran on linux.
Besides that, I don't think I ever heard that they supports container at all.
Besides that, I don't think I ever heard that they supports container at all.
Adding link to VScode extension:
https://marketplace.visualstudio.com/items?itemName=ms-windo...
https://marketplace.visualstudio.com/items?itemName=ms-windo...
Anyone knows when Microsoft will release a local OCR model officially? I haven't seen anyone talking about this, but the one they're shipping with Snipping Tool (OneOcr) is top-tier and beats everything out there like Tesseract, easyOCR etc.
The model is technically in everyone's Windows installs, but we don't have the C++ projected WinRT headers to use the Microsoft.Windows.Vision library.
The model is technically in everyone's Windows installs, but we don't have the C++ projected WinRT headers to use the Microsoft.Windows.Vision library.
Seems they've released a bunch already?
Image-to-text models, filtered by "Microsoft": https://huggingface.co/models?pipeline_tag=image-to-text&sor...
Image-to-text models, filtered by "Microsoft": https://huggingface.co/models?pipeline_tag=image-to-text&sor...
Wow, very nice! Thank you, hadn't come across them in my search.
It looks like it doesn't recognize more than one line at once, but combined with another model or algorithm to detect text bounding boxes it'd be handy.
It looks like it doesn't recognize more than one line at once, but combined with another model or algorithm to detect text bounding boxes it'd be handy.
Not sure if it's what's used in the Azure Document Intelligence service, but my experience with it is pretty good.
The one somewhat unique offering in Azure is the Document Layout model which gives you back the OCR with titles, headers, paragraphs, and tables all labeled.
This is really, really good for RAG since it's often useful to stuff the nearest header into the chunk of text when generating an embedding (much, much better results this way).
The one somewhat unique offering in Azure is the Document Layout model which gives you back the OCR with titles, headers, paragraphs, and tables all labeled.
This is really, really good for RAG since it's often useful to stuff the nearest header into the chunk of text when generating an embedding (much, much better results this way).
Apple has good uniform hardware to enable this, but they are a product company and an “AI studio” would not fit their usual definition of a product.
I do hope they are considering going in that direction though.
I do hope they are considering going in that direction though.
Apple has it: https://github.com/ml-explore/mlx
Two tangential thoughts about this.
1. Why would they miss the opportunity to go all in and make use of the Neural engines for this? Or do they already, and I just don’t know how to interpret it?
2. At what point is Apple going to think - hmm, we have a kick ass processor on our hands. What if we run our server fleet - the ones that serve iCloud, Apple Store, all Apple services, databases etc on M* processors? It sure would help even better the economies of scale for Apple to go to TSMC and say - here is our new CPU design for servers, iPhone, iPad, watch and whatever VR thing. Why no love for the server side that must be orders of magnitude power hungrier today?
1. Why would they miss the opportunity to go all in and make use of the Neural engines for this? Or do they already, and I just don’t know how to interpret it?
2. At what point is Apple going to think - hmm, we have a kick ass processor on our hands. What if we run our server fleet - the ones that serve iCloud, Apple Store, all Apple services, databases etc on M* processors? It sure would help even better the economies of scale for Apple to go to TSMC and say - here is our new CPU design for servers, iPhone, iPad, watch and whatever VR thing. Why no love for the server side that must be orders of magnitude power hungrier today?
That's the weird thing. They seem to have the best hardware for the price, for individuals to develop and use local LLMs, but so far they have been pretty quiet on all this.
Highly depends on the price. Currently 1600€ gets you a laptop with 8gb shared memory, which surely is a nice device for other use cases but for developing local LLMs that seems awfully limiting.
With upgraded ram (for just 230€ for each 8GB) and storage (just over 1000€ for 2tb; a samsung 990 pro is like 170€) that might be another story, but "check your specs before downloading this app" seems very un-appley. Also, no cuda support etc. Maybe if they make it exclusive to the mac studio?
With upgraded ram (for just 230€ for each 8GB) and storage (just over 1000€ for 2tb; a samsung 990 pro is like 170€) that might be another story, but "check your specs before downloading this app" seems very un-appley. Also, no cuda support etc. Maybe if they make it exclusive to the mac studio?
Cuda would be pointless for Apple, because cuda is for Nvidia only,
no?
As for the price - when you get to 32/64/96GB levels of RAM it’s the cheapest setup on the market that can get you this much VRAM. At least that’s what it was half a year ago when I checked the last time
As for the price - when you get to 32/64/96GB levels of RAM it’s the cheapest setup on the market that can get you this much VRAM. At least that’s what it was half a year ago when I checked the last time
> Cuda would be pointless for Apple, because cuda is for Nvidia only, no?
That is the point. Cuda is widely used in particular for training as opposed to tuning and is just flat out not available. So you buy your nice $6000 machine and it just does not work for its intended purpose.
> it’s the cheapest setup on the market that can get you this much VRAM.
shared VRAM is not everything.
That is the point. Cuda is widely used in particular for training as opposed to tuning and is just flat out not available. So you buy your nice $6000 machine and it just does not work for its intended purpose.
> it’s the cheapest setup on the market that can get you this much VRAM.
shared VRAM is not everything.
I think it's clear that GPU manufacturers are scamming their customers more than Apple does with their ridiculous prices. There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM.
Nvidia intentionally nerfs the amount of VRAM in their consumer cards so you need to buy their ridiculously overpriced enterprise cards. I think it's fair to say Apple's lineup is the cheapest way to get >64GB of VRAM-ish memory and that's definitely not because Apple prices their products so fairly.
Nvidia intentionally nerfs the amount of VRAM in their consumer cards so you need to buy their ridiculously overpriced enterprise cards. I think it's fair to say Apple's lineup is the cheapest way to get >64GB of VRAM-ish memory and that's definitely not because Apple prices their products so fairly.
> There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM.
Those are not equivalent in speed. macs RAM are much slower than these GPUs.
Those are not equivalent in speed. macs RAM are much slower than these GPUs.
There is no Apple-to-apples comparison between a Macbook and a GPU. However, the problem with running most models is the lack of simultaneous RAM.
You can swap memory back and forth between RAM and VRAM (with a huge performance penalty) of course, but that's not exactly usable or comparable to what Apple's VRAM sharing setup allows.
Nvidia doesn't sell a nice-but-not-amazing GPU equivalent to Apple's processing power and memory bandwidth that's also capable of operating on >80GB of VRAM at once. Apple's SoC is kind of an oddball in that regard.
I suppose you could take a regular old iGPU (for AMD, Intel, probably also Qualcom/Mediatek) and use its shared memory capabilities as a comparison. However, iGPUs are terrible at machine learning tasks, they don't come close to what Apple can do with their dedicated accelerators.
The best middle ground may be the laptop GPUs with both dedicated RAM and shared RAM, but those will start swapping memory back and forth like crazy running large ML workloads so they're not really comparable.
If you want to run a model that operates on a huge amount of memory at once, I don't think there is a desktop option that can do what Apple does without going for the massive overkill GPUs that will crush the Macbook in terms of performance (at great cost).
Perhaps you know a GPU or iGPU that's capable of running 80GB VRAM workloads at comparable speeds? Because I don't.
You can swap memory back and forth between RAM and VRAM (with a huge performance penalty) of course, but that's not exactly usable or comparable to what Apple's VRAM sharing setup allows.
Nvidia doesn't sell a nice-but-not-amazing GPU equivalent to Apple's processing power and memory bandwidth that's also capable of operating on >80GB of VRAM at once. Apple's SoC is kind of an oddball in that regard.
I suppose you could take a regular old iGPU (for AMD, Intel, probably also Qualcom/Mediatek) and use its shared memory capabilities as a comparison. However, iGPUs are terrible at machine learning tasks, they don't come close to what Apple can do with their dedicated accelerators.
The best middle ground may be the laptop GPUs with both dedicated RAM and shared RAM, but those will start swapping memory back and forth like crazy running large ML workloads so they're not really comparable.
If you want to run a model that operates on a huge amount of memory at once, I don't think there is a desktop option that can do what Apple does without going for the massive overkill GPUs that will crush the Macbook in terms of performance (at great cost).
Perhaps you know a GPU or iGPU that's capable of running 80GB VRAM workloads at comparable speeds? Because I don't.
That sounds like an overly specific target audience:
- you need more than 16 or 32 GB of ram.
- but less than 100.
- speed is important.
- but losing a factor 4 compared to a GPU is fine.
- money is very limited.
- but paying 6000 for a mac studio is cheap.
- this is very important professional work.
- but I don't need servers, ECC ram, raid, another OS...
Don't get me wrong, I also think Nvidia is heavily price gouging, but there are definitely trade offs and it is not clear at all why 80GB is your magical number and not, say, 108 or 28.
- you need more than 16 or 32 GB of ram.
- but less than 100.
- speed is important.
- but losing a factor 4 compared to a GPU is fine.
- money is very limited.
- but paying 6000 for a mac studio is cheap.
- this is very important professional work.
- but I don't need servers, ECC ram, raid, another OS...
Don't get me wrong, I also think Nvidia is heavily price gouging, but there are definitely trade offs and it is not clear at all why 80GB is your magical number and not, say, 108 or 28.
Shared vram is everything of you're running local inference.
My 3 x 3090s can't fit the 180b falcon even at 2 bits into vram. But people with a 128gb studio can, and it's cheaper
It's not weird, is typical Apple playbook.
Year 0 of cool thing - Nothing and silence
Year 1-3 of cool thing - Maybe, if you're lucky, a mention on some hardware thing related to it
Year 3-5 of cool thing - Hardware or software launches that uses thing, no mentions of this besides the earlier one if any.
Year 5-6 of cool thing - Next part of hardware or software launches that uses previous launch, no mentions of this besides the earlier one if any.
Obviously, the time-frames differ, but that's generally how they do things.
Year 0 of cool thing - Nothing and silence
Year 1-3 of cool thing - Maybe, if you're lucky, a mention on some hardware thing related to it
Year 3-5 of cool thing - Hardware or software launches that uses thing, no mentions of this besides the earlier one if any.
Year 5-6 of cool thing - Next part of hardware or software launches that uses previous launch, no mentions of this besides the earlier one if any.
Obviously, the time-frames differ, but that's generally how they do things.
https://github.com/ml-explore/mlx-examples/ is Apple engineers working to get ML models going on Apple Silicon's Metal hardware.
They do have such tools for developers like CreateML to train your own models, pretty sure they will have one for LLM.
Nobody is going to train or even fine-tune large models on Apple hardware. It is too slow for that purpose.
> will run only on NVIDIA GPUs for the preview
I wonder why Microsoft helps nVidia, instead of using their own technology?
Here’s an example: https://github.com/Const-me/Cgml
I wonder why Microsoft helps nVidia, instead of using their own technology?
Here’s an example: https://github.com/Const-me/Cgml
Their own technology is DirectML, not raw DirectX 11.
https://blogs.windows.com/windowsdeveloper/2023/11/15/elevat...
https://blogs.windows.com/windowsdeveloper/2023/12/14/direct...
https://blogs.windows.com/windowsdeveloper/2023/11/15/elevat...
https://blogs.windows.com/windowsdeveloper/2023/12/14/direct...
Companies of the size of Microsoft usually function more like a country with various companies within, rather than one company with a unified view and unified goals.
So probably, the group wasn't even aware of that technology because it's far away by either professional connection, or by personal/relationship connections, or they knew about it but had another goal than "maximize use of own stuff" and made the call that the tradeoffs wasn't worth it.
So probably, the group wasn't even aware of that technology because it's far away by either professional connection, or by personal/relationship connections, or they knew about it but had another goal than "maximize use of own stuff" and made the call that the tradeoffs wasn't worth it.
I don’t work for Microsoft, and I don’t care whether people are using Microsoft’s technologies. As a consumer, I’m not happy with nVidia’s monopoly in the field. I find it disappointing that Microsoft chose to play along, instead of disrupting the landscape.
I think the only reason for that monopoly is the mental inertia of everyone involved. The complexity of these AI models is contained within the data in the models (gigabytes of numbers in these tensors), the GPU-running code is rather simple, most of that code is basic BLAS stuff. Unlike traditional GPGPU applications (FEM, numerical simulations, fluid dynamics), the compute kernels used in AI are easily portable across GPU APIs.
Maybe when I have some free time, I should port my library to Linux + Vulkan, just to prove the point.
I think the only reason for that monopoly is the mental inertia of everyone involved. The complexity of these AI models is contained within the data in the models (gigabytes of numbers in these tensors), the GPU-running code is rather simple, most of that code is basic BLAS stuff. Unlike traditional GPGPU applications (FEM, numerical simulations, fluid dynamics), the compute kernels used in AI are easily portable across GPU APIs.
Maybe when I have some free time, I should port my library to Linux + Vulkan, just to prove the point.
Have you considered ILGPU? https://ilgpu.net/
No, I have not considered ILGPU, it’s the first time I see that thing. Very interesting, but I’m not sure I would have used it for that project.
In my experience, using any non-native GPU API is asking for troubles. On Windows, the native ones are D3D 11 and 12, on Linux and Android it’s often Vulkan, and on MacOS it’s either Metal or that newer thing they have built specifically for AI. It seems ILGPU only has backends for CUDA, OpenCL, and CPU SIMD.
I have general suspicion towards custom compilers. Writing a good compiler is hard, for GPUs even harder due to the weird execution model and insufficient documentation from GPU vendors. HLSL compiler is supported by Microsoft, and compute shaders are used by many millions of gamers every day. Similarly, CUDA is supported by nVidia, usually pretty stable, my only issue with CUDA is vendor lock-in. I have an impression people are often unhappy with the quality and hardware compatibility of less popular GPU APIs like ROCm and OpenCL.
BTW, I asked a friend to test my program on their low-and laptop. The laptop has some Intel core i3 with integrated GPU. The performance wasn’t great at about 1 token/second (single-channel memory), but at least my code worked. I’m not sure it would have worked on that computer if the backend was based on OpenCL.
In my experience, using any non-native GPU API is asking for troubles. On Windows, the native ones are D3D 11 and 12, on Linux and Android it’s often Vulkan, and on MacOS it’s either Metal or that newer thing they have built specifically for AI. It seems ILGPU only has backends for CUDA, OpenCL, and CPU SIMD.
I have general suspicion towards custom compilers. Writing a good compiler is hard, for GPUs even harder due to the weird execution model and insufficient documentation from GPU vendors. HLSL compiler is supported by Microsoft, and compute shaders are used by many millions of gamers every day. Similarly, CUDA is supported by nVidia, usually pretty stable, my only issue with CUDA is vendor lock-in. I have an impression people are often unhappy with the quality and hardware compatibility of less popular GPU APIs like ROCm and OpenCL.
BTW, I asked a friend to test my program on their low-and laptop. The laptop has some Intel core i3 with integrated GPU. The performance wasn’t great at about 1 token/second (single-channel memory), but at least my code worked. I’m not sure it would have worked on that computer if the backend was based on OpenCL.
Microsoft also has a nice AI/ML framework called ML.NET.
Curious it uses Ubuntu 18.04 with 20.04 and 22.04 already released and 24.04 just around the corner.
RAG Project
Coming soon!
=(
Coming soon!
=(
Why is this on GitHub?
Why would it not be?
Because there's no real source included therein.
True.
But it's easy to publish something on github, and you get an bug tracker. So .. I guess they do it because it's easy to do so?
But it's easy to publish something on github, and you get an bug tracker. So .. I guess they do it because it's easy to do so?
When even Microsoft uses their own platform in a non-intended way, you know it's an official Microsoft project.
Installed this, and it didn't let me do anything. Despite the fact the readme says stuff runs locally, first it asked me to link to my github account, and then all models required me to ask someone (I think meta?) for permission on github or use a huggingface token or whatever.
So I uninstalled it and now my wsl prompt starts with (base) and I don't know how to disable it and all my python scripts are broken because they can't find all the libraries I've installed from pip throughout the years.
0/10 would not recommend.
So I uninstalled it and now my wsl prompt starts with (base) and I don't know how to disable it and all my python scripts are broken because they can't find all the libraries I've installed from pip throughout the years.
0/10 would not recommend.
(base) sounds like Anaconda. Try typing conda deactivate?