Egui commit: Implement accessibility APIs via AccessKit(github.com)
github.com
Egui commit: Implement accessibility APIs via AccessKit
https://github.com/emilk/egui/commit/e1f348e4b24c2fa83d25c6a7ddfd9b38b85de161
38 comments
Is Xilem going to be an independent UI toolkit, or is it going to be a "modification" of Druid?
This isn't published to crates.io yet, but I'm excited to share this work I've been doing. This makes egui, as far as I know, the first pure-Rust GUI toolkit to implement platform-native accessibility APIs, and possibly the first immediate-mode GUI to do so as well (albeit with an intermediate retained tree).
Slint has had accessibility support before. (But it was arguably done by calling into a C++ library, so technically not pure rust.)
https://github.com/slint-ui/slint/pull/1294
https://github.com/slint-ui/slint/pull/1294
Ah yes, I didn't mean to dismiss the work your team has already done. Just got over-excited, I guess. Anyway, thanks for the work you've already done on accessibility. It should now be straightforward to connect Slint's existing accessibility support to AccessKit, if you wish.
And now a quick demo video, on Windows using Narrator: https://mwcampbell.us/video/egui-accesskit-win-demo-2022-12-...
Impressive progress, thank you so much for your work on this.
For the live area, do desktop APIs support something like the HTML 'role' attribute? https://developer.mozilla.org/en-US/docs/Web/Accessibility/A...
As an app developer it would be helpful to be able to denote the live region in your example a 'status' (the user can read it but it's not read automatically on change).
And there may be other times when the same area should be an 'alert' (changes trigger 'assertive' messages to inform the user that their change results in an error or issue).
For the live area, do desktop APIs support something like the HTML 'role' attribute? https://developer.mozilla.org/en-US/docs/Web/Accessibility/A...
As an app developer it would be helpful to be able to denote the live region in your example a 'status' (the user can read it but it's not read automatically on change).
And there may be other times when the same area should be an 'alert' (changes trigger 'assertive' messages to inform the user that their change results in an error or issue).
Egui keeps getting better and better! For those unfamiliar you can run the web demo on your phone or desktop: https://egui.rs
I’ve personally built relatively complex UIs with egui. Both a Chrome extension and a VSCode extension for https://membrane.io and couldn’t be happier with it.
I’ve personally built relatively complex UIs with egui. Both a Chrome extension and a VSCode extension for https://membrane.io and couldn’t be happier with it.
egui’s web implementation is purely canvas with no DOM (apart from how it handles text input), so for now it’s completely invisible to screen readers. But even once AccessKit supports the web (and even pretending it wouldn’t be a performance sink that undermines half of the purpose of the canvas rendering approach in web apps), it will still be heavily inferior to something done with DOM for almost all types of content.
Some examples: text rendering is awful on some common browser configurations (mine included—Firefox on Linux/Wayland, rendered at 2× scaling and downsampled to 1.5×; the text is extraordinarily fuzzy, and it’s rendering in its own font because I suppose it’s bundling a complete font, shaper and renderer—since I force my own font selections so only the generic font-families work, and I also block all web fonts at fetch time); the font rendering technique means no fallback which means things like emoji would only render if they provided a font that supported that (at present they don’t, and it’s just a nasty technique all round, you want to be able to use local fonts); links are fake, so various interactions can never work correctly (e.g. Ctrl+click and middle-click should mostly open a background tab, and right-clicking should give you the normal context menu); scrolling can’t be done properly and will thus inevitably be extremely obnoxious on most devices; mobile-style pinch zoom cannot be supported.
⸻And I’m just mentioning some of the unfixable problems with this approach; there are more things that are wonky or painful apart from that: things that could be fixed, but where it’s not surprising that they’re broken because that’s a common problem when you throw away almost all of the stuff the platform gives you and try to implement it all from scratch.
I very strongly recommend against using this general approach at all on the web. The web platform just doesn’t provide the primitives required to make it anything other than miserable.
(I’m not speaking against egui for platforms other than web, only for web. The fact of the matter is that functionally you’re fine building a native app with native tech, a native app with web tech, or a web app with web tech, but that building a web app with native tech is a terrible idea that will always be inferior, in important, unfixable ways.)
Some examples: text rendering is awful on some common browser configurations (mine included—Firefox on Linux/Wayland, rendered at 2× scaling and downsampled to 1.5×; the text is extraordinarily fuzzy, and it’s rendering in its own font because I suppose it’s bundling a complete font, shaper and renderer—since I force my own font selections so only the generic font-families work, and I also block all web fonts at fetch time); the font rendering technique means no fallback which means things like emoji would only render if they provided a font that supported that (at present they don’t, and it’s just a nasty technique all round, you want to be able to use local fonts); links are fake, so various interactions can never work correctly (e.g. Ctrl+click and middle-click should mostly open a background tab, and right-clicking should give you the normal context menu); scrolling can’t be done properly and will thus inevitably be extremely obnoxious on most devices; mobile-style pinch zoom cannot be supported.
⸻And I’m just mentioning some of the unfixable problems with this approach; there are more things that are wonky or painful apart from that: things that could be fixed, but where it’s not surprising that they’re broken because that’s a common problem when you throw away almost all of the stuff the platform gives you and try to implement it all from scratch.
I very strongly recommend against using this general approach at all on the web. The web platform just doesn’t provide the primitives required to make it anything other than miserable.
(I’m not speaking against egui for platforms other than web, only for web. The fact of the matter is that functionally you’re fine building a native app with native tech, a native app with web tech, or a web app with web tech, but that building a web app with native tech is a terrible idea that will always be inferior, in important, unfixable ways.)
On the other hand you have Figma and Google Docs as examples of canvas-based applications that work well enough. I admit there are additional challenges to using egui on a Web context, but when you're building a highly interactive application (!), the benefits can outweigh the drawbacks.
When building DOM-based applications that require frequent updates I've always ended up working around browser limitations in awkward ways, e.g. minimizing re-rendering by using redux's relesect.
I decided to use egui because I get to use Rust for the entire stack, but also because it lets me ship a native app without an entire (slow to init) browser.
When building DOM-based applications that require frequent updates I've always ended up working around browser limitations in awkward ways, e.g. minimizing re-rendering by using redux's relesect.
I decided to use egui because I get to use Rust for the entire stack, but also because it lets me ship a native app without an entire (slow to init) browser.
I can’t speak about Figma, never having used it.
Google Docs I have just now tried. It isn’t like egui.
egui throws away everything the browser gives it, other than a text input element and a canvas element, and does all the rendering itself and dumps it to the canvas as a bitmap.
Google Docs is full DOM for everything other than rendering the document, and it still uses the browser’s text layout and painting facilities. (I say this because I have my browser set to only use my chosen fonts—so for English text, there are only the default font, serif, sans-serif and monospace—and I observe that it renders and shapes everything in my chosen default font, and am mildly disappointed it doesn’t even do a sane fallback, e.g. “Courier New” to monospace, and “Sans Serif” to sans-serif.) Almost all of my criticisms of the egui approach are avoided. Scrolling works fine, because it’s using the DOM for that, so it’s just normal scrolling. Links work fine, because it uses normal links (these are in tooltip-like things when you move the cursor to the link in the normal text flow).
(I retract my remark about mobile-style pinch zoom not being possible in canvas-only, because Google Docs does support it. I can now imagine how it’s done and I’m not sure why I didn’t see it before.)
But I would also remark about Google Docs since I’ve now tried it in edit mode for the first time since canvas rendering: ugh it’s slow. Low throughput, high latency, high jitter. This on a very fast (Ryzen 5800HS), near-idling laptop with a 165 Hz screen and 2× devicePixelRatio. Just try holding a letter down (I have a 40ms repeat rate), and compare the result to the same actions on contenteditable: the difference is stark. It also gets keyboard caret navigation behaviour wrong for my platform (Linux, Firefox), and its behaviour doesn’t match my vague recollection of Windows’ either. (I mean things like where Ctrl+Left and Ctrl+Right stop—whether at the end of one word or the start of the next, and whether to consider punctuation as part of another word or a word on its own; that kind of thing.)
Google Docs I have just now tried. It isn’t like egui.
egui throws away everything the browser gives it, other than a text input element and a canvas element, and does all the rendering itself and dumps it to the canvas as a bitmap.
Google Docs is full DOM for everything other than rendering the document, and it still uses the browser’s text layout and painting facilities. (I say this because I have my browser set to only use my chosen fonts—so for English text, there are only the default font, serif, sans-serif and monospace—and I observe that it renders and shapes everything in my chosen default font, and am mildly disappointed it doesn’t even do a sane fallback, e.g. “Courier New” to monospace, and “Sans Serif” to sans-serif.) Almost all of my criticisms of the egui approach are avoided. Scrolling works fine, because it’s using the DOM for that, so it’s just normal scrolling. Links work fine, because it uses normal links (these are in tooltip-like things when you move the cursor to the link in the normal text flow).
(I retract my remark about mobile-style pinch zoom not being possible in canvas-only, because Google Docs does support it. I can now imagine how it’s done and I’m not sure why I didn’t see it before.)
But I would also remark about Google Docs since I’ve now tried it in edit mode for the first time since canvas rendering: ugh it’s slow. Low throughput, high latency, high jitter. This on a very fast (Ryzen 5800HS), near-idling laptop with a 165 Hz screen and 2× devicePixelRatio. Just try holding a letter down (I have a 40ms repeat rate), and compare the result to the same actions on contenteditable: the difference is stark. It also gets keyboard caret navigation behaviour wrong for my platform (Linux, Firefox), and its behaviour doesn’t match my vague recollection of Windows’ either. (I mean things like where Ctrl+Left and Ctrl+Right stop—whether at the end of one word or the start of the next, and whether to consider punctuation as part of another word or a word on its own; that kind of thing.)
> On the other hand you have Figma and Google Docs as examples of canvas-based applications that work well enough.
At what cost?
Google Docs is run a by a trillion-dollar corporation, they have the resources to rebuild it in canvas.
Figma isn't based on canvas, it's based on WebGL. And here's what Figma had to say about it: "Pulling this off was really hard; we’ve basically ended up building a browser inside a browser... We’ve implemented our own rendering engine" [1]
Good luck getting text to render properly on your own.
Just because there are two great examples made at great cost doesn't mean it's scalable to all other apps.
[1] https://www.figma.com/blog/building-a-professional-design-to...
At what cost?
Google Docs is run a by a trillion-dollar corporation, they have the resources to rebuild it in canvas.
Figma isn't based on canvas, it's based on WebGL. And here's what Figma had to say about it: "Pulling this off was really hard; we’ve basically ended up building a browser inside a browser... We’ve implemented our own rendering engine" [1]
Good luck getting text to render properly on your own.
Just because there are two great examples made at great cost doesn't mean it's scalable to all other apps.
[1] https://www.figma.com/blog/building-a-professional-design-to...
> Figma isn't based on canvas, it's based on WebGL.
Aren't they the same thing?
And egui also talks about challenges with rendering on browsers using WebGL and WASM: https://github.com/emilk/egui/tree/master/crates/eframe#prob...
Aren't they the same thing?
And egui also talks about challenges with rendering on browsers using WebGL and WASM: https://github.com/emilk/egui/tree/master/crates/eframe#prob...
> Aren't they the same thing?
Ah. In my mind Canvas refers to the 2D API, https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API
Which is different and separate from WebGL, https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API
(There will be a third different and separate API, WebGPU: https://www.w3.org/TR/webgpu/)
> And egui also talks about challenges with rendering on browsers using WebGL and WASM
Ah. It's good they are aware of this.
Ah. In my mind Canvas refers to the 2D API, https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API
Which is different and separate from WebGL, https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API
(There will be a third different and separate API, WebGPU: https://www.w3.org/TR/webgpu/)
> And egui also talks about challenges with rendering on browsers using WebGL and WASM
Ah. It's good they are aware of this.
> And I’m just mentioning some of the unfixable problems with this approach
I agree that using a canvas instead of leveraging existing accessibility supplied by HTML/CSS/JS and the DOM is an accessibility nightmare.
However, I disagree that canvas accessibility issues are "unfixable". Difficult? Yes. But not unfixable. In my view, the <canvas> element works best in partnership with its wider environment (HTML/CSS/JS and the DOM), not as a replacement for it. With that in mind, we can start to tackle the accessibility issues you raise - fonts, links, interactions, etc.[1][2]
I have an ambition to one day become intelligent enough to understand/code in Rust, and I'm really glad to see that people are thinking about accessibility as a fundamental part of UIs being developed in Rust.
[1] - Which is what my JS 2d canvas library tries to do: https://scrawl-v8.rikweb.org.uk/
[2] - My thoughts on <canvas> accessibility, and how I try to fix them using my library: https://scrawl-v8.rikweb.org.uk/learn/eleventh-lesson/
I agree that using a canvas instead of leveraging existing accessibility supplied by HTML/CSS/JS and the DOM is an accessibility nightmare.
However, I disagree that canvas accessibility issues are "unfixable". Difficult? Yes. But not unfixable. In my view, the <canvas> element works best in partnership with its wider environment (HTML/CSS/JS and the DOM), not as a replacement for it. With that in mind, we can start to tackle the accessibility issues you raise - fonts, links, interactions, etc.[1][2]
I have an ambition to one day become intelligent enough to understand/code in Rust, and I'm really glad to see that people are thinking about accessibility as a fundamental part of UIs being developed in Rust.
[1] - Which is what my JS 2d canvas library tries to do: https://scrawl-v8.rikweb.org.uk/
[2] - My thoughts on <canvas> accessibility, and how I try to fix them using my library: https://scrawl-v8.rikweb.org.uk/learn/eleventh-lesson/
I am specifically speaking of egui’s strongly canvas-only approach. It doesn’t want partnership with its wider environment, it wants to replace it. In that approach, these issues are fundamentally unfixable.
I’ll focus on the two most obviously significant issues:
① Scrolling. The only complete solution is to draw within scrolling elements. To treat scrolling content distinctly, and allow the browser to act as the compositor. This would require rebuilding egui from the ground up; its current philosophy is just completely incompatible.
② Links. The only complete solution is to use actual link elements. My complaints about scrawl-canvas’s handling of links <https://news.ycombinator.com/item?id=22887235> when you submitted it here 2½ years ago still apply: to get it right, you’ve got to use actual links, placed in the right place, so that that’s what the cursor is actually interacting with the whole time. The difference here from scrolling is that they can be transparent elements on top of everything else. On reflection, I think it might be theoretically possible for AccessKit to achieve this itself in the simplest cases, but that would certainly break egui’s conception of how things should work, and it can’t work generally because of event handling, the possibility of a non-rectangular hit box, and the fact that egui is a compositor (you could have a window completely covering your link).
And so I say: these are unfixable issues in egui’s design.
(I also report strongly negative experiences with every canvas-plus-DOM approach I’ve encountered. They never get everything right, and the duplication required to get things as close to right as possible almost always leaves me wondering why you bothered with the canvas parts in the first place.)
I’ll focus on the two most obviously significant issues:
① Scrolling. The only complete solution is to draw within scrolling elements. To treat scrolling content distinctly, and allow the browser to act as the compositor. This would require rebuilding egui from the ground up; its current philosophy is just completely incompatible.
② Links. The only complete solution is to use actual link elements. My complaints about scrawl-canvas’s handling of links <https://news.ycombinator.com/item?id=22887235> when you submitted it here 2½ years ago still apply: to get it right, you’ve got to use actual links, placed in the right place, so that that’s what the cursor is actually interacting with the whole time. The difference here from scrolling is that they can be transparent elements on top of everything else. On reflection, I think it might be theoretically possible for AccessKit to achieve this itself in the simplest cases, but that would certainly break egui’s conception of how things should work, and it can’t work generally because of event handling, the possibility of a non-rectangular hit box, and the fact that egui is a compositor (you could have a window completely covering your link).
And so I say: these are unfixable issues in egui’s design.
(I also report strongly negative experiences with every canvas-plus-DOM approach I’ve encountered. They never get everything right, and the duplication required to get things as close to right as possible almost always leaves me wondering why you bothered with the canvas parts in the first place.)
For native, it's pretty awesome, and it's nice that you can get a hint of what it can do by checking out the web demo.
I agree. I just wish that it was paired with a banner along the lines of “this demo works on the web, but egui is focused on other platforms and you probably shouldn’t use it for the web” paired with a link to an article detailing the problems and how unsuitable it is.
I was also surprised to learn how nice EGUI is. Big widget kit, clean API (Which is often not true for Rust dependencies!) and integrates into WGPU-rendering. It allows me to use Rust to make Desktop applications.
I'm concerned about the performance hit of an immediate-mode GUI when not tieing it into a 3D render (ie the CPU access from repeatedly re-drawing), but haven't compared.
I'm concerned about the performance hit of an immediate-mode GUI when not tieing it into a 3D render (ie the CPU access from repeatedly re-drawing), but haven't compared.
In my experience it's surprisingly fast. Egui doesn't need to redraw when it's idling, only when you're interacting with it (mouse move) or it's animating something so IMO it's not a big deal.
I have to admit that I've only tested it on high-end laptops/phones though. Not sure about mid/low-end ones yet.
I have to admit that I've only tested it on high-end laptops/phones though. Not sure about mid/low-end ones yet.
Do note that this AccessKit integration doesn't yet help with the web backend of egui, only the winit-based backend on Windows and macOS. A web platform adapter for AccessKit is planned, but not a top priority.
Good to know. Thanks for clarifying!
It seems to cause high CPU usage whenever you move the mouse in the web demo.
That's because egui uses an "immediate mode" approach: it's rendering everything from scratch every time something _might_ have changed (e.g. a mouse move could've trigged a hover effect).
Whereas the DOM uses "retained mode": it continually modifies a representation of the layout (DOM elements) and re-renders when something _actually_ changes.
You can click on the "Backend" button at the top, and "Mean CPU usage" will tell you how long it takes for each frame to be produced. It'll vary depending on how complex the layout is: e.g. how many windows are open. But, contrary to retained mode, it doesn't matter at all if everything is animating simultaneously.
More about immediate vs. retained: https://github.com/emilk/egui#why-immediate-mode
Whereas the DOM uses "retained mode": it continually modifies a representation of the layout (DOM elements) and re-renders when something _actually_ changes.
You can click on the "Backend" button at the top, and "Mean CPU usage" will tell you how long it takes for each frame to be produced. It'll vary depending on how complex the layout is: e.g. how many windows are open. But, contrary to retained mode, it doesn't matter at all if everything is animating simultaneously.
More about immediate vs. retained: https://github.com/emilk/egui#why-immediate-mode
I really like egui, having always viewed immediate mode as only for games or small basic programs was blown away how little resources it actually consumes even for some fairly complicated gui's.
Coming from a retained mode world it's so much easier to read, write and reason about.
Don't find the wasm/web compilation as useful though, it does seem to hit performance quite badly, not sure if that's the nature of the beast in general or something else to do with optimisations for the library.
Coming from a retained mode world it's so much easier to read, write and reason about.
Don't find the wasm/web compilation as useful though, it does seem to hit performance quite badly, not sure if that's the nature of the beast in general or something else to do with optimisations for the library.
AccessKit looks great, and I'm thrilled to see a working integration for it.
Egui is an awesome tool for building quick-and-dirty GUIs: we use it at work for our game's developer tools.
Egui is an awesome tool for building quick-and-dirty GUIs: we use it at work for our game's developer tools.
Excited to see AccessKit. Finally a performant backend among other things for screen readers and voice control. I'm curious to see how subscribing to events and watching for changes in specific controls will be defined. In addition, python bindings would make it much more accessible as the code base solidifies towards production.
The way that screen readers and other assistive technologies subscribe to events is defined by the platform accessibility APIs. AccessKit simply sends the right events to whatever clients are listening. Note, though, that on both Windows and macOS, AccessKit doesn't send any events until the first client makes an accessibility request.
I want to provide Python bindings at some point. That would enable Kivy to implement accessibility, for example. Just haven't gotten to it yet. If anyone wants to step up and work on that, let me know.
I want to provide Python bindings at some point. That would enable Kivy to implement accessibility, for example. Just haven't gotten to it yet. If anyone wants to step up and work on that, let me know.
Funny name. Makes sense, but egui means something like “harsh, bitter, nasty” in Japanese.
Game changing. EGUI just graduated to “able to be considered for serious projects.”
How do i write a long commit message like that?
This particular commit was a squash merge of a GitHub pull request. So GitHub generated the commit message from the commits on the branch that was merged.
Don't pass the -m argument to git commit. It will open you $EDITOR an you can edit the commit message there.
Disclosure: Google Fonts funded Matt's work and I was involved in the decision to do so.