New release of my authentication service written in C. It now offers "multi-reactor" (multiple threads each running their own event loop). In my jmeter tests, it can now support 40,000 req/s fetching the login form and performing login POSTs on my old Xeon E3-1240L v4 (quad-core low-power CPU) which isn't even idle ... of course with everything optional (TLS, request logging, remote host resolving) disabled.
Might be overkill to tune such a simple tool for performance, but I hope to reuse the code eventually for other web services.
Features: Authentication with PAM, a htpasswd-like file with bcrypt hashes, executing some external tool, AND "proof-of-work" crypto challenge (to block malicious bots).
TBH, I've shown this before, but a LOT has changed since then, most notably:
* Performance improvements by adding support for epoll() (with singalfd and timerfd), kqueue() and solaris event ports, and also by fixing a few design issues
* Server side sessions are gone, instead signed Json Web Tokens are implemented -> saves lots of RAM
Among other things, swad can do what Anubis can do, that's why it's in the title. It's much smaller though, and proof-of-work-crypto-challenge-to-defeat-bots is just ONE module, the other modules offering more regular means of authentication: by calling some external tool, by checking a bcrypt hash from a file, or by asking PAM.
Contrary to Anubis, it's NOT a reverse proxy but designed to work with nginx' auth_request facility instead, might also work with other reverse proxies if they offer something similar. Swad is written in C, compiles to a 200-300kiB (depending on compiler and platform) binary, needs only zlib, OpenSSL (LibreSSL) and optionally libpam. Doing some stress testing with 1000 distinct simulated clients, I managed to bump up the resident set on my FreeBSD machine to 100MiB, but no more. Without heavy load, it typically stays at much less.
The general architecture of swad is a reactor with attached thread pool. A HTTP request is parsed (to headers and body) in the main thread, then a thread job is created to run the handler pipeline (including response compression, cookie management, form and query string parsing, etc) that's expected to add a HTTP response to some context object, once that's finished, the main thread takes over again sending that out to the client.
During my stress test mentioned above, I was kind of surprised to see the main (reactor) thread almost maxing out one core, while the pool threads (running request pipelines) didn't have too much to do. Well, serving a small login form and processing login requests indeed isn't THAT much work. Maybe it might make sense to have more than one "reactor thread"? Could this work, e.g. by just listening on the same socket from multiple threads?
BTW, it also offers authentication with PAM, password files similar to Apache's .htpasswd and by calling some external tool. And it compiles to a (currently) 176kiB binary, which has the default templates, style.css and the javascript for the crypto puzzle built in.
I use it myself on my private server to protect my package build logs (!) from hungry AI bots. WTF...
swad is the "Simple Web Authentication Daemon". I just started it and crafted a first release that's still pretty limited in functionality, but does something (IMHO) useful, which I already use myself. More info in the github README.md.
There's a background story. For a few months, I was wondering why most of the time, I couldn't activate my camera in MS Teams meetings (for work) without completely killing my audio ... obviously a bandwidth issue with my upstream that hasn't been there before. Then I stumbled across some article, I think it was ars technica, describing a horrible problem with misbehaving AI bots crawling for data with no limits and no respect (and even trying disguises), causing extreme amounts of traffic and corresponding problems for opensource projects hosting their own infrastructure. Gave me the idea to finally check my nginx logs again. Ok, it was full of bots. The interesting thing is, I don't host any opensource code there. What they downloaded were build logs of my "poudriere" (a package builder for FreeBSD), which can get very large, and I just had them public because there was no reason to hide them ... well until now. WTF do those bots learn from that, uhm, "shit scrolling by"?
So, what to do here? Although I really felt inclined to try one of the nice and naughty tools to poison these bots (IDK, is build logs already some kind of poisoning?), it was clear the pragmatic thing to do is add authentication. Well, you can easily add some HTTP "Basic" auth, but that's far from great. You're typically limited to checking credentials against some "password file", and worse, e.g. Microsoft already decided that "Basic" is "insecure" (oh dear, really? I don't think so as long as you use TLS) and disabled it by default in their "Edge" browser. And chances are you can't change that when using some device you got from your employer.
I did have some old dusty code implementing some very limited HTTP stuff in C lying around that was never published (only used it for experiments with some darknet "hidden services"). I thought it might be fun to implement something. Yep, it was fun. And call me stubborn, but I like doing stuff in plain C, small, efficient, few dependencies. Finally this old HTTP code got some polish and is released as part of something actually useful.
For completeness, I'm well aware I could have taken a very different road. "Just" install some identity provider implementing OIDC. For example keycloak. Well sure, that's awesome stuff. And huge and complex. If you don't have a massively distributed deployment of services and don't intend to ever federate with other organizations, it just felt like using the H bomb to kill a fly.
So, let's look at this group of people knowledgeable in programming et al, but not english. Sure, this group exists. It's relatively tiny though. English is the language in any (modern, IOW except where it didn't replace latin yet) science, including CS. The vast majority of other on-topic literature is english. It's hard to learn a decent amount of stuff concerning programming without knowing english.
You can't randomly mix languages in source code, and any other choice of common language than english would exclude a LOT more people from the project.
All of this isn't related to i18n/l10n of your application at all. People not knowing english is a much more relevant factor when talking about user interfaces. I actually plan to localize my Xmoji tool eventually, it's just postponed until more important stuff is done (I mean, I assume most people would get together enough english to be able to use an emoji keyboard after all, but of course, especially seaching would benefit from l10n).
For Wayland, keyboard input is "out of scope". (edit: Not entirely, just verified it does forward very basic events, but it's a thing wayland doesn't want to handle, while X11 originally had its own keyboard/mouse/etc drivers) What's typically used is XKB (inherited from X11) with a different backend ([edit: raw events]). So yes, in practice, you'll have the same broken design.
> Also, there seem to be no sane API for managing layouts or switching
> them programatically
Layouts only exist as "data" as far as the X server is concerned, clients must fetch them and map themselves from the key codes. Libraries like xkbcommon (or even grandfather Xlib) do the job for you. That said, there are APIs to modify the mapping (by publishing messages/events) as you wish. The ugliness is, apart from the fact that you're forced to fiddle with the mapping at all, that you can't guarantee another client will process everything in sequence. It might apply a new mapping before processing all its queued key press events. That's why my code adds delays between fiddling with the mapping and sending the events.
Yes, it's extremely ugly. Still, at least for me, it works. Try it out ;)
And yes, Windows is doing better here, there's a Unicode-flavor of keyboard events available.
Thanks for the pointer, this made me have another closer look at xkbcommon's API. Probably not too relevent for the tool I'm currently writing, but I'll keep that in mind: Instead of a key symbol, you can also obtain an utf-8 encoded string. Nice.
Still, in practice, it will only work with some applications...
Note input is "out of scope" for wayland and XKB is used there for keyboard input as well. I just had a quick glance at "wtype" source and indeed, it fiddles with the keymap, using xkbcommon. So, assuming a suitable X11 replacement for this "wofi" tool, you could do exactly the same on X11 using "xdotool".
Using input methods is the "technically best" solution of course, but it needs configuration and some application support, therefore I wanted something working with "plain X" :)
Sending fake key events is unfortunately "hacky". You need to temporarily switch the keyboard mapping, and many X applications have races between applying a new mapping and processing key press events, that's why I added (now configurable, new release soon) delays... and also, I found applications (e.g. chromium) having issues to correctly combine ZWJ sequences from individual key events, so I added more "hacks" to help with that, also just made configurable.
To sidestep these issues, my tool also offers transfer via "primary selection": Just middle-click an emoji to select it, and middle-click again where you want to insert it. Downside: Two clicks instead of just one.
Xcompose only offers sequences started with the "compose" key (whatever it is mapped to) and resulting in a single X key-symbol, so to type an emoji needing multiple uniciode codepoints that way, you'd have to enter multiple "compose sequences". Whatever you describe here is not simple Xcompose. AFAIK, GTK offers entering unicode codepoints in a similar way ...
Of course you can do a lot of things with "input methods" (like popping up some "picker" only when wanted). But then you'll depend on X clients being aware of them in some way, e.g. by implementing the XIM protocol. Lots of applications have that (even xterm), so it might be what you prefer and that's fine. My method with faked key press events is definitely hacky, but will work with any X client, that's the point of it ;)
Not sure you really mean the compose mechanism provided by Xlib (and xkbcommon etc ...)? If so, this wouldn't work for all these emojis that are represented by grapheme clusters, ZWJ-sequences or are just "qualified". The result of a compose sequence is limited to a single "key symbol", which can only encode a single unicode codepoint.
Even for those emojis consisting of a singe codepoint, it would really miss the point. There are LOTS of emojis available, an emoji keyboard (or emoji picker) shows them organized in groups and allows searching for them.
In case someone wonders what's "special" about it:
user perspective: As far as I know, it's the only tool offering somewhat reliable emoji input using faked X11 keyboard events. "x11-emoji-picker" comes close, but delegates faking events to "xdotool", with weird consequences like having to restart it to direct emojis to a different window ;)
hacker perspective: It directly integrates xcb in a different event loop (here based on pselect()) and actually works – xcb is IMHO pretty great except for pretty much enforcing its own async model, internally using poll() and reading from the X server into buffers at many occassions, hidden from the consumer. If all your app does is communicate with the X server, you just happily use xcb's model. As soon as you need different things (e.g. local timers), you have a problem. The typical way to address that is dedicating another thread to xcb (AFAIK also done that way by toolkits offering an X11 backend). I hated that idea enough to be stubborn about doing it directly in my main thread, and with quite some trial-and-error, finally succeeded.
If your goal is learning yourself, I guess you should really implement your stuff down on the system API level (using sockets and so on) ;)
If, OTOH, you want to get up some simple/small service quickly, I hope "poser" can help. I'm not sure the docs are already "good enough" (it's still a very young project ...) -- there's also a real example using it here: https://github.com/Zirias/tlsc/blob/master/src/bin/tlsc/tlsc...
A "service" certainly doesn't mean SOA (using SOAP), it doesn't even mean any specific form of communication (there might be cases not needing any communication), it basically just means some hosted piece of software.
The idea of the "core" part of "poser" (the only one implemented so far) is to provide a framework (and toolkit) to easily implement your service targeted at hosting on POSIX platforms. Among other things, it does offer abstractions for socket communication of the STREAM type, as this is what many services will use for communication. At this low level, it is completely protocol agnostic. All that's offered is some convenience API to tell the connection what's expected to be received next: binary data (optionally of a specific size) or strings/text with any strategy to find the end of the message (offering an implementation finding typical "line endings" as this is what many text-based protocols use).
I might add more libraries to "poser" in the future, e.g. to offer HTTP functionality.
SOAP is one thing that certainly won't be added though. SOA is dead. IMHO, rightfully, it's just an over-complex mess. When you offer HTTP-based services nowadays, you typically use REST semantics (using the verbs offered by HTTP for the actions, working on resources identified by the URI). For the documents exchanged in a REST API, JSON is a popular format choice. I don't know whether I will implement anything supporting communications using HTTP/REST, but that might be an option one day.
It certainly is not. Libuv (and friends) abstract platform-specific APIs like epoll (Linux), kqueue (FreeBSD), IOCP (Windows) etc, allowing to scale up to lots of concurrent clients/connections. I would recommend to use such a library for anything where you'd ever expect having to handle more than maybe 100 concurrent clients.
This is a small and simple alternative (the only similarity is offering some event-based API) using nothing but POSIX. This allows to keep the code small and avoid conditional compilation (so far only used for the optional TLS support). You might want to use it when you want to write some "small" service for POSIX systems, and focus on your actual logic (e.g. your protocol implementation) instead of dealing with all the internals of handling sockets, name lookups, daemonizing, signals, threading where needed, and so on.
Might be overkill to tune such a simple tool for performance, but I hope to reuse the code eventually for other web services.
Features: Authentication with PAM, a htpasswd-like file with bcrypt hashes, executing some external tool, AND "proof-of-work" crypto challenge (to block malicious bots).