DMD Compiler as a Library: A Call to Arms(dlang.org)
dlang.org
DMD Compiler as a Library: A Call to Arms
https://dlang.org/blog/2024/02/22/dmd-compiler-as-a-library-a-call-to-arms/
41 comments
'Compilers' should be incremental (and iterative), and the asymptotics are way more important than the constant factors; if the conceptual models are not adequate to effectively express very fine-grain incrementality, no amount of bitpacking is going to save you. https://arxiv.org/pdf/2104.01270.pdf as an example—this is not perfect, but very serviceable.
Being able to serialise is good and valuable and useful, and enables some very interesting things, but you can still do a lot of useful things with just a persistent process working on in-memory structures.
Being able to serialise is good and valuable and useful, and enables some very interesting things, but you can still do a lot of useful things with just a persistent process working on in-memory structures.
Thx for link.
Asymptotics do always win but consider but consider the price of the win i.e. bitpacking is cheap (and fun) — not even just an academic exercise, this is where you get to use your hard won intuition.
Asymptotics do always win but consider but consider the price of the win i.e. bitpacking is cheap (and fun) — not even just an academic exercise, this is where you get to use your hard won intuition.
Not saying it's not worth it, just, it's strange to put them in the same sentence when one is what makes the difference between usable and completely unusable. I lived with 20-60s dirty build times for 1-line changes at symmetry, solely because of redundant compile-time recomputation—I wouldn't wish that on anybody. 10x would have been on the upper edge of tolerable, but you certainly won't get 10x for 'cheap'—and even so, latency should rightfully be measured in the milliseconds, not seconds. (~10x, incidentally, is quoted as an upper bound for newctfe here https://forum.dlang.org/post/[email protected]..., had it ever materialised.)
newCTFE was actually hitting 50x on some things I benchmarked — locality again, to a good approximation.
And my initial post did mention doing too much work in the first place.
And my initial post did mention doing too much work in the first place.
Given how fast the D compiler is, this might be a nice alternative to something like LuaJIT for an embedded scripting language in other apps.
Many people do use D as a scripting language, replacing their use of bash. D programs can be compiled and run in one step.
#!/usr/bin/env rdmd
import std.stdio;
void main()
{
writeln("Hello, world with automated script running!");
}
> ./myprog.d
Hello, world with automated script running!
Classically, you wait until your name is said three times, before appearing before the supplicant!
Do you also have:
1. Statically linked executables (essential but not crucial)
2. Dependency management (import tripleo.rtfm from "git:...#1234567")
3. A good AST API (jdt/roaster, javapoet, etc)
4. Tooling (cf spoon/soot)
(I approve this comment, even given it's limited usefulness/expressivity)
1. Statically linked executables (essential but not crucial)
2. Dependency management (import tripleo.rtfm from "git:...#1234567")
3. A good AST API (jdt/roaster, javapoet, etc)
4. Tooling (cf spoon/soot)
(I approve this comment, even given it's limited usefulness/expressivity)
> 2. Dependency management (import tripleo.rtfm from "git:...#1234567")
Not built into the compiler, but Dub, the official package manager distributed with the compiler, should be what I think your concise statement is referring to.
Not built into the compiler, but Dub, the official package manager distributed with the compiler, should be what I think your concise statement is referring to.
1. yes
2. no
3. that's what this topic is about
4. yes
2. no
3. that's what this topic is about
4. yes
This is what HipremeEngine does for example.
https://github.com/MrcSnm/HipremeEngine
The backend/frontend separation sounds right but the "sub" split of the frontend sounds odd.
You dont care if in the AST some functions are doing too much. In my opinion the most important is the ability to _drive the compilation_, for example stop after lexical, stop after semantics. Then if too much code got compiled into the *.a that's unfortunate but not dramatic.
BTW there's is also the split between the driver and the frontend, but I think this is done.
You dont care if in the AST some functions are doing too much. In my opinion the most important is the ability to _drive the compilation_, for example stop after lexical, stop after semantics. Then if too much code got compiled into the *.a that's unfortunate but not dramatic.
BTW there's is also the split between the driver and the frontend, but I think this is done.
I wanted to like D for a long time, feels like Go without the limitations to me.
But it's taking A LONG TIME to become usable.
But it's taking A LONG TIME to become usable.
I found the importC feature interesting, I tried the example code (with running the hello world in C using the importC flag) and the compiler threw lots of warnings.
https://dlang.org/spec/importc.html#examples
https://dlang.org/spec/importc.html#examples
I did not get any warnings running it on Ubuntu.
What warnings are you seeing? What operating system?
What warnings are you seeing? What operating system?
Thanks for checking up,
I did it on my Winows 10 machine, it's kinda too late to start it up now and run it but I'll do it tomorrow and post both the code and the error message.
Update: I did this quickly on my MBP.
Code: #include <stdio.h> #include <stdlib.h>
int main() { puts("Hello, world!"); return EXIT_SUCCESS; }
Command: dmd test.c
Error: ld: address=0x0 points to section(2) with no content in '/Users/james/.asdf/installs/dmd/2.107.0/dmd2/osx/lib/libphobos2.a[3233](config_a98_4c3.o)' clang: error: linker command failed with exit code 1 (use -v to see invocation) Error: linker exited with status 1
I did it on my Winows 10 machine, it's kinda too late to start it up now and run it but I'll do it tomorrow and post both the code and the error message.
Update: I did this quickly on my MBP.
Code: #include <stdio.h> #include <stdlib.h>
int main() { puts("Hello, world!"); return EXIT_SUCCESS; }
Command: dmd test.c
Error: ld: address=0x0 points to section(2) with no content in '/Users/james/.asdf/installs/dmd/2.107.0/dmd2/osx/lib/libphobos2.a[3233](config_a98_4c3.o)' clang: error: linker command failed with exit code 1 (use -v to see invocation) Error: linker exited with status 1
I'm wondering why you're linking with an OSX library on Windows 10!
I’m not, sorry for the confusion. I compiled the code on my macbook instead because it was too late to start my win 10 computer.
I think windows threw a similar error.
I think windows threw a similar error.
I added your code to the D test suite. It passes on all supported platforms, including Windows and OSX. I am at a loss why it isn't working for you.
https://github.com/dlang/dmd/pull/16238
https://github.com/dlang/dmd/pull/16238
Here's the pipeline spitting out the same error as on my macbook did.
https://github.com/dlang/dmd/actions/runs/8023469412/job/219...
https://github.com/dlang/dmd/actions/runs/8023469412/job/219...
Thank you. Did you mean to #include stdio.h twice?
No I did not mean to include stdio.h twice. I don’t think I am doing that either?
Sorry, my bad, I misread it.
On my windows machine, running the following code:
#include <stdio.h>
#include <stdlib.h>
int main()
{
puts("Hello, world");
return EXIT_SUCCESS;
}
with the following command:
dmd main.c
gives the following error:
main.c(1): Error: C preprocessor directive `#include` is not supported
main.c(1): Error: no type for declarator before `#`
main.c(2): Error: C preprocessor directive `#include` is not supported
main.c(7): Error: no type for declarator before `return`
main.c(8): Error: no type for declarator before `}`
Running the c code using docker just says that it cannot process c extensions.
docker run --rm -it -v .:/src dlanguage/dmd dmd main.c
#include <stdio.h>
#include <stdlib.h>
int main()
{
puts("Hello, world");
return EXIT_SUCCESS;
}
with the following command:
dmd main.c
gives the following error:
main.c(1): Error: C preprocessor directive `#include` is not supported
main.c(1): Error: no type for declarator before `#`
main.c(2): Error: C preprocessor directive `#include` is not supported
main.c(7): Error: no type for declarator before `return`
main.c(8): Error: no type for declarator before `}`
Running the c code using docker just says that it cannot process c extensions.
docker run --rm -it -v .:/src dlanguage/dmd dmd main.c
You might be using an older version of dmd. ImportC's support for the preprocessor is fairly new.
I got it to work on my Windows machine.
The solution was that I had to install Visual studio and add the workload (plugin?) "Desktop development with c++".
The only thing needed to be checked was "MSVC v143" and "Windows 11 SDK", the rest is bloat.
After that I had to put the following path from Visual studio "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\bin\Hostx64\x64" into my PATH.
This is because I needed the "cl.exe", "sal.h" and probably a couple of other files to be available to dmd.
(Also, the outdated version of dmd was another factor of the issue)
Finally, the compilation with main.c went smoothly. I had no idea the compiler needed external tooling, I thought it was self sufficient.
The solution was that I had to install Visual studio and add the workload (plugin?) "Desktop development with c++".
The only thing needed to be checked was "MSVC v143" and "Windows 11 SDK", the rest is bloat.
After that I had to put the following path from Visual studio "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\bin\Hostx64\x64" into my PATH.
This is because I needed the "cl.exe", "sal.h" and probably a couple of other files to be available to dmd.
(Also, the outdated version of dmd was another factor of the issue)
Finally, the compilation with main.c went smoothly. I had no idea the compiler needed external tooling, I thought it was self sufficient.
> I had no idea the compiler needed external tooling, I thought it was self sufficient.
It used to be self-sufficient. But, we had to give up on our linker because we didn't have the staff to keep a linker up to date, and decided to stop chasing windmills and just use what was available. ImportC, in order to work, needs a preprocessor. We do have our own Standard compliant preprocessor, but it had a major problem. All the C preprocessors predefine a ton of macros (cpp predefines up to 400 macros). Different combinations of compiler switches turn on or off those predefines in poorly and usually undocumented ways. It differs from preprocessor to preprocessor, and was a hopeless nightmare.
So, ImportC uses the C preprocessor of the associated C compiler, and that works like a champ. The downside is you'll need to have that preprocessor installed and on the path.
It used to be self-sufficient. But, we had to give up on our linker because we didn't have the staff to keep a linker up to date, and decided to stop chasing windmills and just use what was available. ImportC, in order to work, needs a preprocessor. We do have our own Standard compliant preprocessor, but it had a major problem. All the C preprocessors predefine a ton of macros (cpp predefines up to 400 macros). Different combinations of compiler switches turn on or off those predefines in poorly and usually undocumented ways. It differs from preprocessor to preprocessor, and was a hopeless nightmare.
So, ImportC uses the C preprocessor of the associated C compiler, and that works like a champ. The downside is you'll need to have that preprocessor installed and on the path.
I appreciate the explanation, that makes total sense.
The burden to maintain such a preprocessor doesn’t seem that rewarding compared to using existing ones.
The burden to maintain such a preprocessor doesn’t seem that rewarding compared to using existing ones.
The author, Razvan Nitu, joined the D community a few years ago and is now a key member of the core Dlang team.
This looks like more fun than sudoku or crossword puzzles.
I can vouch for it being a lot more fun! Doing puzzles is a useless waste of time.
Not true! Doing puzzles can help mitigate risk for dementia. Probably not necessary for those of us who do puzzling work day to day, but there is a solid benefit to society if people are in the habit of doing puzzles.
Working in programming is often a puzzle, and something useful is accomplished beyond exercising your brain.
For an analogy, if you walk to work every day instead of driving, you are accomplishing something useful beyond just the exercising.
My dad always tried to get a residence that was about a 15 min walk to work.
For an analogy, if you walk to work every day instead of driving, you are accomplishing something useful beyond just the exercising.
My dad always tried to get a residence that was about a 15 min walk to work.
[deleted]
> puzzling work
You don't say. Appalling, even!
You don't say. Appalling, even!
Swift does something like this I believe. Not too sure on the details but I remember reading about how you can modify the parser to do...things.
Looked interesting.
Looked interesting.
It's part of the LLVM philosophy that the compiler tools should be usable as a library. There was a not insignificant amount of drama[0] when the folks working on LLD (the then-new LLVM linker) decided they weren't going to follow that pattern. It was another 5ish years before they revisited that decision[1], but it's not a simple task to retrofit a project with proper error handling (exit on error is fine for a binary, but not a library), memory management (leaking is faster than freeing memory for a short lived process, but isn't acceptable in a library), etc.
[0]https://discourse.llvm.org/t/lld-status-update-and-performan...
[1]https://discourse.llvm.org/t/rfc-revisiting-lld-as-a-library...
[0]https://discourse.llvm.org/t/lld-status-update-and-performan...
[1]https://discourse.llvm.org/t/rfc-revisiting-lld-as-a-library...
This is kind of like a blind man clapping to echolocate his way around a maze. It's better than nothing but the issue is that the compiler just isn't amenable to non-batch work (e.g. the semantic analysis is all or nothing, it uses WAY too much memory[0], and it can't serialize it's state to the disk).
You could already use the frontend as a library, almost no one does. It's not because the API.
[0] SDC can pack a lot of types into a few bytes. This is the kind of thing real refactoring allows — this change (packing types, to be specific) is something I would happily chip in and help out with.
Memory layout and locality is where performance lies in a compiler (that and doing less work).