What I Learned from porting my projects to FreeBSD(github.com)
github.com
What I Learned from porting my projects to FreeBSD
https://github.com/shlomif/what-i-learned-from-porting-to-freebsd
151 comments
Isn't referencing something like "http://docbook.sourceforge.net/release/xsl/current/manpages/... wrong, because that refers to some unspecified version (note the "/current/" bit)?
--nonet is also a necessity for security reasons. The downloaded XSLT stylesheets are not authenticated in any way.
If your makefiles use gnu-isms (which is a valid choice) then I would lean towards using GNUmakefile and have Makefile call gmake - and don’t forget to use $(MAKE) for any recursion.
This is less fragile and requires less dependence on replacing make in the $PATH.
This is less fragile and requires less dependence on replacing make in the $PATH.
Glad I’m not the only one: https://github.com/neosmart/gmake-proxy
All maintained and ready to go.
All maintained and ready to go.
A genuine question. What is a highly needed feature that GNU Make have that you can't do with POSIX Make?
My personal pet peeve with POSIX Make is that you can't export an environment variable for one target only without calling $(MAKE)[1].
[1] https://stackoverflow.com/q/52427295/1892060.
My personal pet peeve with POSIX Make is that you can't export an environment variable for one target only without calling $(MAKE)[1].
[1] https://stackoverflow.com/q/52427295/1892060.
GNU Make's got tons of features that aren't in POSIX Make (so does BSD Make).
I use GNU-specific Makefile features all the time - here's a list of some of the cool features:
But with that said, I use GNU Make all the time for "glue automation" - like if I had a folder full of files that needed some processing, I'd probably write a Makefile instead of a shell script. I also use top-level Makefiles on projects that have their own build systems, so that I don't need to remember how they work every time. People use shell scripts for that stuff too, but Make is better IMO.
I use GNU-specific Makefile features all the time - here's a list of some of the cool features:
- Built-in functions (see https://www.gnu.org/software/make/manual/html_node/Functions.html for a list)
- User-supplied functions
- Much better pattern substitution
- Ways to tell Make that some files can be skipped in certain cases (.INTERMEDIATE and .SECONDARY modifiers)
- Ways to adjust Make's behavior (SECONDEXPANSION, DELETE_ON_ERROR, ONESHELL)
But with that said, I use GNU Make all the time for "glue automation" - like if I had a folder full of files that needed some processing, I'd probably write a Makefile instead of a shell script. I also use top-level Makefiles on projects that have their own build systems, so that I don't need to remember how they work every time. People use shell scripts for that stuff too, but Make is better IMO.
I'm curious - what does autotools / CMake actually give you? Personally, I find they generate awful Makefiles which are slow and aren't debuggable.
Cheap portabiltiy. Easy use by packagers.
It annoys me quite a bit when someone decides not to use autotools or meson, or other standard build tools. I have a build system set up to build static packages for several architectures, and most of my packages are described like this:
If you roll out your own Makefile, I have to figure out how to build the targets, how to install them, whether you use DESTDIR, whether you respect cross-compile options, how to build static library, how to configure your package... and code all this by hand to build() function.
The same goes for my custom Arch Linux packages.
Make the packagers job easy and just don't roll your own if you want your package to be distributed widely.
It annoys me quite a bit when someone decides not to use autotools or meson, or other standard build tools. I have a build system set up to build static packages for several architectures, and most of my packages are described like this:
name=some
ver=123
url=https://where-to-get/some-$ver.tar.gz
build() {
autotools_build --some-option
}
If you roll out your own Makefile, I have to figure out how to build the targets, how to install them, whether you use DESTDIR, whether you respect cross-compile options, how to build static library, how to configure your package... and code all this by hand to build() function.
The same goes for my custom Arch Linux packages.
Make the packagers job easy and just don't roll your own if you want your package to be distributed widely.
I was so hopeful for cmake in the early days, but have come to be a bit disappointed and avoid it everywhere I can. I shake my head in amazement at the project that spawned cmake (ITK, a medical imaging project) was using Tcl as an embedded scripting language (indeed, one of Tcl’s great features), then faced with needing a scripting language for their build system, decided to ad-hoc build one from scratch. Additionally (perhaps related), as a consumer of software that uses Cmake as its build infrastructure, I do not look fondly on the times I’ve had to debug what’s going on when the promised portability doesn’t work out on my systems. Perhaps it’d also be the same for an autotools build, or sufficiently complex vanilla Makefile, but Cmake does seem to a standout in my mind. This might be because I was so hopeful for it a dozen years ago, but I think the corner cases became too annoying because of their disappointing ad-hoc language, which was an avoidable mistake.
That all said, it’s still clearly managed to be important, and is indeed something the authors can be proud of, though I have personally come to find joy and good success in plain Makefiles.
That all said, it’s still clearly managed to be important, and is indeed something the authors can be proud of, though I have personally come to find joy and good success in plain Makefiles.
I understand. In the past I used plain Makefiles too. GNU make is powerful. But it can get unpleasant and unreadably verbose pretty fast.
For personal projects I use either meson, or if they are sufficienlty complex (targetting multiple architectures in a single project and other tricky things) and the environment is known/fixed, I genereate build.ninja file from a PHP based configure script and it's unbeatable as to the flexibility, predictability and build speeds.
The best thing is that I can change anything and I can be sure that the incremental build will be the quickest and the result predictable. No more need for "make clean all" after changing some flags/variables in a Makefile.
It's the benefit meson/ninja combo will give you too, but meson is replaced by an actual well known programming language like PHP in my case.
For known environment builds, it's really nice. Much easier than fishing around the web for random recipies on how to do stuff in m4/meson/autoconf/automake/whatever custom language someone invented.
I really love the intentional simplicity of ninja build.
https://ninja-build.org/manual.html
For personal projects I use either meson, or if they are sufficienlty complex (targetting multiple architectures in a single project and other tricky things) and the environment is known/fixed, I genereate build.ninja file from a PHP based configure script and it's unbeatable as to the flexibility, predictability and build speeds.
The best thing is that I can change anything and I can be sure that the incremental build will be the quickest and the result predictable. No more need for "make clean all" after changing some flags/variables in a Makefile.
It's the benefit meson/ninja combo will give you too, but meson is replaced by an actual well known programming language like PHP in my case.
For known environment builds, it's really nice. Much easier than fishing around the web for random recipies on how to do stuff in m4/meson/autoconf/automake/whatever custom language someone invented.
I really love the intentional simplicity of ninja build.
https://ninja-build.org/manual.html
Could do all that, but FreeBSD could also have GNU make as the standard "make". I know there's some "license purity" reason why they don't want to do that, but don't expect everyone else to work around your self-imposed problem.
GNU make is ported to FreeBSD and very frequently used to build ports that use GNU-isms in their Makefiles. It's a non-issue really.
> GNU make is ported to FreeBSD and very frequently used to build ports that use GNU-isms in their Makefiles. It's a non-issue really.
At least the OpenBSD developers hated the Linuxisms so much that they made a song "Aquarela do Linux!" to name and shame this:
> https://www.openbsd.org/lyrics.html#52
At least the OpenBSD developers hated the Linuxisms so much that they made a song "Aquarela do Linux!" to name and shame this:
> https://www.openbsd.org/lyrics.html#52
No, GNU devs should understand Unix beyond the GNU ecosystem. I'd be retired on my private planet if I had gotten a dime each time I ported Linux software (mostly the build system) to work on non-linux unixoid operating system.
To be fair: Nothing in GNUiverse teaches you interoperability; I too learned it the hard way when switching from Linux to BSD.
To be fair: Nothing in GNUiverse teaches you interoperability; I too learned it the hard way when switching from Linux to BSD.
GNU is literally a platform. If you want to run the software on another platform (particularly one which is not as widely used as GNU) then you have to port the software.
Or you have to write portable software.
isn't make older than gmake? it is normally gnu that adds extensions not compatible with the rest of the Unix world. making people believe sh == bash, make == gmake, m4 == gm4, and the list goes on for everything they touch.
But BSD make has extensions too. They're just different extensions.
To be frank, many of the GNU extensions are very useful. BSD tools have their own extensions over POSIX as well. Anyway, GNU software is FOSS and tends to be portable and so you can create something like GNU/FreeBSD or GNU/OpenBSD or whatever (Debian did it in the past). Naturally, the GPL/LGPL/etc. may not be "free" enough for the tastes of the BSD devs, but this is getting into politics.
Every Unix used to have its own implementation of those standard tools. They were often quite buggy and outdated (eg the C compiler that came with SunOS and Solaris in the mid 1990's only supported K&R syntax).
When installing Solaris, Sinix, AIX etc, you'd typically install the GNU tools immediately.
I understand the FreeBSD folks' frustration, though.
When installing Solaris, Sinix, AIX etc, you'd typically install the GNU tools immediately.
I understand the FreeBSD folks' frustration, though.
“Embrace, extend, extinguish.”
For anyone interested, I'm providing FreeBSD CI services on sr.ht. We use it to test wlroots[0].
https://builds.sr.ht/~sircmpwn/job/9348
I'm also working on OpenBSD and NetBSD, other kernels, and non-x86 architectures. If anyone is interested in using this to make their software more portable, please let me know! Email in my profile.
[0] https://github.com/swaywm/wlroots
https://builds.sr.ht/~sircmpwn/job/9348
I'm also working on OpenBSD and NetBSD, other kernels, and non-x86 architectures. If anyone is interested in using this to make their software more portable, please let me know! Email in my profile.
[0] https://github.com/swaywm/wlroots
TravisCI has been asked repeatedly to provide FreeBSD as a build OS, and it doesn’t appear they’re interested in expanding beyond Linux/macOS on x86:
* https://github.com/travis-ci/travis-ci/issues/1818 * https://github.com/travis-ci/travis-ci/issues/6671 * https://github.com/travis-ci/travis-ci/issues/5473
Do you plan on providing Github integration? Providing other OSes and platforms (e.g. ARM64, ppc64el) would be really useful for testing low-level projects like .NET Core runtime, node.js / libuv, etc., and integrating with Github would make it really easy for people to start adding this to their existing projects.
If you do end up providing Github integration, consider listing a paid option for your service in the Github marketplace: https://github.com/marketplace/category/continuous-integrati...
* https://github.com/travis-ci/travis-ci/issues/1818 * https://github.com/travis-ci/travis-ci/issues/6671 * https://github.com/travis-ci/travis-ci/issues/5473
Do you plan on providing Github integration? Providing other OSes and platforms (e.g. ARM64, ppc64el) would be really useful for testing low-level projects like .NET Core runtime, node.js / libuv, etc., and integrating with Github would make it really easy for people to start adding this to their existing projects.
If you do end up providing Github integration, consider listing a paid option for your service in the Github marketplace: https://github.com/marketplace/category/continuous-integrati...
I already provide Github integration, but it's not very polished. That's how wlroots gets built, for example:
https://github.com/swaywm/wlroots/pull/1341
(click "show all checks")
I may add it to the marketplace once it's open for public registration and more polished.
https://github.com/swaywm/wlroots/pull/1341
(click "show all checks")
I may add it to the marketplace once it's open for public registration and more polished.
Are these services free?
We currently test Rust under qemu-system for FreeBSD and have no real good solution for OpenBSD and DragonflyBSD. It would be very interesting for us to do "native" testing.
We currently test Rust under qemu-system for FreeBSD and have no real good solution for OpenBSD and DragonflyBSD. It would be very interesting for us to do "native" testing.
They are currently free (and invite-only) but will soon be paid. I plan on letting users pick their price between $2, $5, and $10 per month, and all price points will have access to all features.
If you'd like an invite, feel free to reach out - [email protected]
If you'd like an invite, feel free to reach out - [email protected]
Just wait till you discover FreeNAS, pfSense, and OpenBSD! A world of awesome, battle-tested software awaits once you join the BSD dark side.
OpenBSD is one of those things you could be stranded on a desert island with the base system and still be productive with just that and the manpages.
pfSense is great for a prosumer router. What gets me about it is cleaning up the mess it creates in business use cases, as pfSense is designed with a mindset that the pro will always be there, ready to attach a screen at a moments notice when interfaces change or packages get updated. Running PHP as root also isn't great, I'm not sure how that passes in the context of HIPPA.
I think network security best practices addresses that issue. Keep network management interfaces off unrestricted local area or routeable networks. Huge swaths of popular software as well as very expensive hardware have many unresolved security issues. There is nothing you can do to make a lot of that stuff secure short of putting them on their own isolated network and explicitly controlling all ingress/egress traffic to it.
The HIPAA Technical Safeguards are compact, readable, and basic: https://www.law.cornell.edu/cfr/text/45/164.312
Okay, but how does running one of the most commonly exploited languages as root pass any HIPAA audit? That was the question asked in my prior comment.
We ended up ripping out PFSense at a few medical practices due in part to this issue.
We ended up ripping out PFSense at a few medical practices due in part to this issue.
You seem to be under the impression that the management UI is on a public network. Ideally locked down to certain vlans.
The UI controls setting up the network and other services. Please explain how to do this without root access. (Write another process which runs as root and controls the settings and is talked to over a Unix socket actually isn't a bad idea, however, it is not void of its problems either.)
Also "most commonly exploited languages" is a bit of hyperbole, no? First, C probably takes that slot. Second, being one of the most common languages for web development makes it a target. Third, most php exploits are bad code, which, while easy to write in php (and c!), can be and is done in all languages.
Isn't the administrator interface on _any_ router essentially root access on said router? Do you complain that juniper or Cisco equipment is insecure because you can login?
BMCs (Baseboard management controllers) are something with very ... questionable ... security, yet network segregation is used to ensure its use securely. Given that many HIPAA complaint organizations such as AWS and GCP (Google Cloud Platform) I find it hard to believe that a management interface would disqualify something from HIPAA compliance.
Which part of the HIPAA audit did pfsense fail? Was it simply an abundance of caution" on your part? If so, what did you replace it with that didn't have a management or has a management interface with no bugs (hint: even Cisco and juniper have CVEs for the management interface)?
The UI controls setting up the network and other services. Please explain how to do this without root access. (Write another process which runs as root and controls the settings and is talked to over a Unix socket actually isn't a bad idea, however, it is not void of its problems either.)
Also "most commonly exploited languages" is a bit of hyperbole, no? First, C probably takes that slot. Second, being one of the most common languages for web development makes it a target. Third, most php exploits are bad code, which, while easy to write in php (and c!), can be and is done in all languages.
Isn't the administrator interface on _any_ router essentially root access on said router? Do you complain that juniper or Cisco equipment is insecure because you can login?
BMCs (Baseboard management controllers) are something with very ... questionable ... security, yet network segregation is used to ensure its use securely. Given that many HIPAA complaint organizations such as AWS and GCP (Google Cloud Platform) I find it hard to believe that a management interface would disqualify something from HIPAA compliance.
Which part of the HIPAA audit did pfsense fail? Was it simply an abundance of caution" on your part? If so, what did you replace it with that didn't have a management or has a management interface with no bugs (hint: even Cisco and juniper have CVEs for the management interface)?
PHP makes it trivial to write bad code, and hard to write good code.
So does C?
Also, php in no way makes it hard to write good code, it's just easy not to.
Also, php in no way makes it hard to write good code, it's just easy not to.
Why not? Which part of the Security Rule would it violate?
It's probably a bad idea. But HIPPA is a specific and public block of text, which says less than most people seem to imagine it does. (I'm no lawyer. I've just read the thing).
It's probably a bad idea. But HIPPA is a specific and public block of text, which says less than most people seem to imagine it does. (I'm no lawyer. I've just read the thing).
You can always roll your own headless openBSD + pf
I can only recommend the same, OpenBSD + pf is simply perfect. If you manage a couple of firewalls add Ansible and you'll never consider vendor firewall appliances again:
https://github.com/liv-io/ansible-roles-bsd/tree/master/open...
> Running PHP as root
OPNsense was created to fix that. Now it is better than pfSense.
OPNsense was created to fix that. Now it is better than pfSense.
How is a disappearing network interface handled? What about package deperecation?
On both issues, PFSense left me with a non-working router, which is unacceptable when every other router software and hardware appliance out there will bring itself up in a hobbled state to pass traffic.
An optional ethernet port disappearing is not a good reason for your router to halt and catch fire.
On both issues, PFSense left me with a non-working router, which is unacceptable when every other router software and hardware appliance out there will bring itself up in a hobbled state to pass traffic.
An optional ethernet port disappearing is not a good reason for your router to halt and catch fire.
and DragonFlyBSD !
I use pfsense for my router at home and I love it! <3 I tried freenas but I couldn't get it to join my domain I should have tried harder.
Or just switch to Meson+Ninja [1][2] and enjoy all features of modern build systems.
[1] https://mesonbuild.com/
[2] https://ninja-build.org/
[1] https://mesonbuild.com/
[2] https://ninja-build.org/
Before mesom/ninja, you had one problem - install gmake. Now you have three problems - install python, ninja and meson ;)
Maybe fourth - learn how to use it
Maybe fourth - learn how to use it
> Before mesom/ninja, you had one problem - install gmake.
No, you had as many problems as there are '%' and '$' symbols in this web page : http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ multiplied by the number of permutations of tools in the different platforms you want to support, e.g. (gcc-mingw-windows, clang-cl-windows, msvc-windows (this one counts as 10), xcode-macos, gcc-macos). And let's not even get started with cross-compiling.
No, you had as many problems as there are '%' and '$' symbols in this web page : http://www.cs.colby.edu/maxwell/courses/tutorials/maketutor/ multiplied by the number of permutations of tools in the different platforms you want to support, e.g. (gcc-mingw-windows, clang-cl-windows, msvc-windows (this one counts as 10), xcode-macos, gcc-macos). And let's not even get started with cross-compiling.
Meson and Ninja cannot handle "C, Perl 5, Python, Ruby, GNU Bash, XML, CMake, XSLT, XHTML5, XHTML1.1, Website META Language, JavaScript and more."
One of make's great strengths is that it is so generic.
One of make's great strengths is that it is so generic.
Ninja is generic and minimalist. You can use it for anything that can be defined as a command with inputs and outputs.
Writing ninja files by hand is tedious and those extra features make provides can make it easier to use, but in principle ninja should be able to handle anything make could.
It also works in windows.
Writing ninja files by hand is tedious and those extra features make provides can make it easier to use, but in principle ninja should be able to handle anything make could.
It also works in windows.
[deleted]
For my projects which are written in C or C++, I use cmake which can generate BSD makefiles among other targets. For some other projects (websites/etc.) I use gmake directly because they are not related to C. Since some of my makefiles (e.g: https://github.com/shlomif/shlomi-fish-homepage/blob/master/... ) have become quite large, I've started looking for better build systems - see my current research at https://www.shlomifish.org/open-source/resources/software-to... .
I tried using meson for https://github.com/shlomif/mini-factor , and while it worked fine locally it turned out to require python 3.6 which made it hard to install on Travis-CI. I ended up created a complementary cmake build system.
I tried using meson for https://github.com/shlomif/mini-factor , and while it worked fine locally it turned out to require python 3.6 which made it hard to install on Travis-CI. I ended up created a complementary cmake build system.
In what sense isn't gnu make a modern build system?
Make is a much of a build system as C++ as you can call stat() and system() from it.
A modern build system by default should be able to provide utilities to make sure your build is correct and reproducible. Those features need to be coded into your Makefile and aren't available easily or by default.
If you need to code those rules, then you don't have a build system, just a way to run commands and possibly add some (usually poor) logic around them.
A typical example that most handcrafted Makefiles will fail is that changing a build flag should only rebuild the files affected. Most will either require a full clean (aka ignoring the flag change) or rebuild everything.
A modern build system by default should be able to provide utilities to make sure your build is correct and reproducible. Those features need to be coded into your Makefile and aren't available easily or by default.
If you need to code those rules, then you don't have a build system, just a way to run commands and possibly add some (usually poor) logic around them.
A typical example that most handcrafted Makefiles will fail is that changing a build flag should only rebuild the files affected. Most will either require a full clean (aka ignoring the flag change) or rebuild everything.
> A typical example that most handcrafted Makefiles will fail is that changing a build flag should only rebuild the files affected.
If you change a build flag, how aren't all the files affected?
If you change a build flag, how aren't all the files affected?
I've noticed a plethora of build options over the past decades. I keep hearing how they are all "better" than make.
Then to build one particular bit of software, there is a dependency upon a particular version of cmake, which you don't have.
Or you have to upgrade Maven. Or install some software you do not have by default on your environment.
I've never had to do that with make.
I've been told that other tools are better because the are simpler ... that is ... until you run into one of those aforementioned compatibility issues with the build tool.
This isn't an infrequent occurrence for me, I see lots of hardwiring for various packages that I don't have installed by default in some projects.
Then you get the gophers/rust folk, and their need to run on the most recently released kit. With all its undiscovered bugs. The gophers want to replace make with mage ... well, to be more correct, they want to replace every non-go based tool with one built in go, because ... go. A modern version of the insufferable pythonistas of years past.
Throughout this, I've been using and building with Make just fine. My make files do the heavy lifting in my projects. And I get a good laugh each time I hear "oh I have to fix my magefile/CMakeFile.txt/yadda yadda yadda".
As the great philosopher Inigo Montoya once opined ... you keep using that word [better]. I do not think it means what you think it means.
Then to build one particular bit of software, there is a dependency upon a particular version of cmake, which you don't have.
Or you have to upgrade Maven. Or install some software you do not have by default on your environment.
I've never had to do that with make.
I've been told that other tools are better because the are simpler ... that is ... until you run into one of those aforementioned compatibility issues with the build tool.
This isn't an infrequent occurrence for me, I see lots of hardwiring for various packages that I don't have installed by default in some projects.
Then you get the gophers/rust folk, and their need to run on the most recently released kit. With all its undiscovered bugs. The gophers want to replace make with mage ... well, to be more correct, they want to replace every non-go based tool with one built in go, because ... go. A modern version of the insufferable pythonistas of years past.
Throughout this, I've been using and building with Make just fine. My make files do the heavy lifting in my projects. And I get a good laugh each time I hear "oh I have to fix my magefile/CMakeFile.txt/yadda yadda yadda".
As the great philosopher Inigo Montoya once opined ... you keep using that word [better]. I do not think it means what you think it means.
Usability.
I find gnu make very usable and very clear on its purpose. Sometimes I have used cmake and it was uglier, more complicated, and harder to understand. What would you suggest as a modern, usable build system?
> Sometimes I have used cmake and it was uglier, more complicated, and harder to understand.
okay, please provide a makefile which:
* detects if, say, libpng is available
* compile a library, an executable and a unit test if that's the case
* links the executable to the library and the library to libpng
* does so for mac, windows, linux, creates a .app for mac, an installer for windows, a .deb package for linux
* can be used from IDEs, with IDEs understanding where your code has to be fetched, what the includes are, etc
here's more-or-less the CMake version :
okay, please provide a makefile which:
* detects if, say, libpng is available
* compile a library, an executable and a unit test if that's the case
* links the executable to the library and the library to libpng
* does so for mac, windows, linux, creates a .app for mac, an installer for windows, a .deb package for linux
* can be used from IDEs, with IDEs understanding where your code has to be fetched, what the includes are, etc
here's more-or-less the CMake version :
project(myapp)
find_package(PNG REQUIRED)
add_library(mylib lib.c)
add_executable(myapp MACOS_BUNDLE WIN32 app.c)
target_link_libraries(mylib PRIVATE PNG::PNG)
target_link_libraries(myapp PRIVATE mylib)
install(
TARGETS myapp mylib
RUNTIME DESTINATION bin
ARCHIVE DESTINATION lib)
include(CPack)
> * detects if, say, libpng is available
You just compile with -lpng. If libpng is available, it will work, otherwise it will fail with a clear error message. What else do you need?
> * links the executable to the library and the library to libpng
The .a is useless often, but there you have it just in case. Otherwise, just remove all references to $(LIB) and get a shorter makefile.
> you can't seriously say that this (which covers an immense part of software project needs) is easier to do in make, right ?
This is rather subjective. In my case the makefile seems easier. It will work for all unices, but probably not for windows, and will not create an "app" package, your are right about that.
But the makefile covers indeed all my personal needs, and it is much more powerful and easier to use than the cmake/make combo. For example, you can test multiple compilers and compiler options with a shell loop:
You just compile with -lpng. If libpng is available, it will work, otherwise it will fail with a clear error message. What else do you need?
> * links the executable to the library and the library to libpng
LDLIBS = -lpng
BIN = foo bar baz
OBJ = lib1.o lib2.o lib3.o
LIB = libmine.a
default : $(BIN) $(LIB)
$(BIN) : $(OBJ)
$(LIB) : $(LIB)($(OBJ))
clean : ; $(RM) $(BIN) $(LIB) $(OBJ)
check : $(BIN) ; ./foo -test
The .a is useless often, but there you have it just in case. Otherwise, just remove all references to $(LIB) and get a shorter makefile.
> you can't seriously say that this (which covers an immense part of software project needs) is easier to do in make, right ?
This is rather subjective. In my case the makefile seems easier. It will work for all unices, but probably not for windows, and will not create an "app" package, your are right about that.
But the makefile covers indeed all my personal needs, and it is much more powerful and easier to use than the cmake/make combo. For example, you can test multiple compilers and compiler options with a shell loop:
# test all compilers in debug and release modes
for i in gcc clang tcc icc; do
for m in "-O3 -DNDEBUG" "-g"; do
make clean check CC=$i CFLAGS=$m
done
done
> You just compile with -lpng. If libpng is available, it will work, otherwise it will fail with a clear error message. What else do you need?
well, no you don't "just compile", because one of the most used compilers out there does not support "-l". Also because you may have multiple libpng versions on your machine - debug, with special testing flags, in your ~/work/libpng_build, etc.
Also, you may have libpng.so but not the headers. You may be compiling on Nix or GoboLinux or whatever system which does not respect the FHS. It won't work as is on OS X because you have to add -L/usr/local/lib. Users won't know what to do when they see whatever missing include error the compiler spits out.
> doing that in cmake would be a nightmare, wouldn't it?
.. it would be mostly the same than your example ? just tested and this works fine (normal practice with cmake is to build outside of src which helps):
well, no you don't "just compile", because one of the most used compilers out there does not support "-l". Also because you may have multiple libpng versions on your machine - debug, with special testing flags, in your ~/work/libpng_build, etc.
Also, you may have libpng.so but not the headers. You may be compiling on Nix or GoboLinux or whatever system which does not respect the FHS. It won't work as is on OS X because you have to add -L/usr/local/lib. Users won't know what to do when they see whatever missing include error the compiler spits out.
> doing that in cmake would be a nightmare, wouldn't it?
.. it would be mostly the same than your example ? just tested and this works fine (normal practice with cmake is to build outside of src which helps):
for i in gcc clang; do
for m in "-DCMAKE_BUILD_TYPE=Debug" "-DCMAKE_BUILD_TYPE=Release"; do #not using -g because that's something compiler-specific
rm -rf ** ; CC=$i cmake ../ $m ; cmake --build .
done
doneSpare your fingers ;) - you'll never convince anybody of the value of cmake by telling them about it. You just have to put them in a situation where they're looking after a program that has to build on multiple desktop platforms, and they'll figure it out eventually...
(The combination I've always had to deal with is Linux, OS X and Windows/VC++ - but any combination that includes both Windows/VC++ and a POSIX-type OS is probably enough to make it worthwhile. The POSIX/non-POSIX differences are rather annoying to deal with otherwise, and that quite apart from how most Windows programmers will quite reasonably want to use a Visual Studio project rather than a Makefile.)
506c
(The combination I've always had to deal with is Linux, OS X and Windows/VC++ - but any combination that includes both Windows/VC++ and a POSIX-type OS is probably enough to make it worthwhile. The POSIX/non-POSIX differences are rather annoying to deal with otherwise, and that quite apart from how most Windows programmers will quite reasonably want to use a Visual Studio project rather than a Makefile.)
506c
Yeah, CMake and autotools solve different problems than make. I'm definitely not a fan of either one but if I need to really distribute and compile something on more than one OS there isn't much of a choice.
The posted makefile example fails right away on multiple platforms since headers are often in somewhere else than in /usr/include directly, for example in Debian the right (atm) path would be /usr/include/libpng16 and Makefile does not handle that.
The posted makefile example fails right away on multiple platforms since headers are often in somewhere else than in /usr/include directly, for example in Debian the right (atm) path would be /usr/include/libpng16 and Makefile does not handle that.
> The posted makefile example fails right away on multiple platforms since headers are often in somewhere else
And that's the intended behavior. Finding system headers around your disk is not a problem that a build system must try to solve. This is the task of a distribution or a packaging system, which is a different problem as you say.
And that's the intended behavior. Finding system headers around your disk is not a problem that a build system must try to solve. This is the task of a distribution or a packaging system, which is a different problem as you say.
Yeah I agree, it's not the ideal situation but it is the real world. Simple makefiles do not work often and there are systems that can solve the problem. Not elegant, brilliantly designed tools but ones that work anyway.
[deleted]
I hope this day never arrives for me... In any case, if I ever had to compile something for windows I would rather install cygwin and a posix toolchain than cmake and visual studio.
Thanks for your answer, I see that cmake is not as cumbersome as I thought.
> Also, you may have libpng.so but not the headers.
Then how on earth are you supposed to compile it? You write the definitions verbatim on your code?
> It won't work as is on OS X because you have to add -L/usr/local/lib.
For that case it is better to not mess with compiler options and set up the compiling environment so that -lpng works, e.g. by setting LIBRARY_PATH=/usr/local/lib, and similarly for C_INCLUDE_PATH. These variables are recognized by all unix linkers and compilers that I care about.
Of course, this attitude may not be appropriate for everyone. Yet, I am in the happy position to be able to say say "this program requires gcc, clang, tcc or icc to be compiled, otherwise, please edit the makefile to suit your needs". Are there really other C compilers around? (notice that visual studio is not a C compiler, and cannot compile modern C code, so it does not enter into the discussion if you are a C programmer).
> Also, you may have libpng.so but not the headers.
Then how on earth are you supposed to compile it? You write the definitions verbatim on your code?
> It won't work as is on OS X because you have to add -L/usr/local/lib.
For that case it is better to not mess with compiler options and set up the compiling environment so that -lpng works, e.g. by setting LIBRARY_PATH=/usr/local/lib, and similarly for C_INCLUDE_PATH. These variables are recognized by all unix linkers and compilers that I care about.
Of course, this attitude may not be appropriate for everyone. Yet, I am in the happy position to be able to say say "this program requires gcc, clang, tcc or icc to be compiled, otherwise, please edit the makefile to suit your needs". Are there really other C compilers around? (notice that visual studio is not a C compiler, and cannot compile modern C code, so it does not enter into the discussion if you are a C programmer).
Yes, quite a few, mainframes, embedded OSes, bare metal deployments for SoC, surviving commercial UNIXes.
> Then how on earth are you supposed to compile it? You write the definitions verbatim on your code?
what I mean is that an user might download your source archive, run `make`, and have hard-to-understand failures because of this.
what I mean is that an user might download your source archive, run `make`, and have hard-to-understand failures because of this.
For example it checks the last modified time on all your files, every build. This doesn’t scale at a certain point. More modern build systems have better ways of doing this.
How to modern build systems check for file updates?
I can't speak for other build systems, but tup [0] provides the option to listen for filechanges via the fuse library.
[0] http://gittup.org/tup/make_vs_tup.html <edit substitutes the make comparison link instead of the main page>
[0] http://gittup.org/tup/make_vs_tup.html <edit substitutes the make comparison link instead of the main page>
Meson is a joy to use.
Ever tried Premake?
Long time ago, I thought it is not that actively maintained as before.
Another great thing about FreeBSD is it's not too hard going the other way: Embedding FreeBSD is a robust choice, one that can take you quickly to market as it did for me and a GPS on golf cart project I did in the late 90s with it, busybox, and some valuable help from PHK on getting support for the weirdo LCD panel the suits picked (we had to turn the panel upside down in the cart, the viewing angle was atrocious).
"It is easier to port a shell than a shell script."
This is gold!
This is gold!
I wish more VPS providers offered OpenBSD and FreeBSD. I remember using FreeBSD a lot when I was 12-17 just messing around and setting up servers, etc but eventually moved to Debian because VPS' at the time didn't really offer these that much (reliably.)
now you should try openbsd or netbsd, as they are even less tolerant of linux-isms
Given that Linux is the de facto standard, wouldn't it be more correct to describe the BSDisms as idiosyncrasies?
Which distro? Which libc? Which minimum kernel version? Which minimum version of GNU Make? Which specific version of GCC or clang?
Linux isn't a static environment. The closest thing to static is POSIX, and by and large every Unix environment, including various Linux environments as well as BSD environments, asymptotically approach POSIX compliance.
If you want to maintain a project long-term (5+ years), attempting to minimize deviations from POSIX is a solid strategy in terms of avoiding bit rot and long-term maintenance burdens. Sometimes it's unavoidable, but the point is to weigh your options carefully and don't undervalue the benefit of POSIX compliance. If you want to maximize the chance that your software will compile and execute correctly on a Linux distro 5+ years from now, a good first step is making sure it builds and runs natively on some of the BSDs.
There's a trend to discount the value of POSIX and to even actively break POSIX compliance in a way that makes it very difficult to support both POSIX semantics and some feature du jure. Developers should be wary of those who do this because if such projects and their maintainers don't see any value in POSIX they're not likely to see value in the long-term maintenance and stability of whatever feature they're selling you on today.
Else thread somebody says that they see Linux as the new SunOS. I agree.[1] But where is SunOS today?
[1] Except that Sun put much more emphasis on backwards compatibility. The Linux kernel developers emphasize backward compatibility, but the distros don't, and if the distros rip something out then inevitably it gets removed from upstream. OTOH, while the kernel developers rightfully have said they won't compromise sane design for POSIX, it turns out that such situations have been exceedingly rare and in general they take POSIX compliance seriously.
Linux isn't a static environment. The closest thing to static is POSIX, and by and large every Unix environment, including various Linux environments as well as BSD environments, asymptotically approach POSIX compliance.
If you want to maintain a project long-term (5+ years), attempting to minimize deviations from POSIX is a solid strategy in terms of avoiding bit rot and long-term maintenance burdens. Sometimes it's unavoidable, but the point is to weigh your options carefully and don't undervalue the benefit of POSIX compliance. If you want to maximize the chance that your software will compile and execute correctly on a Linux distro 5+ years from now, a good first step is making sure it builds and runs natively on some of the BSDs.
There's a trend to discount the value of POSIX and to even actively break POSIX compliance in a way that makes it very difficult to support both POSIX semantics and some feature du jure. Developers should be wary of those who do this because if such projects and their maintainers don't see any value in POSIX they're not likely to see value in the long-term maintenance and stability of whatever feature they're selling you on today.
Else thread somebody says that they see Linux as the new SunOS. I agree.[1] But where is SunOS today?
[1] Except that Sun put much more emphasis on backwards compatibility. The Linux kernel developers emphasize backward compatibility, but the distros don't, and if the distros rip something out then inevitably it gets removed from upstream. OTOH, while the kernel developers rightfully have said they won't compromise sane design for POSIX, it turns out that such situations have been exceedingly rare and in general they take POSIX compliance seriously.
maintainers don't see any value in POSIX
The problem is that POSIX only provides a tiny subset of the API surface that an application might need today. Just to give two random examples: how can an application inhibit system sleep or use a webcam? The best one often can do is rely on some widely-used library and hope that they (will continue to) support multiple platforms.
But where is SunOS today?
Comparing apples and oranges. For a large part of its lifetime, SunOS was a proprietary UNIX that only worked with Sun hardware. If we include SunOS 5/Solaris, they open sourced Solaris way after Linux already had a stronghold in the market.
Linux is now so dominant in mobile phones, servers, network devices, and IoT, that even it will take much longer to disappear than SunOS.
The problem is that POSIX only provides a tiny subset of the API surface that an application might need today. Just to give two random examples: how can an application inhibit system sleep or use a webcam? The best one often can do is rely on some widely-used library and hope that they (will continue to) support multiple platforms.
But where is SunOS today?
Comparing apples and oranges. For a large part of its lifetime, SunOS was a proprietary UNIX that only worked with Sun hardware. If we include SunOS 5/Solaris, they open sourced Solaris way after Linux already had a stronghold in the market.
Linux is now so dominant in mobile phones, servers, network devices, and IoT, that even it will take much longer to disappear than SunOS.
The other problem is that POSIX is horrible. See eg:
- http://www.daemonology.net/blog/2011-12-17-POSIX-close-is-br... It's impossible to reliably close a file if you have active signal handlers that might be triggered.
- https://www.sqlite.org/src/artifact/c230a7a24?ln=994-1081 (discussed at https://news.ycombinator.com/item?id=17601581 "POSIX advisory locks are broken by design.")
- Seemingly anything to do with terminal devices or job control. (Allocating ptys, sending control sequences like ^D or ^Z, adding processes to a group that will all recieve terminal signals together, etc.)
- 'Basic' regular expressions (as in `echo baaar | grep 'a+' || echo WTF`).
- The existence of hard links as anything beyond a destination cache for symlinks.
Most operating systems have problems like these, but POSIX proceeds to enshrine them as unfixable 'features' rather than "DEPRECATED: DO NOT USE". (On the plus side, F_SETLK(W) and close seem likely to be fixed at some point and close doesn't tend to get maliciously exploited the way eg integer overflow does by C compilers.)
- http://www.daemonology.net/blog/2011-12-17-POSIX-close-is-br... It's impossible to reliably close a file if you have active signal handlers that might be triggered.
- https://www.sqlite.org/src/artifact/c230a7a24?ln=994-1081 (discussed at https://news.ycombinator.com/item?id=17601581 "POSIX advisory locks are broken by design.")
- Seemingly anything to do with terminal devices or job control. (Allocating ptys, sending control sequences like ^D or ^Z, adding processes to a group that will all recieve terminal signals together, etc.)
- 'Basic' regular expressions (as in `echo baaar | grep 'a+' || echo WTF`).
- The existence of hard links as anything beyond a destination cache for symlinks.
Most operating systems have problems like these, but POSIX proceeds to enshrine them as unfixable 'features' rather than "DEPRECATED: DO NOT USE". (On the plus side, F_SETLK(W) and close seem likely to be fixed at some point and close doesn't tend to get maliciously exploited the way eg integer overflow does by C compilers.)
Every implementation of close, except on HP/UX, will close the file descriptor before returning upon a signal. But POSIX has actually approved posix_close, which will guarantee the proper, desired semantics. See http://austingroupbugs.net/view.php?id=529
The semantics of POSIX advisory locks were inherited; they codified the most widely used existing implementation. The ergonomics suck, but they offer one distinct advantage: you can query the PID of the lock holder, something not possible when a lock can be inherited. This can be useful for PID file locks where you can query the PID directly rather than trying to write and read it from the PID file; there's no loaded gun lying around, although there's still the TOCTTOU race between querying and kill'ing. In any event, "file-private" advisory locks are likely to be standardized and are already implemented by both Linux and FreeBSD. See http://austingroupbugs.net/view.php?id=768 The semantics are not coincidentally very similar to BSD flock, and the types similar to POSIX. Which is noteworthy because it's an example of how paying attention to POSIX and portability issues can help you triangulate the evolution of Linux interfaces down the road.
It's not like Linux is without its warts. The kernel's implementation of setuid & friends is per thread. That's not only horribly inconvenient semantics for 99% of use cases, it's a security nightmare. And glibc and musl must implement an obscenely complex dance to get process-wide, atomic behavior. There are also many legacy Linux interfaces that the project is committed to supporting because of its commitment to backwards compatibility. I don't understand the logic of pointing out all the anachronisms of POSIX (e.g. advisory locks) while ignoring the anachronisms of Linux.
I never said POSIX was perfect, and I'm not arguing to only stick to POSIX. There's no substitute for using your own good judgment. I'm simply saying that too many people don't appreciate the benefits of POSIX and portability more generally, which are substantial. The benefits aren't just the standard itself or the ability to use an alternative OS like FreeBSD--if you didn't care before you're almost certainly never going to--but the fact that every Unix vendor tracks the standard and, to a lesser extent, avoids gratuitous incompatibilities with other POSIX environments. Standardization signals important information about the stability of an interface, and the process of standardization channels the evolution of interfaces long before they become standardized (if ever). POSIX and the BSDs matter to Linux development; and they should matter to Linux developers even if they'll never run code outside of a Linux environment.
It's also worth mentioning that Red Hat basically steers POSIX, now, judging by the Austin Group Bugs issue tracker. So while some teams are rather antagonistic toward POSIX (e.g. Poettering and systemd team), others put considerable effort into it.
The semantics of POSIX advisory locks were inherited; they codified the most widely used existing implementation. The ergonomics suck, but they offer one distinct advantage: you can query the PID of the lock holder, something not possible when a lock can be inherited. This can be useful for PID file locks where you can query the PID directly rather than trying to write and read it from the PID file; there's no loaded gun lying around, although there's still the TOCTTOU race between querying and kill'ing. In any event, "file-private" advisory locks are likely to be standardized and are already implemented by both Linux and FreeBSD. See http://austingroupbugs.net/view.php?id=768 The semantics are not coincidentally very similar to BSD flock, and the types similar to POSIX. Which is noteworthy because it's an example of how paying attention to POSIX and portability issues can help you triangulate the evolution of Linux interfaces down the road.
It's not like Linux is without its warts. The kernel's implementation of setuid & friends is per thread. That's not only horribly inconvenient semantics for 99% of use cases, it's a security nightmare. And glibc and musl must implement an obscenely complex dance to get process-wide, atomic behavior. There are also many legacy Linux interfaces that the project is committed to supporting because of its commitment to backwards compatibility. I don't understand the logic of pointing out all the anachronisms of POSIX (e.g. advisory locks) while ignoring the anachronisms of Linux.
I never said POSIX was perfect, and I'm not arguing to only stick to POSIX. There's no substitute for using your own good judgment. I'm simply saying that too many people don't appreciate the benefits of POSIX and portability more generally, which are substantial. The benefits aren't just the standard itself or the ability to use an alternative OS like FreeBSD--if you didn't care before you're almost certainly never going to--but the fact that every Unix vendor tracks the standard and, to a lesser extent, avoids gratuitous incompatibilities with other POSIX environments. Standardization signals important information about the stability of an interface, and the process of standardization channels the evolution of interfaces long before they become standardized (if ever). POSIX and the BSDs matter to Linux development; and they should matter to Linux developers even if they'll never run code outside of a Linux environment.
It's also worth mentioning that Red Hat basically steers POSIX, now, judging by the Austin Group Bugs issue tracker. So while some teams are rather antagonistic toward POSIX (e.g. Poettering and systemd team), others put considerable effort into it.
> If you want to maximize the chance that your software will compile and execute correctly on a Linux distro 5+ years from now, a good first step is making sure it builds and runs natively on some of the BSDs.
Like everything, this is a trade-off: what you are advocating is that people use a more or less frozen sub-set of features, and avoid improvements that are already de-facto standard.
I looked at what would happen if my employer standardised shell scripting on "sh", rather than "bash", and quickly concluded that it was a bad tradeoff for us: Bash is, has has been, the standard shell in every place that we would want to run non-trivial shell scripts: if we ran Alpine Linux containers, we could just treat those as a special case.
Like everything, this is a trade-off: what you are advocating is that people use a more or less frozen sub-set of features, and avoid improvements that are already de-facto standard.
I looked at what would happen if my employer standardised shell scripting on "sh", rather than "bash", and quickly concluded that it was a bad tradeoff for us: Bash is, has has been, the standard shell in every place that we would want to run non-trivial shell scripts: if we ran Alpine Linux containers, we could just treat those as a special case.
Using bash instead of sh is ok enough (some people won't agree), but what's bad is writing bash script and using #!/bin/sh
Linux may be the de facto standard, but BSD cares about the de jure standard. It's a matter of perspective.
Also many of these are actually GNU idiosyncrasies, and from it's name GNU suggests that there is a standard, and they are going to be different.
Also many of these are actually GNU idiosyncrasies, and from it's name GNU suggests that there is a standard, and they are going to be different.
Fun to think it's come full circle; BSD used to be the rule-breakers.
Isn't that the history of computing? the word UNIX points at the disdain its makers had for Multics. They broke every rule IBM had.
PCs were these toys for people who had no acces to real computers, until they were the standard. Microsoft provided toy OSes until they wiped out almost anybody else. Java, MySql and subversion are other examples.
I've read somewhere that every succesfull product has a phase where the world laughs at it. If that hadn't happened, it would have been smothered in the cradle by the behemots of its time. Being laughed at gave it time to grow.
PCs were these toys for people who had no acces to real computers, until they were the standard. Microsoft provided toy OSes until they wiped out almost anybody else. Java, MySql and subversion are other examples.
I've read somewhere that every succesfull product has a phase where the world laughs at it. If that hadn't happened, it would have been smothered in the cradle by the behemots of its time. Being laughed at gave it time to grow.
They break the rule these days by actually giving a shit about security instead of doing what many of us are guilty of doing: dumbing it down for the lowest common denominator user to make money.
Has created some incredible UI paradigms and elements, but...
Has created some incredible UI paradigms and elements, but...
I think of linux as the new SunOS 4.x. I remember back in the day when random crap you got would work fine on SunOS, but was a pain on HP-UX, AIX, or whatever other weird unix you had around. Now linux is like SunOS 4.x was back then.
That's a totally weird way to think of it, since up to SunOS 4.x, it was mostly based on BSD, and afterwards, after being renamed to Solaris, on SysV.
It's not usually seen that way because the BSDs are considered to be directly descended from the original UNIX codebase and GNU and Linux are clones.
The analogy isn't used correctly : BSD was initially a codebase clone of UNIX, in other words, BSD is a fork.
GNU/Linux was initially developed from scratch to behave like UNIX to a certain extent (hence the name "Unix-like"). In other words, it's a rewrite inspired from a model.
GNU/Linux was initially developed from scratch to behave like UNIX to a certain extent (hence the name "Unix-like"). In other words, it's a rewrite inspired from a model.
"BSD is what you get when a bunch of Unix hackers sit down to try to port a Unix system to the PC. Linux is what you get when a bunch of PC hackers sit down and try to write a Unix system for the PC."
UNIX is a trademark, so you may not call something UNIX unless you are allowed to use that trademark (which costs several 100k USDs to get certified). Apple paid that price for macOS, so (certain versions of) macOS are UNIX.
macOS has a marketshare in server market segment of around 0% (client of around 5%, iOS substantial though), and Linux is the defacto standard in that area. Which OS do people learn nowadays? Linux distributions. So it makes sense to see it from that PoV. Because chances are that you're going to use that defacto standard. And even if you prefer UNIX(tm) or BSD or Solaris you'd rather use Linux than Windows or OpenVMS. Cause the latter are aliens from outer space.
macOS has a marketshare in server market segment of around 0% (client of around 5%, iOS substantial though), and Linux is the defacto standard in that area. Which OS do people learn nowadays? Linux distributions. So it makes sense to see it from that PoV. Because chances are that you're going to use that defacto standard. And even if you prefer UNIX(tm) or BSD or Solaris you'd rather use Linux than Windows or OpenVMS. Cause the latter are aliens from outer space.
As a purely technical note, there is a Linux which is a UNIX: Inspur K-UX.
More philosophically, insisting on a definition of UNIX which includes z/OS and excludes Seventh Edition Unix isn't going to win you many friends.
http://www.opengroup.org/csq/repository/norationale=1&norefe...
And speaking of aliens from outer space, z/OS is a JCL-eating Martian speaking EBCDIC through its virtual cardpunch. ;)
More philosophically, insisting on a definition of UNIX which includes z/OS and excludes Seventh Edition Unix isn't going to win you many friends.
http://www.opengroup.org/csq/repository/norationale=1&norefe...
And speaking of aliens from outer space, z/OS is a JCL-eating Martian speaking EBCDIC through its virtual cardpunch. ;)
We've written about certified UNIX wrt macOS vs. Linux here - https://github.com/shlomif/Freenode-programming-channel-FAQ/... .
Aliens have something to teach us. The current OS monoculture is not conducive to innovation.
[deleted]
Is there a jquery equivalent for shell scripting?
Quoting a quote from the article:
> Perl is portable shell
Which ironically I do use in this way but never formulated it in such a compact and witty way.
I found Perl to be available basically by default on every platform I could encounter, from AIX 3 to Windows (it comes along with Git for Windows, which gives you bash too).
> Perl is portable shell
Which ironically I do use in this way but never formulated it in such a compact and witty way.
I found Perl to be available basically by default on every platform I could encounter, from AIX 3 to Windows (it comes along with Git for Windows, which gives you bash too).
FreeBSD is great! Even as a desktop. NVIDIA builds official drivers for FreeBSD. I dont know whats the status for newer AMD graphics card. Nvidia + Intel graphics also works very well.
There is a port system where users can build the packages from sourcecode on their own machines. Its +- like gentoo.
There is also a binary pkg system. It is faster to install since the packages are already built (binary).
To upgrade from minor versions ie. 11.1 to 11.2 it is very simple and well documented. Just see in the manual freebsd-update
There is native support for ZFS also which is great.
Virtualbox works very well, so its possible to install other OSes in virtualbox. VMware i dont think is supported.
There is a port system where users can build the packages from sourcecode on their own machines. Its +- like gentoo.
There is also a binary pkg system. It is faster to install since the packages are already built (binary).
To upgrade from minor versions ie. 11.1 to 11.2 it is very simple and well documented. Just see in the manual freebsd-update
There is native support for ZFS also which is great.
Virtualbox works very well, so its possible to install other OSes in virtualbox. VMware i dont think is supported.
> FreeBSD’s grep does not have a "-P" flag by default
(I continue to wonder what's stopping everyone from just ditching `grep` altogether and switching to `ag`/`rg`)
(I continue to wonder what's stopping everyone from just ditching `grep` altogether and switching to `ag`/`rg`)
grep is standardized.
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/gr...
Using ag et al in a script makes your script non-portable, which is the opposite of the goal here.
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/gr...
Using ag et al in a script makes your script non-portable, which is the opposite of the goal here.
At the very least ag does not handle large files. It bails with an error after the file gets too big.
It can crash, on large files, particularly on macOS because mmap is disabled by default.
But that should go away by streaming the file. e.g.
But that should go away by streaming the file. e.g.
cat haystack | ag 'Needle' ag Needle < haystack
Your version requires fewer keystrokes. It's also mental model-breaking and un-Unixy. Pipes, my good man! Pipe everything to everything!
How come knowing how to redirect stdin is un-Unixy?
Pipes are a good tool, and so are many others. Having a good command of other basic mechanisms of the shell is going to hugely pay off, try to have a brief shot if you want [0].
[0] https://www.gnu.org/software/bash/manual/html_node/Redirecti...
Pipes are a good tool, and so are many others. Having a good command of other basic mechanisms of the shell is going to hugely pay off, try to have a brief shot if you want [0].
[0] https://www.gnu.org/software/bash/manual/html_node/Redirecti...
No. Useless Use of Cat is sufficiently unidiomatic that there is an award for it:
http://porkmail.org/era/unix/award.html
http://porkmail.org/era/unix/award.html
No. Being smug doesn't make someone right. Especially when the smug person is advocating a special case. Special cases are inherently wrong, even when they "save" a process.
If the most logical syntax is
grep needle haystack grep needle < haystack cat haystack | grep needleMm, input redirection with the shell is the very epitomy of basic Unix functionality.
< haystack ag Needle
Will work too, and now it's in the logical order.
Will work too, and now it's in the logical order.
Why? They're not compatible and not a drop-in replacement.
They're also likely a little bit slower, especially on large datasets. It's really hard to beat GNU grep. (Which is famously 3 cycles/byte in the trivial case. Great exercise though.) But for the most part performance is secondary to feature partity.
They're also likely a little bit slower, especially on large datasets. It's really hard to beat GNU grep. (Which is famously 3 cycles/byte in the trivial case. Great exercise though.) But for the most part performance is secondary to feature partity.
FWIW rg is faster than gnu grep https://github.com/BurntSushi/ripgrep#quick-examples-compari...
Right, by "very hard to beat" I didn't mean to imply it was somehow impossible.
Looking at the whole benchmark that that page links to, the difference is within a few percent for the actual searching, but it is also the only software in the author's own benchmark that is even close. The other software that was mentioned in the comment above is some 5x worse.
Nobody is going to change the standard grep for performance reasons.
Looking at the whole benchmark that that page links to, the difference is within a few percent for the actual searching, but it is also the only software in the author's own benchmark that is even close. The other software that was mentioned in the comment above is some 5x worse.
Nobody is going to change the standard grep for performance reasons.
ripgrep author here.
It really depends on what you're searching. In some types of searches, GNU grep can get very very slow. The page you were linked to includes some benchmarks for that. In particular, cases where you're searching large amounts of non-ASCII (but UTF-8) text. There are some benchmarks for that case here: https://blog.burntsushi.net/ripgrep/ specifically: https://blog.burntsushi.net/ripgrep/#subtitles-no-literal
Also, it's probably not about "feature parity" per se, but standardization. In terms of feature parity, there is very little that grep can do that, say, ripgrep cannot. But there is quite a bit that ripgrep can do that grep cannot do, mostly by design.
"replacing" grep with ripgrep has at least a dual meaning, and it is the source of unending miscommunication. In the one sense, I know plenty of people that have replaced grep with ripgrep (or whatever) in daily usage. This is a perfectly valid thing to do. In another sense, nobody is going to replace grep with ripgrep in a distribution because they almost certainly do not want to stop being POSIX compatible. That is, asking to "just replace" grep with ripgrep is equivalent to asking distributors to stop being POSIX compatible in exchange for some other feature that ripgrep provides. That question is a non-starter, but a lot of people either don't realize that or are confusing what it means to "replace" something.
It really depends on what you're searching. In some types of searches, GNU grep can get very very slow. The page you were linked to includes some benchmarks for that. In particular, cases where you're searching large amounts of non-ASCII (but UTF-8) text. There are some benchmarks for that case here: https://blog.burntsushi.net/ripgrep/ specifically: https://blog.burntsushi.net/ripgrep/#subtitles-no-literal
Also, it's probably not about "feature parity" per se, but standardization. In terms of feature parity, there is very little that grep can do that, say, ripgrep cannot. But there is quite a bit that ripgrep can do that grep cannot do, mostly by design.
"replacing" grep with ripgrep has at least a dual meaning, and it is the source of unending miscommunication. In the one sense, I know plenty of people that have replaced grep with ripgrep (or whatever) in daily usage. This is a perfectly valid thing to do. In another sense, nobody is going to replace grep with ripgrep in a distribution because they almost certainly do not want to stop being POSIX compatible. That is, asking to "just replace" grep with ripgrep is equivalent to asking distributors to stop being POSIX compatible in exchange for some other feature that ripgrep provides. That question is a non-starter, but a lot of people either don't realize that or are confusing what it means to "replace" something.
Could ripgrep offer a POSIX compatible mode?
It "could" in the sense that it's of course technically possible. But it's a _ton_ of work, and it's not clear to me that it would actually be beneficial. GNU grep is doing just fine as far as I can tell at being a POSIX compatible grep.
ripgrep is split into libraries, so if someone wanted this, they could just go do it themselves. :-) It would actually be pretty easy to get something that smells like POSIX. But the details make up a very long tail, and some of those tasks are fairly significant (supporting BRE syntax, supporting POSIX-style alternations, supporting POSIX locales).
ripgrep is split into libraries, so if someone wanted this, they could just go do it themselves. :-) It would actually be pretty easy to get something that smells like POSIX. But the details make up a very long tail, and some of those tasks are fairly significant (supporting BRE syntax, supporting POSIX-style alternations, supporting POSIX locales).
> 3 cycles/byte in the trivial case
Shouldn't it be less than that in the best case, by means of Boyer–Moore?
Shouldn't it be less than that in the best case, by means of Boyer–Moore?
Per processed byte, IIRC.
grep is assumed to be installed everywhere.
ag and gmake and several GNU or linux-specific extensions not.
ag and gmake and several GNU or linux-specific extensions not.
-P being PCRE (Perl regex).
By ag, do you mean ack-grep (https://beyondgrep.com/install/)?
By ag, do you mean ack-grep (https://beyondgrep.com/install/)?
ag = The Silver Searcher https://github.com/ggreer/the_silver_searcher
They do different things. So we need both :)
[deleted]
>Perl is portable shell — Guy Keren
Which still has many of the terrible problems of the shell, because its original design was an imitation of several different shell scripting languages.
If all you know is Perl and you're incapable of learning any other languages, or for some perverse reason you like the way Perl scripts look just as incomprehensible as shell scripts, and you're absolutely sure nobody else will ever need to look at or maintain your code, then by all means use Perl. But if you don't hate the people who will have to clean up after you or decipher and maintain what you're done, then please use Python or JavaScript instead of Perl or shell scripts.
Which still has many of the terrible problems of the shell, because its original design was an imitation of several different shell scripting languages.
If all you know is Perl and you're incapable of learning any other languages, or for some perverse reason you like the way Perl scripts look just as incomprehensible as shell scripts, and you're absolutely sure nobody else will ever need to look at or maintain your code, then by all means use Perl. But if you don't hate the people who will have to clean up after you or decipher and maintain what you're done, then please use Python or JavaScript instead of Perl or shell scripts.
> and you're absolutely sure nobody else will ever need to look at or maintain your code
By "nobody else" you really mean "other people with the same mindset who aren't willing to invest some time in getting used to perl syntax". Thing is that modern javascript is just as incomprehensible to sysops/devops people who're not familiar with the new JS syntax. You come from one world and think that's the "normal", they come from the other and expect that everyone already knows perl as it is half bash, half C. And in the end it's really much more about how much time did you spend working in it, than in one language being so much clearer than the other.
By "nobody else" you really mean "other people with the same mindset who aren't willing to invest some time in getting used to perl syntax". Thing is that modern javascript is just as incomprehensible to sysops/devops people who're not familiar with the new JS syntax. You come from one world and think that's the "normal", they come from the other and expect that everyone already knows perl as it is half bash, half C. And in the end it's really much more about how much time did you spend working in it, than in one language being so much clearer than the other.
Too many people have for too long treated Perl as way to blackmail their employers into not firing them, by developing an unmaintainable code base that nobody else can understand. Do you really want to be "that guy", or clean up after them yourself? (I say "that guy" because the Perl community is historically hostile to women, with the "hooters girls" at trade show booths, etc.)
http://geekfeminism.wikia.com/wiki/Randal_Schwartz
"He's the Hooter's guy, right?"
https://web.archive.org/web/20080328132126/https://www.oblom...
http://blogs.perl.org/users/joe_mcmahon1/2012/08/why-im-cons...
And have you ever tried to hire a competent experienced Perl programmer? They're extremely hard to find, and very expensive, and usually would rather be working with some other language. Sure, incompetent ones are a dime a dozen, but hiring those and setting them loose just starts the vicious cycle again.
All of the competent Perl programmers have long since been hired up by companies desperately trying to find people qualified to work on their old toxic legacy code bases that they're stuck with. Like Booking.com for example.
And writing new code in Perl is insane. There's absolutely nothing special about Perl 5 or Perl 6 that solves any problems you can't easily solve in most other languages.
Perl's much larger problems totally overwhelm the minor conveniences from its "syntactic syrup of ipecac" that perversely appeals those few people who think saving a few keystrokes at the expense of readability, instead of spelling words out with letters instead of line-noise punctuation and acronyms, is the sole goal of software development.
Perl 6 is a joke, a slow moving parody of itself that missed the boat decades ago, and it's absolutely never going to catch up with JavaScript or Python.
If you really want to optimize your career for programming toxic legacy code that's too ugly for anyone else to touch, you should have learned COBOL before the Y2K "crisis". But in the long term, you might have regretted it:
https://medium.com/@donhopkins/cobol-forever-1a49f7d28a39
http://geekfeminism.wikia.com/wiki/Randal_Schwartz
"He's the Hooter's guy, right?"
https://web.archive.org/web/20080328132126/https://www.oblom...
http://blogs.perl.org/users/joe_mcmahon1/2012/08/why-im-cons...
And have you ever tried to hire a competent experienced Perl programmer? They're extremely hard to find, and very expensive, and usually would rather be working with some other language. Sure, incompetent ones are a dime a dozen, but hiring those and setting them loose just starts the vicious cycle again.
All of the competent Perl programmers have long since been hired up by companies desperately trying to find people qualified to work on their old toxic legacy code bases that they're stuck with. Like Booking.com for example.
And writing new code in Perl is insane. There's absolutely nothing special about Perl 5 or Perl 6 that solves any problems you can't easily solve in most other languages.
Perl's much larger problems totally overwhelm the minor conveniences from its "syntactic syrup of ipecac" that perversely appeals those few people who think saving a few keystrokes at the expense of readability, instead of spelling words out with letters instead of line-noise punctuation and acronyms, is the sole goal of software development.
Perl 6 is a joke, a slow moving parody of itself that missed the boat decades ago, and it's absolutely never going to catch up with JavaScript or Python.
If you really want to optimize your career for programming toxic legacy code that's too ugly for anyone else to touch, you should have learned COBOL before the Y2K "crisis". But in the long term, you might have regretted it:
https://medium.com/@donhopkins/cobol-forever-1a49f7d28a39
Wow. Just wow. FUD. Pure and simple.
Perl is doing well, thank you. Perl6 is interesting, but I don't have a project for it yet. There are jobs in perl, there are needs, and we (perl devs) aren't appreciably more expensive than non-perl devs, though some of us may be better at negotiation than the dime-a-dozen developers in more "common" languages.
There is just so much that is wrong with the post above. Its actually sad.
Perl is doing well, thank you. Perl6 is interesting, but I don't have a project for it yet. There are jobs in perl, there are needs, and we (perl devs) aren't appreciably more expensive than non-perl devs, though some of us may be better at negotiation than the dime-a-dozen developers in more "common" languages.
There is just so much that is wrong with the post above. Its actually sad.
I can understand Python (it has a library in stdlib for running subprocess, and it is specially good after Python 3.4). However AFAIK JavaScript is bad for shell scripting. You need a external library to have sane subprocess call, unless you give up and allow command injection inside your scripts. Last time I looked even the alternatives in npm were not as good as what Python offered in stdlib.
Actually, I don't know many languages that have a good way to call subprocess in stdlib. Ruby's Open3 comes in mind, however the API is kind horrible.
Actually, I don't know many languages that have a good way to call subprocess in stdlib. Ruby's Open3 comes in mind, however the API is kind horrible.
You didn't think I meant running JavaScript in the browser, did you??!
Of course JavaScript has an external library for sane subprocess calls, and is perfectly fine and commonly used for writing process management and system administration scripts. Please google "node.js" and "npm".
[Edited for tone -- thanks mkesper.]
Of course JavaScript has an external library for sane subprocess calls, and is perfectly fine and commonly used for writing process management and system administration scripts. Please google "node.js" and "npm".
[Edited for tone -- thanks mkesper.]
Javascript and numerics. That is my takedown of javascript. If you don't know why its bad for numerics[1] or just generally weird [2][3] ...
No, JS shouldn't be used as a systems language IMO. It gives you enough rope to shoot yourself in the foot (multiple mixed metaphors for a reason).
[1] https://charlieharvey.org.uk/page/javascript_the_weird_parts
[2] https://hackernoon.com/js-wtf-with-math-79da9a941ec1
[3] https://github.com/denysdovhan/wtfjs
No, JS shouldn't be used as a systems language IMO. It gives you enough rope to shoot yourself in the foot (multiple mixed metaphors for a reason).
[1] https://charlieharvey.org.uk/page/javascript_the_weird_parts
[2] https://hackernoon.com/js-wtf-with-math-79da9a941ec1
[3] https://github.com/denysdovhan/wtfjs
Yeah, I was referring to Node.js. And no, what Node.js offers by default seems horrible[1]. It pass its entries to system shell, that is completely insane.
Python's subprocess actually run the process inside a Python process, so there is no possibility to hang youself up in case of a vulnerability in shell code. You can explicitly run the subprocess in shell script if you need shell capabilities (glob expansion comes in mind), however you shouldn't.
[1]: https://nodejs.org/api/child_process.html
Python's subprocess actually run the process inside a Python process, so there is no possibility to hang youself up in case of a vulnerability in shell code. You can explicitly run the subprocess in shell script if you need shell capabilities (glob expansion comes in mind), however you shouldn't.
[1]: https://nodejs.org/api/child_process.html
Doesn't seem that different to what Python offers? There's ways to use the shell, most options do not use it by default.
No need for that tone, right?
So we should instead write scripts for people that all they know is Python and Javascript and are incapable of learning any other languages, right?
Sorry but Perl is perfectly fine even by today’s standards.
Sorry but Perl is perfectly fine even by today’s standards.
Oh dear, we seem to have unearthed a pythonista.
Perl is still one of the best glue/high level systems languages around, partisan missives not withstanding. Debian's apt system is based upon it, as is git (though it uses C for parts that are intensive), and many other tools.
Python is fine as a systems language if you don't mind the insane indentation issues, and the extra boilerplate you need to do for "simple" things.
Honestly, I see Julia as the up and comer here. Julia has IMO the best syntax, is JIT compiled, is a real programming language, and is IMO python done right.
If you are stuck believing python is a good language to develop new projects in, great. Knock yourself out. I personally find myself most productive in Julia, Perl, C, Octave/Matlab, and yes, even Fortran.
Perl is still one of the best glue/high level systems languages around, partisan missives not withstanding. Debian's apt system is based upon it, as is git (though it uses C for parts that are intensive), and many other tools.
Python is fine as a systems language if you don't mind the insane indentation issues, and the extra boilerplate you need to do for "simple" things.
Honestly, I see Julia as the up and comer here. Julia has IMO the best syntax, is JIT compiled, is a real programming language, and is IMO python done right.
If you are stuck believing python is a good language to develop new projects in, great. Knock yourself out. I personally find myself most productive in Julia, Perl, C, Octave/Matlab, and yes, even Fortran.
> Honestly, I see Julia as the up and comer here. Julia has IMO the best syntax, is JIT compiled, is a real programming language, and is IMO python done right.
I think Julia is very exciting, and yes, the core language is very good. But so far I see very little uptake outside numerics. Hopefully it will get there.
I think Julia is very exciting, and yes, the core language is very good. But so far I see very little uptake outside numerics. Hopefully it will get there.
It's got the support in the core language. That's what is so exciting to me ... its not just numerics, its an excellent programming language in and of itself, before you even get to the interesting type system.
It is JIT compiled, though some of us who like to deploy apps would love to have an AoT compilation. Likely we could work around that with a singularity container on linux.
Unlike many other languages, parallelism is a first class citizen, be it SIMD, GPU, distributed machine, and combinations.
It also works very nicely on FBSD 11.x. I've tried compiling it on recent 12.x nightlies, but not had great luck.
I am actively looking to find projects to work on in it.
It is JIT compiled, though some of us who like to deploy apps would love to have an AoT compilation. Likely we could work around that with a singularity container on linux.
Unlike many other languages, parallelism is a first class citizen, be it SIMD, GPU, distributed machine, and combinations.
It also works very nicely on FBSD 11.x. I've tried compiling it on recent 12.x nightlies, but not had great luck.
I am actively looking to find projects to work on in it.
I haven't really seen enough Python code in my life to have an opinion about readability of Python code in the wild. But I have seen quite a lot of unreadable (unminimized) Javascript code.
In the end it really depends on how much the developer is suffering from some form of impostor syndrome: if they're not, you have a good chance they will create code that will be unmaintanable even for them in a few months time. Regardless of language.
In the end it really depends on how much the developer is suffering from some form of impostor syndrome: if they're not, you have a good chance they will create code that will be unmaintanable even for them in a few months time. Regardless of language.
Of course there are incompetent programmers in every language. But in absolute numbers, there are vastly more competent JavaScript programmers available to hire than competent Perl programmers. There really is no comparison.
Can you name any schools that teach Perl to undergrads on a non-trivial scale? Maybe there's a reason you can't. But I can name a lot of them that teach Python and JavaScript.
https://ocw.mit.edu/courses/electrical-engineering-and-compu...
And of course Python code is more readable and maintainable than Perl code, by its very design and nature, and also by its widespread culture and philosophy, which is 180 degrees opposite of Perl's philosophy and culture.
It's not just a matter of "There's more than one way to do it" -vs- "There should be one -- and preferably only one -- obvious way to do it." The Zen of Python goes much deeper than that.
https://www.python.org/dev/peps/pep-0020/
There's also the idea that instead of using arbitrary idiomatic and personalized punctuation and line noise in "more than one way" to accomplish the same task, you should simply use letters, by combining them to form words, connected by underscores or CamelCaseCaps to form coherent phrases, which have meanings, that communicate with the person reading the code, explaining the ideas and intentions behind the programmer who wrote the code. Subtle little things like that, you know.
Line noise and random punctuation do not make self documenting code. That's why it's traditionally used as a stand-in for censored obscenities, because it's meant to obscure its true meaning.
Can you name any schools that teach Perl to undergrads on a non-trivial scale? Maybe there's a reason you can't. But I can name a lot of them that teach Python and JavaScript.
https://ocw.mit.edu/courses/electrical-engineering-and-compu...
And of course Python code is more readable and maintainable than Perl code, by its very design and nature, and also by its widespread culture and philosophy, which is 180 degrees opposite of Perl's philosophy and culture.
It's not just a matter of "There's more than one way to do it" -vs- "There should be one -- and preferably only one -- obvious way to do it." The Zen of Python goes much deeper than that.
https://www.python.org/dev/peps/pep-0020/
There's also the idea that instead of using arbitrary idiomatic and personalized punctuation and line noise in "more than one way" to accomplish the same task, you should simply use letters, by combining them to form words, connected by underscores or CamelCaseCaps to form coherent phrases, which have meanings, that communicate with the person reading the code, explaining the ideas and intentions behind the programmer who wrote the code. Subtle little things like that, you know.
Line noise and random punctuation do not make self documenting code. That's why it's traditionally used as a stand-in for censored obscenities, because it's meant to obscure its true meaning.
Again, pure FUD. Once you bring the "line noise" mischaracterization up, you've blown your credibility.
I encountered this last in the mid oughts with insufferable Pythonistas. Seems not all of them have had time to think deeply as they gaze at go/rust/julia coming along to eat their lunch.
Perl continues to advance, people continue to start projects within it. Code continues to be contributed. CPAN (and CPAN6) continue to grow.
Quite sad.
I encountered this last in the mid oughts with insufferable Pythonistas. Seems not all of them have had time to think deeply as they gaze at go/rust/julia coming along to eat their lunch.
Perl continues to advance, people continue to start projects within it. Code continues to be contributed. CPAN (and CPAN6) continue to grow.
Quite sad.
You should also forbid your tools to access these URLs directly (--nonet), as it would be incredicly slow and you rather want your build to fail in this case instead of downloading on-demand.
If you prefer to work with files instead of passing long URLs on command line, use a small wrapper file to import the real stylesheet and pass that on command line instead of using the URL. The advantage is that you can also put additional <xsl:param> tags to customize the output right into this file.