MinUnit – A minimal unit testing framework for C (2002)(jera.com)
jera.com
MinUnit – A minimal unit testing framework for C (2002)
http://www.jera.com/techinfo/jtns/jtn002.html
55 comments
I actually implemented a basic unit test framework in C yesterday and started with basic asserts like you point out, however once I got it working I switched to return values not unlike TFA. The obvious problem with assertions in that they kill your test program immediately at the first failure instead of doing additional tests and potentially give a clearer picture of what's broken exactly. They're also not very flexible if you want to customize the error message since they only stringify the assert parameter. "Assertion `ret == 0' failed." is not super helpful. As sibling comments mention there are ways to work around that by using literal strings and && but it doesn't help if you want to wrap around tests using helper functions etc...
I'm also tempted to say that if you need to use GDB to figure out where your tests have failed exactly your framework is not particularly user friendly. Normally when a test fails it should be pretty clear where it failed and why. The root cause of the issue might be further away of course but then again assertions won't help with that either.
I'm also tempted to say that if you need to use GDB to figure out where your tests have failed exactly your framework is not particularly user friendly. Normally when a test fails it should be pretty clear where it failed and why. The root cause of the issue might be further away of course but then again assertions won't help with that either.
Assertions don't actually have to kill the program. They send an abort signal, which means you can catch them with a signal handler.
I similarly wrote a simple C testing framework for a little project I was working on, based on assertions. The framework uses signal handlers for aborts and segfaults, and then setjmp/longjmp to handle resuming the test suite at the next test case. This has the particularly nice effect of turning segfaults into (marked as such) test failures, instead of just terminating everything. It probably wouldn't be too hard to fit custom messages in too, but I hadn't felt the need yet.
I similarly wrote a simple C testing framework for a little project I was working on, based on assertions. The framework uses signal handlers for aborts and segfaults, and then setjmp/longjmp to handle resuming the test suite at the next test case. This has the particularly nice effect of turning segfaults into (marked as such) test failures, instead of just terminating everything. It probably wouldn't be too hard to fit custom messages in too, but I hadn't felt the need yet.
I concede that being able to handle segfaults as test failure is a very nice feature, however dealing with signal handlers is a bit more involved that simply using return values and I believe that it might cause portability issues (does it work on Windows for instance?).
I agree that for a decent test framework it might be the best approach but I think at this point we're no longer talking about "a minimal unit testing framework" which was the point of TFA.
I agree that for a decent test framework it might be the best approach but I think at this point we're no longer talking about "a minimal unit testing framework" which was the point of TFA.
It would be safer to fork before the segfault, to preclude other less obvious memory errors.
I mostly agree with you, but I would like to make something clear.
>I'm also tempted to say that if you need to use GDB to figure out where your tests have failed exactly your framework is not particularly user friendly.
Am I missing something?
>I'm also tempted to say that if you need to use GDB to figure out where your tests have failed exactly your framework is not particularly user friendly.
Am I missing something?
#include <assert.h>
int main(void)
{
assert(0);
return 0;
}
assert.bin: assert.c:5: int main(void): Assertion `0' failed.My point was that depending on how you structure your tests knowing the location of the assert might not help you, for instance you could have a helper function that would validate something and generate a failure if something goes wrong, in this case knowing that this specific function failed doesn't help much understanding what test specifically failed. My point was that if you need GDB to figure out what exactly failed then you don't have a very user friendly test framework.
Oh, I see now. Yeah, trying to make assert output a dynamically constructed message does seem like a PITA.
I was about the point this out!
At the end of the day, unit testing is just contract assertions, and contract assertions are usually conditions! But actually this isn't really the crux of unit testing, the value of unit testing is to iterate and refactor your program for a more predictable and bug-free application, not just writing a bunch of conditions.
One thing I'm pretty doctrinaire about when it comes to this sort of thing is printing out more than just a simple message. Quite often, this makes the problem obvious with no need for deeper investigation.
To do this I have a bunch of macros like this:
The FILE:LINE notation is probably clickable in your favourite text editor. (For VC++, use "FILE(LINE):". Just do #ifdef _MSC_VER or something.) Very convenient if you run tests as part of the build.
And you can flesh it out for strings, arrays, floats, doubles, and all the rest. You can fit everything you need into about 500 lines.
This isn't quite as impressive as the 3 lines here, but compared to something like Catch - which is a huge amount of C++ code, crazy C++ code to boot, that adds literally seconds to your build time - and, no, the fact that seconds is a drop in the ocean in C++land is not an excuse - it's in the same ballpark. At least, its extra utility should prove, over the course of a project, in my view, commensurate with the extra LOC.
To do this I have a bunch of macros like this:
/* check A and B are equal. */
#define EQ_II(A,B,M) (CheckEQII((A),(B),M,#A,#B,__FILE__,__LINE__))
EQ_II(i,3,"blah blah blah");
void CheckEQII(int64_t a,int64_t b,const char *message,const char *a_str,const char *b_str,const char *file,int64_t line) {
if(a!=b) {
printf("%s:%" PRId64 ": test failed: %s\n",file,line,message);
printf(" Values not equal.\n");
printf(" Got expr : %s\n",a_str);
printf(" Wanted expr : %s\n",b_str);
printf(" Got value : %" PRId64 " (0x%" PRIx64 ")\n");
printf(" Wanted value: %" PRId64 " (0x%" PRIx64 ")\n");
DEBUG_BREAK();
exit(1);
}
}
The FILE:LINE notation is probably clickable in your favourite text editor. (For VC++, use "FILE(LINE):". Just do #ifdef _MSC_VER or something.) Very convenient if you run tests as part of the build.
And you can flesh it out for strings, arrays, floats, doubles, and all the rest. You can fit everything you need into about 500 lines.
This isn't quite as impressive as the 3 lines here, but compared to something like Catch - which is a huge amount of C++ code, crazy C++ code to boot, that adds literally seconds to your build time - and, no, the fact that seconds is a drop in the ocean in C++land is not an excuse - it's in the same ballpark. At least, its extra utility should prove, over the course of a project, in my view, commensurate with the extra LOC.
That's why I love pytest.
You just `assert` an expression, and on failure it'll break down each sub-expression and show whatever it yielded, plus by default it captures all output (stdout and stderr) and prints it out only on failure (so logging and other console output is not painfully annoying while testing) and the fixtures systems is just fantastic and parameterised tests make for much clearer data-driven (/table-driven) tests and the tests selection at the CLI is awesome.
And all the complexity is in the test framework and runner, all you do is write
You just `assert` an expression, and on failure it'll break down each sub-expression and show whatever it yielded, plus by default it captures all output (stdout and stderr) and prints it out only on failure (so logging and other console output is not painfully annoying while testing) and the fixtures systems is just fantastic and parameterised tests make for much clearer data-driven (/table-driven) tests and the tests selection at the CLI is awesome.
And all the complexity is in the test framework and runner, all you do is write
def test_thing(a_fixture):
assert foo(bar) == a_fixture.some_thing()
On top of that, it'll compare complex structures to each other, so you could have a very large list of dicts that contain a custom data structure and it'll show you a diff and exactly where the mismatch is. Then it can auto-drop to the debugger and you can inspect previous variables to inspect the root cause, if not immediately apparent. Going back to C land, as much as I enjoy it, the tooling feels like the stone ages. I get that it's a lot harder for a language that compiles down to machine code, but dang.
Static reflection is one of my most looked-forward-to features in C++2x. All the information needed to introspect complex data structures is available to the compiler. We just need a way to access it.
> pretty doctrinaire about
And that is the problem.
I have met teams that have gone into analysis paralysis about which unit test framework to use.
I have met teams where one of the impediments to unit testing was "we don't have time to learn something that complicated, we don't understand why all this stuff is necessary".
Answer:
It isn't necessary. Get off your ass and start unit testing with the simplist framework possible.
It's more important, way way more important to just do it than fuss about the framework.
Like TFA.
Once you have actually done that for awhile, someone will say, "Wouldn't it be nice if..."
And _that's_ the time to introduce a framework. Or grow what you have.
On a C project I elected to grow, the nice thing about it is test setup and teardown? It's exactly the same thing as process set up and teardown.
So valgrind will tell you _exactly_ if anything leaked or was uninitialized!
And that is the problem.
I have met teams that have gone into analysis paralysis about which unit test framework to use.
I have met teams where one of the impediments to unit testing was "we don't have time to learn something that complicated, we don't understand why all this stuff is necessary".
Answer:
It isn't necessary. Get off your ass and start unit testing with the simplist framework possible.
It's more important, way way more important to just do it than fuss about the framework.
Like TFA.
Once you have actually done that for awhile, someone will say, "Wouldn't it be nice if..."
And _that's_ the time to introduce a framework. Or grow what you have.
On a C project I elected to grow, the nice thing about it is test setup and teardown? It's exactly the same thing as process set up and teardown.
So valgrind will tell you _exactly_ if anything leaked or was uninitialized!
Oops, missed this at the time. Rant on... though only for posterity.
There's various reasons why one might be doctrinaire about stuff. One common reason: you're an arsehole. Another one: you're an engineer, and this was what you were trained to do. A bonus third: you had it beaten into you by bitter experience.
Me? Well, I will say that I'm not clever enough to be an engineer.
There's various reasons why one might be doctrinaire about stuff. One common reason: you're an arsehole. Another one: you're an engineer, and this was what you were trained to do. A bonus third: you had it beaten into you by bitter experience.
Me? Well, I will say that I'm not clever enough to be an engineer.
So, the revelation is that a unit test is an if statement, and if you wrap that into a macro you can call it a framework.
For many novice engineers this is a huge revelation.
Really? Is it perhaps because of those libraries that do their best to make test call sites look like weird pseudo-English. Stuff like having a library function named it():
it(“should be confusing”, function() { ... })
it(“should be confusing”, function() { ... })
It always really bothered me that there had to be so much syntactical sugar on top of testing frameworks. Unit testing should be written in exactly the syntax of the language you are working with. I don't want to have to carry that additional crap in my working memory while writing code. I can't stand having to look up Gherkin syntax just to write a few dumb tests.
The only reason I see the usefulness of additional syntax just for tests is the case to have non-technical people writing tests. From my experience, this is a terrible idea.
The only reason I see the usefulness of additional syntax just for tests is the case to have non-technical people writing tests. From my experience, this is a terrible idea.
OTOH never rule out how KISS is the enemy of billable hours and don't underestimate CDD's ability (consultability driven development) to come up with solutions to non problems.
YMMV, the BDD implementations I've seen so far did not convince me of its value to put it mildly.
YMMV, the BDD implementations I've seen so far did not convince me of its value to put it mildly.
I never understood the value of Gherkin/Cucumber style testing until I worked with a QA engineer. The engineer started writing the tests (in Cucumber format) as a way to write __manual__ tests: they allow you to write 'do this, then that, verify this' in a structured manner. Automating this was a secondary goal: once you have (relatively) structured manual tests, it's nice if you can automate them, while still retaining the possibility to run them manually. This is actually important, and they would typically be end-to-end UI tests, which can be quite fragile when completely automated. Having them in a human-friendly format allows you to have a manual fallback if e.g. a changed identifier blocks you from running the automated test.
Yep. The value of Given When Then ends, not begins, once you have implemented the steps with some automation.
Oh man, I wonder how your opinions would change if you used a language with great first class testing. Elixir's unit testing framework is part of the language, and is excellent. https://hexdocs.pm/ex_unit/master/ExUnit.html
Its not really about syntactical sugar, and more about being able to see the data the failed, and make the testing experience easy so that you can write tests with less effort
Its not really about syntactical sugar, and more about being able to see the data the failed, and make the testing experience easy so that you can write tests with less effort
You're describing "fluent programming" ("it().shouldEqual(x)..."). Unfortunately it's not just unit testing frameworks where this nightmare exists. Plenty of production code abuses the builder pattern to have this "it's like reading English" stuff.
It's not really that terrible of a concept -- some of the earliest high-level languages (e.g., COBOL) were designed to be "readable" by less-technical staff. It just requires careful design and structure.
IMO, it only falls apart (and becomes such a nightmare) because of the imprecision of spoken & written word. Computers don't understand idioms, colloquialisms, and the like, and so non-technical staff are tricked into thinking a language/program can do more than it actually can do.
IMO, it only falls apart (and becomes such a nightmare) because of the imprecision of spoken & written word. Computers don't understand idioms, colloquialisms, and the like, and so non-technical staff are tricked into thinking a language/program can do more than it actually can do.
This sort of programming seems popular with languages that are already verbose (Java), and it just adds pointless extra verbosity.
Except that’s not a whole test any more than the punchline is the joke.
It’s sample data, verification, analysis, and reporting. The if statement is just the verification step.
It’s sample data, verification, analysis, and reporting. The if statement is just the verification step.
And any small set of functions is an API.
(2002) according to the HTTP headers:
$ curl -s --head http://www.jera.com/techinfo/jtns/jtn002.html | grep ^Last
Last-Modified: Fri, 06 Sep 2002 07:16:46 GMTI'm not very familiar with C and macros so it took me a while to realize that this line:
For anyone else like me, I put together the "inlined" version of the code to help me understand what is happening:
mu_assert("error, bar != 5", bar == 5);
For anyone else like me, I put together the "inlined" version of the code to help me understand what is happening:
#include <stdio.h>
#include "minunit.h"
int tests_run = 0;
int foo = 7;
static char * test_foo() {
do {
if (!(foo == 7))
return "error, foo != 7";
} while (0);
return 0;
}
static char * all_tests() {
do {
char *message = test_foo();
tests_run++;
if (message)
return message;
} while (0);
// ... more tests here
return 0;
}I have a library [link redacted] that was originally designed to extend MinUnit. I love the idea of an all-in-one unit testing library without complicated set up.
Well, there's a fine welcome from HN: new account that's fifteen minutes old, which posts what could be useful information (I just skimmed it) in a non-controversial manner and...insta-dead.
I obviously vouched for the comment, but yeesh.
I obviously vouched for the comment, but yeesh.
I hear you, but since we don't have software that can determine whether posts follow the guidelines, sometimes (especially for brand new accounts) the anti-abuse software gets it wrong so then we have vouching.
Yeah, I get what you're up against (I do skim New once in a while). Just a good thing someone has Show Dead on and has sufficient points to vouch, eh? :-)
Thank you :)
For everybody who demands from their test framework that it prints expected vs actual values and file/line references, and that it can be adapted to embedded and bare metal targets I recommend the excellent Unity framework[1], example[2].
--
[0] http://www.throwtheswitch.org/unity/
[1] https://gitlab.com/oliver117/blut/blob/master/test/test_math...
--
[0] http://www.throwtheswitch.org/unity/
[1] https://gitlab.com/oliver117/blut/blob/master/test/test_math...
And then in D it's built-in to the compiler so it's a 1 liner of sorts:
unittest { ... }
and they all run during compilation.
https://dlang.org/spec/unittest.html
unittest { ... }
and they all run during compilation.
https://dlang.org/spec/unittest.html
I'm surprised that isn't used in other languages, having the test right next to the code seems such a good idea.
Rust went this path too, there's something I love about having unittesting readily available. I can choose to use it or not use it. My asserts are available to me as I so desire. Python too sorta has some unittesting in the standard library, but that's not a compiler level thing.
what happens if you write a really slow test? is there some way to take advantage of the inbuilt console reporting, etc. without running them after every compile?
It's not compiled into your binary unless you want it to be. It's ran at compile time every time you compile, though I'm sure there's a flag to not run unit testing, and maybe with version() you might be able to block out the time ocnsuming tests. Not sure what you'd do in D that'd be so time consuming though.
Edit:
URL to version() docs:
https://dlang.org/spec/version.html
Edit:
URL to version() docs:
https://dlang.org/spec/version.html
Well, big minds meet.
Here is mine Minimal Unit Testing FW for C:
https://github.com/gon1332/minut
Here is mine Minimal Unit Testing FW for C:
https://github.com/gon1332/minut
Why stick to three lines? With only a couple more you could reduce the boiler-plate needed to use it.
I wrote a minimal C test framework too: https://github.com/codeplea/minctest It's got some real-world usage. Still only just one header file, but it'll time each test and if an equal assertion fails it'll print the variable values.
I wrote a minimal C test framework too: https://github.com/codeplea/minctest It's got some real-world usage. Still only just one header file, but it'll time each test and if an equal assertion fails it'll print the variable values.
Whats up with the do/while loops? Does that just make the return syntactically valid?
Apologies, its been a while since I've touched C/C++
Apologies, its been a while since I've touched C/C++
This is dealt with at the bottom of the article. It's a C FAQ: http://c-faq.com/cpp/multistmt.html
GCC has an extension that can be used instead of the do-while loop, see example macro definition here: https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html
I did something like that once in Classic ASP (VBScript). I think I only wrote two Assert functions and that was enough. The code ended up being really robust after I wrote a bunch of test cases that eventually passed (after writing the code of course).
I did mine using assert.h and a signal handler. More code, but advantage is you can also use assert.h internally anywhere in your program without it needing to know about the test framework. On other hand, makes testing optimized builds difficult.
FCTX is in a similar space, and quite good: https://github.com/imb/fctx/blob/master/README.rst
Libcheck does the nice thing and let's you optionally fork each test resulting in parallelism and test failures from segfaults and such not causing the whole suite failing. Does this do something similar?
The examples suggest that maybe the implementation should use “#str” in the macros (i.e. when you want your string to contain the same thing as the expression itself).
Works great in Embedded Linux, and it is easy to crosscompile.
Usage:
Output would be something like
If you run the application in a debugger like GDB, you can see the frame when the abort trap is called.