Author here. I didn't mention it because I wasn't writing an evaluation of fixtures. Just writing about how to make better use of fixtures. I actually use both fixtures and factories depending on the project specifics and also whether it is even my decision to make. :)
It's required for tests to clean up after themselves. With Rails and fixtures this is handled by default: each test runs inside a transaction which is rolled back at the end of the test. That way each test starts with the same initial state.
I'm with you on the one assertion per test. I bundled two assertions into the same test here because my whole point was to have them effectively together describe a single test, just in a more maintainable manner.
Regarding the fact that I'm not fully testing the contract of the method, you're absolutely correct. But also, no example based test suite is fully doing that. As long as the test suite is example based it is always possible to find a counter-case where the contract is violated but the test suite misses it.
These counter-cases will be more contrived and less likely the better the test suite. So all of us at some point decide that we've done enough and that more contrived cases are so unlikely and the cost of mistake is so small that it's not worth it to put in the extra testing effort. Some people don't explicitly think about it but that decision is still made one way or another.
This is a long way of saying that I both agree with you but that also, in most cases, I would still take the tradeoff and go for more maintainable tests.
Ah, ok, now I understand. Ok, I wasn't talking about that. From what I understand about property based testing it's sort of half way between regular example based testing and formal proofs: It tries to prove a statement but instead of a symbolic proof it does it stohastically via a set of examples?
Unfortunately, I'm not aware of a good property based testing library in Ruby, although it would be useful to have one.
Even so I'm guessing that property based testing in practice would be too resource intensive to test the entire application with it? You'd probably only test critical domain logic components and use regular example tests for the rest.
Ah, Ok, yes, for API endpoints it makes a lot of sense. Especially if it's a public API, you need to inspect the output anyway, to ensure that the public contract is not broken.
But, I spend very little or no time on API endpoints since I don't work on projects where the frontend is an SPA. :)
Author here. Yes, what you describe sound where much like what I call Factories (and that's what they're usually called in Ruby land, and some other languages).
The problem arises when they're used to generate Database records, which is a common approach in Rails applications. Because you're generating a lot of them you end up putting a lot more load on the test database which slows down the whole test suite considerably.
If you use them to generate purely in memory objects, this problem goes away and then I also prefer to use factories (or generators, as you describe them).
I think I'm getting what you mean and I almost completely agree with you, let me address one part, the only part where I don't agree:
> Plus you do gain testing power, because you can test more things. For example, you can confirm it returns _every_ active project.
Imagine this:
1. You start with some fixtures. You crafted the fixtures and you're happy that the fixtures are good for the test you're about to write.
2. You write a test where you assert the EXACT collection that is returned. This is, as you say, a test that "confirms the scope returns _every_ active project".
3. You now rewrite the test so that it checks that the collection includes ALL active projects and excludes all inactive projects.
Do you agree that nothing changed when you went from 2 to 3? As long as you don't change the fixtures, those 2 version of the test will behave exactly the same: if one passes so will the other and if one fails so will the other. As long as fixtures don't change they have exactly the same testing power.
If you agree on that, now imagine that you added another project to the fixtures. Has the testing power of the tests changed just because fixtures have been changed?
I'm familiar with snapshot testing for UI and I agree with you, they can work really well for this because they're usually quick to verify. And especially if you can build in some smart tolerance to the comparison logic, it can be really easy to maintain.
But how would you do snapshot testing for behaviour? I'm approaching the problem primarily from the backend side and there most tests are about behaviour.
When you say inheritance do you mean DRY as in "Don't repeat yourself"?
I'm not sure what you mean by inheritance in tests but DRY is criminally overused in tests. That could be a whole separate article but the tradeoffs are very different between test and app code and repetition in the test code is much less problematic and sometimes even desirable.
In theory you're 100% right, a true unit test is completely isolated from the rest of the system and than a lot of the problems disappear.
In reality, that is also not free. It imposes some restrictions on the code. Sometimes being pragmatic, backing off from the ideal leads to faster development and quicker deliver of value to the users. Rails is big on these pragmatic tradeoffs. The important thing is that we know when and why we're making the tradeoff.
Usually I go with Rails defaults and usually it's not a problem. Sometimes, when the code is especially complex and perhaps on the critical path, I turn up the purity dial and go down the road you describe exactly for the benefits you describe.
But when I decide that sticking to the defaults the is right tradeoff I want to get the most of it and use Fixtures (or Factories) in the optimal way.
Your thinking is sound. At the end of the day Rails default fixtures is nothing more than some code that reads yaml files and creates records once at the start of test suite run.
So you can definitely use FactoryBot to create them. However, the reason I think that's rarely done is that you're pretty likely to start recreating a lot of the features of Rails fixtures yourself. And perhaps all you need to do is to dynamically generate the yaml files. Rails yaml fixtures are actually ERB files and you can treat is an ERB template and generate its code dynamically: https://guides.rubyonrails.org/testing.html#embedding-code-i...
If that is flexible enough for you, it's a better path since you'll get all the usual fixture helpers and association resolving logic for free.
Author here. I'm a big fan of factories but the slowness is a real drag on large test suites. If you're considering switching, remember that you can do it gradually, there's no law against using both fixtures and factories in the same project, in some cases (mostly on very complex domain data models) even makes sense: fixtures for the base setup that all tests share, factories for additional test specific records.
Hm, if you missed it, perhaps I should have emphasised this part more, maybe add a blank line before it ...
Regarding the fact that the test does not check that the scope returns "all" active projects, that's a bit more complex to address but let me let tell you how I'm thinking about it:
The point of tests is to validate expected behaviours and prevent regressions (i.e. breaking old behaviour when introducing new features). It is impossible for tests to do this 100%. E.g. even if you test that the scope returns all active projects present in the fixtures that doesn't guarantee that the scope always returns all active projects for any possible list of active projects. If you want 100% validation your only choice is to turn to formal proof methods but that's whole different topic.
You could always add more active project examples. When you write a test that is checking that "Active projects A,B and C" are returned that is the same test as if your fixtures contained ONLY active projects A,B and C and then you tested that all of them are returned. In either case it is up to you to make sure that the projects are representative.
So by rewriting the test to check:
1. These example projects are included.
2. These other example projects are excluded.
You can write a test that is equally powerful as if you restricted your fixtures just to those example projects and then made an absolute comparison. You're not loosing any testing power. Expect you're making the test easier to maintain.
Does that make sense? Let me know which part is still confusing and I'll try to rephrase the explanation.
Have you used it over the last few years? It has it been rapidly improving, mainly because Shopify put a team full time on it. It doesn’t take a lot of people to optimize a VM/interpreter it just has to be the right people.
And the question is always “fast enough for what?” Different languages are more suitable for different types of projects. I wouldn’t code a rendering engine in Ruby but for web apps it’s amazing.
That was my misunderstanding as well. That's why I wrote the article.
Btw, it's not even about the RubyLLM gem. The gem abstracts away the calling of various LLM providers and gives a very clean and easy to use interface. But it's not what gives the "agentic magic". The magic is pretty much all in the underlying LLMs.
Seeing all the claims made by some closed source agent products (remember the "world's first AI software engineer"?) I thought that a fair amount of AI innovation is in the agent tool itself. So I was surprised when I realised that almost all of the "magic" parts are coming from the underlying LLM.
It's also kind of nice because it means that if you wanted to work on an agent product you can do that even if you're not an AI specialised engineer (like I am not).
Personally, I even slightly prefer to use Factories and I also previously wrote about a better way to use them: https://radanskoric.com/articles/test-factories-principal-of...