Show HN: Smelt — an open source test runner for chip developers(silogy-io.github.io)
silogy-io.github.io
Show HN: Smelt — an open source test runner for chip developers
https://silogy-io.github.io/smelt/
6 comments
Interesting concept for a project...
From what I have seen in the industry, it seems like this is something every organization ends up developing at least one custom tool for. EDA vendors even generally sell their own solution to this problem, but we always end up back at custom tools. It would be interesting to see if community collaboration could find a better general solution.
That said, it seems like Smelt is far too early in development to be practically used at this point. Some basic table stakes that I didn't see:
- Test weighting. Having a way to specify a relative repeat counts per test is essential when using constrained random tests like we do in the ASIC world.
- Some form of tagging to identify a test as belonging to multiple different groups. When combined with a way to intersect and join groups when specifying the tests to run.
- Control over the random seed for an entire test run. I was glad to see some support for test seeds. However when invoking smelt to run multiple tests, it would be nice to be able to reproducibly seed the generation of each individual test seed. Maybe this is outside the scope of this project?
Great things to see:
- Procedural test generation is a key feature.
- Extendable command invocation
- SLURM support is in the roadmap, also an important feature for groups that use SLURM.
That said, it seems like Smelt is far too early in development to be practically used at this point. Some basic table stakes that I didn't see:
- Test weighting. Having a way to specify a relative repeat counts per test is essential when using constrained random tests like we do in the ASIC world.
- Some form of tagging to identify a test as belonging to multiple different groups. When combined with a way to intersect and join groups when specifying the tests to run.
- Control over the random seed for an entire test run. I was glad to see some support for test seeds. However when invoking smelt to run multiple tests, it would be nice to be able to reproducibly seed the generation of each individual test seed. Maybe this is outside the scope of this project?
Great things to see:
- Procedural test generation is a key feature.
- Extendable command invocation
- SLURM support is in the roadmap, also an important feature for groups that use SLURM.
Thanks for checking out smelt!
Reproducible seeding, and more generically having "test list arguments" are definitely something we have in our view, and wouldn't be outside the scope of the project
Tagging is an interesting problem -- do you find that you use any querying schemes emerge when trying to run a specific set of tests, or is running a test group named "A" normally suffice? Test tagging should be on the roadmap, thanks for calling it out.
Re:test weighting, unsure what you're describing exactly -- would you like to have a mechanism to describe how many times a "root test" has been duplicated with some sampling of random input parameters?
Reproducible seeding, and more generically having "test list arguments" are definitely something we have in our view, and wouldn't be outside the scope of the project
Tagging is an interesting problem -- do you find that you use any querying schemes emerge when trying to run a specific set of tests, or is running a test group named "A" normally suffice? Test tagging should be on the roadmap, thanks for calling it out.
Re:test weighting, unsure what you're describing exactly -- would you like to have a mechanism to describe how many times a "root test" has been duplicated with some sampling of random input parameters?
You say this isn't a build system. Is there a build system it is designed to work with? I'm not sure I understand the point of separating the test runner from the build system. Is it to allow any build system to work with it?
Smelt is build system agnostic, but it seeks to be the "invoker" so to speak -- we've used it with a many build systems including make, cmake and bazel.
There are a few reasons to separate build and test systems:
build systems struggle to express constrained random testing, and some build systems even struggle to express directed testing sweeps, which are common patterns in design verification.
The other reason is that testing is often treated as a "leaf node" in a build graph, and it's not possible to describe tests depending on other tests.
Overall, testing is a different problem than building, and circuit design often requires complex testing flows that build systems aren't designed for.
There are a few reasons to separate build and test systems:
build systems struggle to express constrained random testing, and some build systems even struggle to express directed testing sweeps, which are common patterns in design verification.
The other reason is that testing is often treated as a "leaf node" in a build graph, and it's not possible to describe tests depending on other tests.
Overall, testing is a different problem than building, and circuit design often requires complex testing flows that build systems aren't designed for.
Agreed. One of the main properties of a (good) build system, is very aggressive caching. This only works if your build steps are deterministic. From my experience (testing firmware for embedded devices), even if your firmware simulation is totally deterministic, there is plenty of non-determinism created by UB, memory corruption, crazy things interns do with test scripts... So we don't typically cache our test results.
So do you support the APIs of different tools? Is there a list somewhere of told that are supported? Questa? VCS? Our so users need to figure that out?
We’re excited to open-source our test runner, Smelt. Smelt is a simple and extensible test runner optimized for chip development workflows. Smelt enables developers to:
* Programmatically define numerous test variants
* Execute these tests in parallel
* Easily analyze test results
As chip designs get more complex, the state space that needs to be explored in design verification is exploding. In chip development, it's common to run thousands of tests, each with multiple hyperparameters that result in even more variation. Smelt offers a straightforward approach to generating test variants and extracting valuable insights from your test runs. Smelt integrates seamlessly with most popular simulators and other chip design tools.
Key features:
* Procedural test generation: Programmatically generate tests with python
* Automatic rerun on failure: Describe the computation required re-run failing tests
* Analysis APIs: All of the data needed to track and reproduce tests
* Extensible: Define your tests with a simple python interface
Yves (https://github.com/silogy-io/yves) is a suite of directed performance tests that we brought up with smelt – check it out if you’d like to see smelt in action.
Repo: https://github.com/silogy-io/smelt
We built Smelt to streamline the testing process for chip developers. We're eager to hear your feedback and see how it performs in your projects!