When you are trying to design high performance IP, you are often trying to ensure that your design is mathematically correct, inputs and outputs are matching a complicated 100 page specification. You are also trying to parse out the minimum set of workable requirements for "version 1" all with fitting into utilization constraints that ultimately are undefined.
Your mindset is really split. "Building up a software dashboard" to visualize your test results is really the last thing on your mind. You definitely don't want to be building the dashboard for all your customer's platforms.
Having somebody (a company) help on this front is really useful.
As a non-website designer, I used to think the same of tools like netlify, but they seem to be popular as ever, especially in a collaborative workspace when you need to handoff a project from one team to the next.
Thank you for tackling this critical problem for logic designiners. I think the tools available are much too old for fast paced workflows.
From my experience attempting to get a similar workflow down for my company:
I tried to use verilator a while back but ultimately I couldn't because it didn't have same constraints in the verilog language features that I was going to use in production. It doesn't even matter who was missing a feature, verilator or the proprietary tool, it was just about getting them to be same that caused the cognitive dissonance that I didn't want to deal with.
I ultimately decided to move away from verilator and use the clunky proprietary tools since it was what would be used in production. Getting "verilator compatibility" seemed like a "nice to have".
Second, the a winning local-first framework of verilator wasn't really established. You show in your example running a test from the yaml file using what looks like a bash script. Even as an experienced programmer who knows bash and sh well, I still find it very hard to write complex thoughts in it. The last high level attempt I found to bridge this gap is likely https://www.myhdl.org/ I don't know them personally, but it seemed like they had some very good thoughts on what makes writing good hardware level tests good. I think it would be worth reaching out to them if you haven't already.
The one thing that even more critical was a way to run our tests locally. The 10-20 seconds it takes to start a docker image (best case) in the cloud is really frustrating when you are "so close to finding a bug" and you "just want to see if this one line change is going to fix it". Once we got our whole pipeline going, it would take 1-6 minutes to "start a run" since it often had to rebuild previous steps that cache large parts of the design.
So I think you will want to see how you can help bring people's "local's first" workflows slowly into the cloud. Some tools (or just tutorials) that help you take a failing test, and run it locally and on the cloud will be really good especially as you get people to transition!
the other thing i notice from the story was that an update on something considered mission Critical was not given an update on within 24 hours.
IT should have volunteered the info regarding how far back in the backlog this was classified as soon as that prioritization was made. "Behind 14" and with many people on the testing side occupied is obviously not going to help with "layoff level priority".
To me, the classification of "enhancement" just doesn't seem to capture the urgency.
This is a story where a one line change of a hardcoded value actually went well.
I could image a scenerio where somebody stored the number of months of backlog as a 2 bit value. 0, 1, 2 or 3, you know, to be smart and clever. This may not appear as a problem during testing because it may be hidden many layers down, in some downstream service that is untested. Maybe in some low code automation service....
Changing it to 4 would mean the backlog is 0. Who knows what the consequences might be. Would that service go and cancel all jobs in the production queue?
Would it email all customers mentioning their stuff is cancelled?
I get that this is a seem gly easy change, but if a change of policy is expressed to the software team as an urgent problem, this seems like the management team needs better planning, and not randomly try to prioritize issues.....
They also seemed to have avoided libraries like numpy.
In my mind, python made it possible to hardware optimize with libraries like numpy quite easily. Avoiding it is a mistake. I'll try to see if I have time to play the game myself and throw my attempt in there.
Do you link to the shared object files? Or do you use subprocess?