The claude provided skill-creator provides a decent jumping off point. It is easy enough to start with, but unless the skill is really simple I found it best to consider it a scaffold for building more tailored evals and reports.
The report leaves out a lot of detail. Several changes I found useful were: Pair with/without on same screen as left/right for easier viewing, token count for skill consumed, token used per run, time, pass rate, estimated cost, detailed aggregate stats, a parsed version of the conversation log (capturing the jsonl with each run, sometimes reading the log is the only way to find out why it's screwing up), work output logging (in my case screenshots and outputted script code), better formatting (syntax highlighting, log formatting).
Finally, I think the most useful thing was adding a self-reflection pass. After an eval is done, another agent looks at everything from that eval and tries to evaluate what went wrong along the way and what should be added to the skill, and conversely, from the without skill run what was in the skill that didn't need to be. It produces a skill change recommendation file for each eval. A further summary agent aggregates up all those recommendations in a way I can feed back to an agent.
fwiw, I just tried running the agent-skill they provide for fun to migrate an app-router based next 15 site and the end result is it entirely failed to start.
Vite just hangs when running vinext dev, with no output in logs whatsoever beyond printing`vinext dev (Vite 7.3.1)`.
Typo's keyboard was very much a copy and probably infringed on even more then was listed.
I can't think of any of their design patents this would interfere with. There's a small chance of some internal mechanical or light guide related patents, but that would be pretty unlikely. Even more unlikely would be BlackBerry having anyone around still that would even know what to look for.
This study doesn't make any practical sense. These pages weren't designed to convey the maximal amount of information in the least amount of space, they were designed to sell a product. It's impossible to claim if these designs have a negative impact due to content dispersion or not unless you are measuring them against the purpose they were designed for.
They explicitly studied ecommerce/product pages here. The relevant metrics are which page had a higher perceived product value? Which page had a higher conversion ratio? Which page resulted in a higher NPS? Which page created a more positive brand affinity?
You don't sell portable speakers using specs, you sell it with aspirational images of it being used on a beach. Of course expanding an accordion of product details then asking "On a scale from 1-7, How well do you feel you understood the offering communicated on the page?" results in a higher survey score. If you said the more dense page converted better, then I would be surprised.
It's like designing a study on the negative impact of hard F1 race car seats, adding a bunch of foam, testing which is more comfortable, then proclaiming one is better than the other because it was rated more comfortable, when the only metric they were designed for is lap time.
The report leaves out a lot of detail. Several changes I found useful were: Pair with/without on same screen as left/right for easier viewing, token count for skill consumed, token used per run, time, pass rate, estimated cost, detailed aggregate stats, a parsed version of the conversation log (capturing the jsonl with each run, sometimes reading the log is the only way to find out why it's screwing up), work output logging (in my case screenshots and outputted script code), better formatting (syntax highlighting, log formatting).
Finally, I think the most useful thing was adding a self-reflection pass. After an eval is done, another agent looks at everything from that eval and tries to evaluate what went wrong along the way and what should be added to the skill, and conversely, from the without skill run what was in the skill that didn't need to be. It produces a skill change recommendation file for each eval. A further summary agent aggregates up all those recommendations in a way I can feed back to an agent.