nijave

2,247 karmajoined 8 ปีที่แล้ว

[ my public key: https://keybase.io/nickv; my proof: https://keybase.io/nickv/sigs/qMdHcggaKRYKr002ExXFme5MjZeOQbHt-PyAl_07HDk ]

comments

1 ชั่วโมงที่ผ่านมาdiscuss

Debugging. Memory leak hunting, figuring out latency issues

For things where I'm not familiar with the code base, it can take days or weeks to become familiar with structure and flows.

If I can get an agent to help visualize and analyze the architecture and zero in on a subset of code, that can be a huge win.

For instance, we had some archaic "cache" that just dumped things into a static/module-level map. I tried a few different things to try to find it over the course of a few days and eventually gave Claude a Python REPL into a running process with pyrasite after some memory leaked and it traced through heap allocations and references to find the referent. It would have taken me probably 2-4+ weeks of just learning about Python heap to figure that out of my own.

1 ชั่วโมงที่ผ่านมาdiscuss

Seem to work fine for me but I usually add 1 sentence things to keep the agent from long winded rediscovery.

Remember credentials to query database are in secret store x

Remember company documentation is in platform y

Remember bug fixes always need tests added

12 ชั่วโมงที่ผ่านมาdiscuss

I write all my code in assembly on paper then manually translate into opcodes

12 ชั่วโมงที่ผ่านมาdiscuss

>already outsourced their thinking

With the reliability of current LLMs, if you're outsourcing that much of your thinking, you're producing mainly slop and were never a good engineer to begin with.

If you have a quality threshold beyond "it appears to work" then agents still require a lot of hand holding and guidance

12 ชั่วโมงที่ผ่านมาdiscuss

Eh, agents commits are always too verbose. I don't need a 3 paragraph mini novel. Design can go in a design docs folder, project info goes in the issue tracker.

The commit should be a short summary <1 paragraph for quick context

19 ชั่วโมงที่ผ่านมาdiscuss

It could be variations in system prompt but I'm not sure how "change the user prompt to encourage tool usage" proves that either way

เมื่อวานdiscuss

Yeah, that's fair. I think they're optimized for non technical users without a decent escape hatch.

เมื่อวานdiscuss

It's both. Some models, given web search and web fetch may only run a search and assume the summary text is correct and blindly return it. Others will validate by running a web fetch and checking the whole page contents. Even better, the model will run multiple searches and fetches to cross check the information. The best models will attempt to verify whether a source is authoritative or not and try to only return authoritative results.

I have an example here: https://gist.github.com/nijave/2873b8b10d8c732e46264237b0755...

Tldr; all the Claude models had identical tools and some used them efficiently and verified data while others did a crap job and hallucinated responses. Additionally, Exa MCP tools generally worked better even on older/smaller model (Llama)

เมื่อวานdiscuss

True but you can have cert-manager issue public certs then create service accounts for off cluster things to be able to pull the cert from the Secret so k8s+cert-manager acts as a local broker that handles renewal.

You can also invert and have k8s cronjobs provision the generated certs into other infra

With this setup, you don't have to worry about the RHEL certbot snap updating to a broken version which gets blocked by SELinux...

เมื่อวานdiscuss

We have a few subdomains for white labeling 3rd party SaaS where we do what is basically the AWS ACM equivalent and add a persistent record from a vendor.

With this setup, I don't have to grant 3rd parties DNS access.

I actually made a webhook that allows per hostname API keys to wrap dnsimple because they only had per zone keys and I didn't want each VM to have access to the entire zone. These challenges would have solved that by allowing the DNS record automation to pull record values from the VM instead of having the VMs push values.

I think someone told me dnsimple might have more granular keys now but I haven't checked. Iirc we have the same concern with external-dns at work (some things need subdomains on the TLD but we don't want to give external-dns access to the whole zone so we usually cname the TLD subdomain to a per environment zone external-dns is allowed to update). With this, we could have the same pull based setup that applies arbitrary rules to decide if a requested record should be created.

I think the main takeaway is allowing pull instead of push model

Could be achieved with cnames but it's an extra layer of indirection to deal with and doesn't fully solve the "semi trusted 3rd party" case

เมื่อวานซืนdiscuss

On k8s, there's cert-manager but also you need k8s...

Most browsers support trust on first use for leaf certs

เมื่อวานซืนdiscuss

My IaC is on public GitHub. They could do a network scan to find software then fingerprint to find version anyway.

Removing attack surface is better than trying to hide it.

เมื่อวานซืนdiscuss

Arguably it's 1/2. You can put public certs on proxy then give proxy private CA to backend services. Then you don't need public certs for all the private stuff nor need to trust the private CA on all your devices.

เมื่อวานซืนdiscuss

[dead]

เมื่อวานซืนdiscuss

I didn't realize the prevalence until I worked at a company trying to get enterprise market share desperate to "add logos" (the customers were always referred to as "logos" if that tells you anything)

เมื่อวานซืนdiscuss

Arguably I'd call that the 90%. In my case, answering the restaurant question correctly with "Rishi" in my tests was the sole intent and 90% of the problem. All the models "helpfully" added extra junk about the closure, dates, quotes, etc and many of them got these details wrong--the 10% or extra crap not central to the question.

If the central question was "what is the bus schedule on `day`" and the model screws that up, it gets a fail in my book.

Also curious if Google Maps gets the timetables correct (assuming it has them).

Semi-related, I also discovered that the default web search/fetch tools are pretty primitive and Exa MCP annihilates them. I ended up doing some comparisons with Claude Code comparing built-in server-side to Exa and to a Python MCP that used SearXNG for search and Exa was a clear winner and Python+SearXNG ended up coming out roughly the same after a few cycles of letting Claude optimize the Python code and adjust SearXNG settings. Ultimately it landed on this (making some changes to optimize returning relevant context directly in the search results so the model didn't need an additional web fetch call) https://gist.github.com/nijave/604c43e3e0fdcd60f5280d3a6b109...

เมื่อวานซืนdiscuss

My gut feel is Anthropic is very technical and pedantic which makes their models really technical and pedantic. They're top at code and technical benchmarks but anecdotally I've found OpenAI to be significantly farther ahead for general usage.

Opus 4.8 will burn 10k tokens trying to answer something 100% whereas GPT-5.5 will burn 2k getting it 90% which is good enough for many things.

Some personal testing on a "help me find that restaurant" prompt https://gist.github.com/nijave/2873b8b10d8c732e46264237b0755...

3 วันที่ผ่านมาdiscuss

Yeah, just another example of "connections pools are hard in general"

3 วันที่ผ่านมาdiscuss

Can also use different database names since pgbouncer lets you remap that

3 วันที่ผ่านมาdiscuss

You can rename the tabs in tmux but not sure if there's a way to do dynamic status (I suspect there is but it's probably non trivial to get it to play nice with the harness)