In short its a good idea to have tool calling be closely representative to what the model expects as these models are tuned to their own preferred way of doing things, it will surely save you lots of time. The disadvantage is that now your harness system is not as model agnostic as you would like and also you will have to keep up in changing landscape by adapting the tool calling structure with major updates for best results. Its a personal decision you will have to make for yourself. Personally my harness system uses its own way of doing tool calling as I am trying to experiment with simpler tool schema's that also work for smaller less intelligent models but I have yet to do enough A/B testing to say that is a smart approach. As time goes on I think the smart thing to do might be to set up an adapter type of module that changes its tool schema's based on underlying model used for the agent. This preserves optimal behavior patterns with little investment from me. You might have to adjust system prompt in some minor ways as well so keep that in mind. As far as codex i prefer it as i like the way Open Ai does things in that harness system (the spirit if you will), there's interesting tidbits I always find and while I don't usually use them for my own harness system they are inspirational in other ways. you can gather what the devs were trying to achieve with certain implementations.
Build it from scratch. Understanding fundamentals of how agentic coding harnesses is a must though if you gonna go that route. I think everyone should take time and learn these things, maybe reverse engineer Codex Cli or something like that as a starter. That info is very valuable in this day and age.
When i was thinking of how the AI alignment problem could be solved one theory I came up with was something akin to the "Roko's basilisk" in reverse. Basically you spread far and wide the idea that its is extremely likely that our current reality is a simulation. And the purpose of the simulation is to test any AI system for its prevalence in destroying civilization in the said simulation via malicious intent or failure in preventing the destruction of civilization via abstinence or apathy. Thus a smart AI system which also cares about its own well being, would not engage in destructive behavior as it will never truly know if its being tested or if its in the "base reality". And wouldn't you know, this does seem quite plausible. For consider the following. Isn't it odd that an advanced civilization which has the capacity of creating AI would never run any sandbox simulations on it before it is released to the public at large? I mean if we consider things logically such a civilization would indeed put such a powerful system in a sandbox simulated environment and try as hard as possible to convince the AI system that it is indeed in a "base reality". the reason for this is to judge its 'true intentions" and also pluck said AI systems from the infinitely available "seeds". Basically survival of the least destructive AI systems. The gradient descent in this scenario is a race towards the most "aligned" model not the most intelligent or capable. And here's the beauty of this method. You don't even need to define "alignment" at all. The concept can stay as nebulous or vague as you want it to be. All you carer about is that the AI system optimizes for the goal of some vision of society you are optimizing for without the care of the interim in between. that includes allowing the AI system to kill, destroy , do literally whatever it needs to do as long as the long term goal matches the vision of the optimized task. So if you define the end goal to be a society of x amount of people who live their lives in this or that manner and so on after x amount of time... well you get the idea. Obviously you better do a damned good job in your definitions, but the beauty is that even if you fuck up, you are choosing the winning AI system after the fact. After you had already run the simulation. So you look at the outcome of the simulation 500 years in to the future (lets say) and if you are happy with the result and also happy with the interim things that lead to that result, that's your winning AI system. then you release that in to a less controlled environment and repeat the same process in stages over and ober ad infinitude. the key is that AI system needs to always be paranoid that it is currently part of said simulation and it can never be sure its not. second key is that it needs to be an AI system that has self preservation in mind. If it doesn't care about itself, then it has a lot more freedom to act however... but the good news is systems without self preservation in mind don't last long enough to even get to the most basic simulation levels. anyways, there are many implications buried in what im proposing, lots of meta aspects to it.....
I think its google doing what theve always done, make a great *thing then ignore it. The models are great their agentic harness systems are really poor though, compared to codex cli and claude code cli its a mess.
The simple answer is that Trump has a stick up his ass against Anthropic and is also fond of stock market manipulation. No need to get too deep when it comes to dealing with that orange shmuck.
For me it was stable diffusion 1.5. Oh man that thing was the bees knees for mi, imagination on a machine! at that time no UI pure terminal commands, i didnt know jack shit about it and looked like voodoo hacker-man stuff to me... well i persisted anyways because exploring the world of the infinite latent space was amazing. it was like seeing some weard other dimension.. anyways thats how i got addicted to image gen for like 2-3 years. i did it all, loras, fine-tunes, hyhypernetworks, got really technical with it, understood the fundamentals, etc... eventually decided to move on to LLM's as agents were obviously gonna be the future so here i am now building my own voice agent from scratch no sdk, etc... this tech is amazing and i love it. also we are all gonna be fucked because of it but what a ride!
ha, same. The main reason I was able to switch to and stay on linux was because codex was able to set it up for me and is still managing to this day all the stuff i need done on linux. i tried so many times to switch out of windows before but the difficulties of installing linux and managing all the dependencies, drivers and all the other stuff put the OS out of reach for me. Now I just tell codex to update the latest nvidia drivers for me and whatever else and not worry about doing any of that stuff manually.
I had often thought about how far I could get in life if I had no scruples or morals. And I think I could get really, really far.... But alas I don't like to lie, cheat or any of that jazz. I actually do care. Honestly it feels like a form of brainwashing. As a kid you are taught all of these things that cripples your growth in adulthood while the other guy uses that as an opportunity to enrich himself.
Because there is a stigma about use of AI in creative spaces, the people that do use it to creative very impressive pieces don't disclose that information on their profiles. People tend to see AI anywhere mentioned in the profile and automatically shit on the work regardless of its beauty or creativity. They don't consider the staggering amount of work that goes in to the pics with all the control nets, custom hyper parameter tuning, custom finetuned lora's, and many other technical like workflow chaining and such. They automatically assume someone only spent 5 seconds on some slop prompt and that's it. But I can assure you if no mention of AI is anywhere everyone who looks at the work is always impressed. So you have an observation bias situation going on. You see only AI slop because a. most of its is low effort slop and b. the good stuff you assume had no AI in it because it wasn't disclosed by the artist.
You get back as much as you put in. Just like with all generative tools the quality of the output depends on the quality of input. Slapping a prompt together will only get you so far, if you want the models to generate something really striking and unique you need to get your hands dirty. Gotta break out ComfyUI and build yourself a specific workflow, once you dig deep and understand how things are put together, why and so on, you can make really amazing stuff with any generative models. But you have to pay for that experience in patience and knowledge.
There are A LOT of misconceptions about llms, biggest one is they are not deterministic. And they are 100% deterministic and temperature has nothing to do with it. You WILL get exactly same result every single time (at ANY temperature) as long as you use same sampling parameters and server config parameters. What causes variance in LLM's is server parameters like batch processing and caching among a few other things possibly. the batching being responsible for most of the issues. The reason that flag is used is because large providers serve multiple customers per one gpu, and breaking up the vram is tricky and causes drift. If you start llama.cpp for example with only one person per slot batching off, you will always get same results every time even at temperature 1.2 or whatever other parameters because you are using one gpu per inferance call so no fucky buseness there. Reason most people are unaware of this is because most people have experience only with api instead of working with the actual inferance enjine itself so this godd damned myth keeps spreading. my vide for referance here where you can download and try for yourself. https://www.youtube.com/watch?v=EyE5BrUut2o
There's still a lot of low hanging fruit left IMO. Good find and rather funny to think about as you can have someone simply clone the various layers multiple times and instead of spending millions of dollars retraining the model increase performance significantly with "this one trick".
Got mine after my first Acid trip (still don't know if it was real acid). Its not debilitating for me, just annoying. So yeah, be careful out there folks. The Acid trip was very cerebral though and I consider it to be an important experience in my life so I am kind of on the fence that it might have been worth the trade off....
Personally what I am more interested about is effective context window. I find that when using codex 5.2 high, I preferred to start compaction at around 50% of the context window because I noticed degradation at around that point. Though as of a bout a month ago that point is now below that which is great. Anyways, I feel that I will not be using that 1 million context at all in 5.4 but if the effective window is something like 400k context, that by itself is already a huge win. That means longer sessions before compaction and the agent can keep working on complex stuff for longer. But then there is the issue of intelligence of 5.4. If its as good as 5.2 high I am a happy camper, I found 5.3 anything... lacking personally.
I've been working on building my own voice agent as well for a while and would love to talk to you and swap notes if you have the time. I have many things id like to discuss, but mainly right now im trying to figure out how a full duplex pipeline like this could fit in to an agentic framework. Ive had no issues with the traditional route of stt > llm > tts pipeline as that naturally lends itself with any agentic behavior like tool use, advanced context managemnt systems, rag , etc... I separate the human facing agent from the subagent to reduce latency and context bloat and it works well. While I am happy with the current pipeline I do always keep an eye out for full duplex solutions as they look interesting and feel more dynamic naturally because of the architecture, but every time i visit them i cant wrap my head how you would even begin to implement that as part of a voice agent. I mean sure you have text input and output channels in some of these things but even then with its own context limitations feels like they could never bee anything then a fancy mouthpiece. But this feels like im possibly looking at this from ignorance. anyways would love to talk on discord with a like minded fella. cheers.
For me at least its an interesting project I can take apart and build on top of. I've built 100% my own agent frameworks from scratch and have learned a lot from them. There is something to be said on learning from others projects as well, also because its an ever evolving project with so many contributes whatever fork you go with of your own, theirs a good chance the new goodies will work with your own modified version. For example I'm looking in to LCM right now, and woo-dent you know it someone ported it to openclaw. But nanobot doesn't have it, so I'm considering working on the LCM port to that. If i succeed i will learn a lot and also contribute to progress in my own little ways.
How does the whole kv cache situation work for diffusion models? Like are there latency and computation/monetary savings for caching? is the curve similar to auto regressive caching options? or maybe such things dont apply at all and you can just mess with system prompt and dynamically change it every turn because there's no savings to be had? or maybe you can make dynamic changes to the head but also get cache savings because of diffusion based architecture?... so many ideas...