amorroxic · HackerTrans

mese scorsodiscuss

Curious on what backs this assertion. As a counterpoint we’ve been running 200+ models in production for more than 5 years - language models, embedding, classifiers, low tens to hundred M params. Traffic in the order of 1-2M requests/day and everything is enabled by onnx with some cgo (or Rust) plumbing on top. What’s your SLA?

amorroxic

7 mesi fadiscuss

Thanks a lot for this. Also one question in case anyone could shed a bit of light: my understanding is that setting temperature=0, top_p=1 would cause deterministic output (identical output given identical input). For sure it won’t prevent factually wrong replies/hallucination, only maintains generation consistency (eq. classification tasks). Is this universally correct or is it dependent on model used? (or downright wrong understanding of course?)

amorroxic

6 anni fadiscuss

Agree, this being a cyber-fraud company I believe one could keep a team busy for a long time focusing on building secure systems rather than operating them to the point where no-one gets to see what's actually being stored within. The paper trail though I'm still puzzled about, he couldn't have acted alone.

comments