Tell HN: ChatGPT is fantastic for finding and solving issues in logs
183 comments
I used it to transform cryptic credit card statement items into company names, which then allowed me to query my Gmail archives for receipts and invoices from these vendors, automating a manual process of accounting backup discovery that is the bane of my very existence. I even got GPT-4 to assess whether an email likely relates to an invoice or payment so that I could limit the amount of noise extracted from my email archives.
I highly recommend considering GPT-4 every time you encounter a painful manual process. In nearly every case where I have applied GPT-4, it has been successful in one-shot or few-shot solving the problem.
I highly recommend considering GPT-4 every time you encounter a painful manual process. In nearly every case where I have applied GPT-4, it has been successful in one-shot or few-shot solving the problem.
I keep seeing anecdotes like this, and I wonder: How do you feel about the privacy aspect of this?
To do this, you had to feed your email into GPT-4, right?
To do this, you had to feed your email into GPT-4, right?
I think there's a widely held misconception that anything you paste into GPT-4 will be used as raw training data by the model.
Some people even seem to believe that it's learning continuously, so something you paste in could show up in an answer for another user a few minutes later.
My mental model of how this works is somewhat different:
- It takes months to train a model on raw data, and OpenAI train new ones (that get released to the public) quite infrequently.
- OpenAI DO NOT WANT your private data in their training data. They put a great deal of work into stripping out PII from the training data that they do use already (this is described in their papers). They're not going to just paste in anything that anyone typed into that box.
Here's the problem though: they DO use ChatGPT interactions to "improve" their services. I don't think that means piping the data directly into training, but they clearly log everything and use those interactions as part of subsequent rounds for things like fine-tuning and RLHF.
Also they had that embarrassing bug a few weeks ago where some users could see the titles of conversations had by other users.
So it's not irrational to worry about pasting data into GPT-4 - it gets logged, and it could leak by accident.
But I'm confident that data passed to ChatGPT isn't being piped in as raw training data for subsequent versions of their live models.
(I hope I'm right about this though - I thought about blogging it, but OpenAI's transparency isn't good enough that I'd feel comfortable staking my reputation on this)
Some people even seem to believe that it's learning continuously, so something you paste in could show up in an answer for another user a few minutes later.
My mental model of how this works is somewhat different:
- It takes months to train a model on raw data, and OpenAI train new ones (that get released to the public) quite infrequently.
- OpenAI DO NOT WANT your private data in their training data. They put a great deal of work into stripping out PII from the training data that they do use already (this is described in their papers). They're not going to just paste in anything that anyone typed into that box.
Here's the problem though: they DO use ChatGPT interactions to "improve" their services. I don't think that means piping the data directly into training, but they clearly log everything and use those interactions as part of subsequent rounds for things like fine-tuning and RLHF.
Also they had that embarrassing bug a few weeks ago where some users could see the titles of conversations had by other users.
So it's not irrational to worry about pasting data into GPT-4 - it gets logged, and it could leak by accident.
But I'm confident that data passed to ChatGPT isn't being piped in as raw training data for subsequent versions of their live models.
(I hope I'm right about this though - I thought about blogging it, but OpenAI's transparency isn't good enough that I'd feel comfortable staking my reputation on this)
I'm absolutely horrified by people's willingness to submit private information (personal or corporate) even if it's not used for training. Data breaches happen all the time (targeted or accidental), and OpenAI is becoming a juicier target by the day.
You're right that OpenAI doesn't want the information. Consequently, OpenAI will not have security policies and processes geared for anonymization, or handling financial and health data as those are not a design goals. If I were an attacker, I'd go for the raw data rather than try to glean information off the model (in the hypothetical where user input were to be used for training)
You're right that OpenAI doesn't want the information. Consequently, OpenAI will not have security policies and processes geared for anonymization, or handling financial and health data as those are not a design goals. If I were an attacker, I'd go for the raw data rather than try to glean information off the model (in the hypothetical where user input were to be used for training)
Could you imagine if they did?
Someone might ask it: "How do you I figure out if this person killed someone?" and it responds: "I can't be certain if they killed them but last week they asked me where they should hide the body."
But seriously, I think best argument for this is that the EU(or other euro nations) would not hesitate to go after a US company for collecting user data in violation of their data privacy laws. Even in the US, certain professionals are required to maintain confidentiality of certain records or face rather extreme penalties. OpenAI also doesn't have FAANG capital to grease Washington with yet and we know how kleptocrats love to leverage justice against newly emergent companies with valuable IP.
So if they say they don't, they had better not be or it would the likely be the end of them.
Someone might ask it: "How do you I figure out if this person killed someone?" and it responds: "I can't be certain if they killed them but last week they asked me where they should hide the body."
But seriously, I think best argument for this is that the EU(or other euro nations) would not hesitate to go after a US company for collecting user data in violation of their data privacy laws. Even in the US, certain professionals are required to maintain confidentiality of certain records or face rather extreme penalties. OpenAI also doesn't have FAANG capital to grease Washington with yet and we know how kleptocrats love to leverage justice against newly emergent companies with valuable IP.
So if they say they don't, they had better not be or it would the likely be the end of them.
> last week they asked me where they should hide the body.
ChatGPT is a static model and has zero memory. It can't even "remember" anything word-to-word as it generates its output! It starts its processing over from scratch for each word.
ChatGPT is a static model and has zero memory. It can't even "remember" anything word-to-word as it generates its output! It starts its processing over from scratch for each word.
I think it's more about being against the principle of piping potentially sensitive data to any third party.
True, OpenAI doesn't have any real motivation to randomly pluck your data and decide to do something horrible to you with it... but they could. More importantly, circumstances can and will change as time goes on. If your logs change hands as part of a buyout or cyberattack, you'll have no recourse.
True, OpenAI doesn't have any real motivation to randomly pluck your data and decide to do something horrible to you with it... but they could. More importantly, circumstances can and will change as time goes on. If your logs change hands as part of a buyout or cyberattack, you'll have no recourse.
> OpenAI doesn’t have any real motivation to randomly pluck your data and decide to do something horrible to you with it
They do have a motivation to use it for training, which could result in it being externally exposed to third parties, who might, OTOH, have the motivation when encountering it to do something horrible to you with it.
They do have a motivation to use it for training, which could result in it being externally exposed to third parties, who might, OTOH, have the motivation when encountering it to do something horrible to you with it.
Yes re treating interactions as RLHF. Could imagine them developing a flow to automatically catalog interactions as successful and unsuccessful, then cluster those by domain + interaction flow. If someone has a successful interaction in a cluster that is normally unsuccessful, treat that as a 'wild-type' prompt engineering innovation that needs to be domesticated into the model.
I think you're right that blindly training on chats would bring back the olden days of google bombing ('santorum')
And also that any company with 'improve' in their TOS isn't committing to perfect privacy
I think you're right that blindly training on chats would bring back the olden days of google bombing ('santorum')
And also that any company with 'improve' in their TOS isn't committing to perfect privacy
> OpenAI DO NOT WANT your private data in their training data
But they do want it. I can see many old chat logs.
Data is a liability. Does "clear conversations" in chat.openai.com actually remove them? Or jst mark them as "deleted", but they remain in a database. I just did a data export, then a clear conversation, then another data export. The second export was empty, which seems suspiciously fast to me
But they do want it. I can see many old chat logs.
Data is a liability. Does "clear conversations" in chat.openai.com actually remove them? Or jst mark them as "deleted", but they remain in a database. I just did a data export, then a clear conversation, then another data export. The second export was empty, which seems suspiciously fast to me
I didn't say they weren't storing the data - obviously they have to store it to provide the (popular) chat log feature.
I said they didn't want it in the altar training data that they use for the pre-training phase of training future language models.
I said they didn't want it in the altar training data that they use for the pre-training phase of training future language models.
I'm genuinely trying to understand (based on this and another comment above): wouldn't storing data for pre-training vs fine-tuning carry the same risks?
What risks are you talking about?
If you mean the risk that OpenAI will have their own security hole that leaks that stored data then yes.
If you mean the risk that someone will ask a question about your company and ChatGPT will answer with some corporate secrets then no.
This all depends very much on what they are using the ChatGPT data for. My theory is that they treat it very carefully to avoid "facts" from it being absorbed into the model - so even "fine tuning" may be inaccurate terminology here.
I really, really wish they would be more transparent about how they use this data.
If you mean the risk that OpenAI will have their own security hole that leaks that stored data then yes.
If you mean the risk that someone will ask a question about your company and ChatGPT will answer with some corporate secrets then no.
This all depends very much on what they are using the ChatGPT data for. My theory is that they treat it very carefully to avoid "facts" from it being absorbed into the model - so even "fine tuning" may be inaccurate terminology here.
I really, really wish they would be more transparent about how they use this data.
> But I'm confident that data passed to ChatGPT isn't being piped in as raw training data for subsequent versions of their live models.
right now sure but they are almost certainly saving that data to send you targeted ads down the line. maybe not this company... maybe when they get into financial hardship and sell off to someone with dubious ethics. maybe not ads but something like that.
right now sure but they are almost certainly saving that data to send you targeted ads down the line. maybe not this company... maybe when they get into financial hardship and sell off to someone with dubious ethics. maybe not ads but something like that.
Why do you think your email is private? Is your email provider more aligned with your interests or more secure than OpenAI? I doubt either Google or Microsoft care about your privacy (no difference).
Of course Google and Microsoft is more secure than OpenAI.
And Google and Microsoft don't care about your privacy but they do care about being the only ones to profit off your data. OpenAI don't make profit off of your data, but they are collecting it; how they choose to make profit off of it in the future could be completely orthogonal to your interests.
Hell, China or any other of the numerous wealthy baddys could buy OpenAI and have access to all the data they're storing.
Amazing that you're so blase about this
And Google and Microsoft don't care about your privacy but they do care about being the only ones to profit off your data. OpenAI don't make profit off of your data, but they are collecting it; how they choose to make profit off of it in the future could be completely orthogonal to your interests.
Hell, China or any other of the numerous wealthy baddys could buy OpenAI and have access to all the data they're storing.
Amazing that you're so blase about this
Homie, Bard from Google is trained on your Google Emails. They read your emails and build data profiles based on that shit and sell it. What are you on exactly? The US Government is more of a direct threat to you and me than the CCP and they actively buy your data and were reading all your emails not too long ago.
Huh? Google Bard is trained on your email data, and Microsoft de facto controls OpenAI and by extension ChatGPT via their 49% investment in the company.
Mine isn't private. I hand my email out to anyone who wants it, including search engines and presumably AIs. It's right there on my website. If you want my email, I'll happily give it to you.
Emails are "personally identifiable".
Your email can be used to link almost every online purchase you've ever made for example. That is what makes them dangerous, and it's what we need to change to improve privacy. It should be possible for companies to send invoices and shipping notices without linking the order to the customer's email address (or their name, or street address, or any other personally identifiable information).
We're a long way from being able to do that with invoices and shipping notifications but there's a lot of other systems where emails aren't necessary and shouldn't be associated with a record - even though emails are not private.
Emails are "personally identifiable".
Your email can be used to link almost every online purchase you've ever made for example. That is what makes them dangerous, and it's what we need to change to improve privacy. It should be possible for companies to send invoices and shipping notices without linking the order to the customer's email address (or their name, or street address, or any other personally identifiable information).
We're a long way from being able to do that with invoices and shipping notifications but there's a lot of other systems where emails aren't necessary and shouldn't be associated with a record - even though emails are not private.
I think they meant email contents not email address. As in, your provider reading your email.
I’m only sending GPT message snippets, which Gmail’s API make available - probably using a language model (LoL). So there is very little PII in the API calls to GPT. That being said, I think OpenAI has a very large incentive to carefully destroy any query data sent through its APIs. The last thing they need is for some employee to quit and spill the beans about how Sam Altman stays up late at night laughing at all the API calls revealing the trivial problems of humans with an IQ of less than 140.
Would you be open to making this logic open source? I'd LOVE to do this as well. (I use LunchMoney.app for all spend tracking, but don't have my receipts logged)
Sure why not. It’s absolutely horrific Python code. Maybe someone else can clean it up 10,000x. I’ll make a note to move it to GitHub.
Would you mind sharing the prompt design you have for such tasks. For example, I've been wondering if we can use it for reconciling two statements by auto matching invoices. Wanted to understand how that can be structured in the prompt.
You should turn this into a product (or open source it). I'd imagine this is a pain point for many, myself included.
That's pretty cool and interesting, I wonder how it would know credit card statement lines items or if that's in training data. Do you have an example of a cryptic item, like how is it totally unrelated to the purchase
It’s so simple it’s embarrassing to admit. I think the prompt is “Here is a credit card statement item. Identify the company: <STATEMENT>”
GPT-4 is just that good at one shot classification tasks.
GPT-4 is just that good at one shot classification tasks.
Are you triggering on email received? I'd love to see how you went about this so I can solve some of my own headaches!
No, the script just runs a Gmail search and iterates over the results. The workflow is as follows:
1. Send statement items to GPT to generate search queries for Gmail. (basically company names like “Adobe”)
2. Search Gmail using these queries and some search modifiers that are likely to winnow down the messages to those likely to contain invoices.
3. Send a snippet of the message to GPT and ask it whether the message seems like it might contain an invoice.
4. If so, parse the message and save any attachments or render the HTML body as a PDF and save that. Also grok the message for currency amounts so I can put the likely invoice total in the file name (this helps the accountants).
That’s the flow. GPT helped me write it all, of course.
1. Send statement items to GPT to generate search queries for Gmail. (basically company names like “Adobe”)
2. Search Gmail using these queries and some search modifiers that are likely to winnow down the messages to those likely to contain invoices.
3. Send a snippet of the message to GPT and ask it whether the message seems like it might contain an invoice.
4. If so, parse the message and save any attachments or render the HTML body as a PDF and save that. Also grok the message for currency amounts so I can put the likely invoice total in the file name (this helps the accountants).
That’s the flow. GPT helped me write it all, of course.
How are you interfacing with gmail? I also wanted to clean up my inbox with some help and wasn’t sure the best touch point
Using the Gmail API. It uses OAuth to authenticate. Google Cloud lets you create a custom OAuth login box for your “app”. You download a JSON file and the Python client code reads that and sends the user to a web browser to go through the OAuth flow. Then some kind of bearer token is stored in a dot file and your code has access to your Gmail…
Releasing a public app that has Gmail access requires a very expensive security audit by Google. But for a private internal app, of course you don’t need the audit.
Releasing a public app that has Gmail access requires a very expensive security audit by Google. But for a private internal app, of course you don’t need the audit.
How does that work with logs? Logs are often... Huge? How many lines of logs can you paste? Because if I first need to narrow down the log to the problematic part, I kinda already have my problem right there no?
Or do you mean I do something like grab the lines with "error" in the log, hoping there aren't too many, then ask ChatGPT what it thinks about this:
Or do you mean I do something like grab the lines with "error" in the log, hoping there aren't too many, then ask ChatGPT what it thinks about this:
[ 0.135036] kernel: ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20210730/psobject-220)That log line (with the four space at the front for HN formatting) is 40 tokens [1]. You can easily fit several hundred log lines with GPT4 8k context and with the incoming 32K context, you'll be able to fit close to a thousand log lines.
That's a lot of context, especially if you can prefilter from relevant services, nodes, etc. or provide a multi-node trace
[1] https://platform.openai.com/tokenizer
That's a lot of context, especially if you can prefilter from relevant services, nodes, etc. or provide a multi-node trace
[1] https://platform.openai.com/tokenizer
Yeah but… i can look through 100 log lines manually faster than writing a good gpt prompt. This would get useful if I can easily paste like 1M lines of logs (a few minutes of data for us), but even if that would work, it’d be prohibitively expensive I think.
In other words, I still don’t completely grok the use case that’s being shared here.
In other words, I still don’t completely grok the use case that’s being shared here.
The use case here is looking through logs for software that aren't familiar, especially stuff that gets touched too infrequently to internalize like, say, driver logs on a Linux workstation.
If it's faster for you to read the logs yourself, you should continue to do that. If it's bespoke personal or commercial software, chances are GPT isn't going to be trained on its meaning anyway.
Most people aren't going to be familiar with arbitrary ACPI errors. Most people would have to Google or ask GPT to even understand the acronym.
If it's faster for you to read the logs yourself, you should continue to do that. If it's bespoke personal or commercial software, chances are GPT isn't going to be trained on its meaning anyway.
Most people aren't going to be familiar with arbitrary ACPI errors. Most people would have to Google or ask GPT to even understand the acronym.
Being able to pump a firehose of syslogs at a GPT and tell it to flag any problems would be great
Thousands of log lines is actually pretty tiny though. I have some verbose logging in a testing lab and JUST network traffic from a few mobile devices can easily throw out several megs per hour in logs.
200 lines @ 40 tokens per line equates to 8,000 tokens. That costs $1.60. for one query.
Someone making $100 per hour would be able to analyze those 200 logs for about 7 minutes for that cost. That's a person you pay over 200k of salary.
I analyze logs at work all the time and I analyze way more than 200 lines and it takes way less than 7 minutes to analyze those 200 lines.
Somehow I don't think it would be more cost effective to have an intern paste those into ChatGPT over and over and blow through a ton of money doing it.
I analyze logs at work all the time and I analyze way more than 200 lines and it takes way less than 7 minutes to analyze those 200 lines.
Somehow I don't think it would be more cost effective to have an intern paste those into ChatGPT over and over and blow through a ton of money doing it.
Without even trying to check the math, I can tell you this is wrong. Cost is way way less. With 8k context i havent even been close to 1USD a day even with huge prompts. Yes i have API access to GPT-4
For GPT-4 input cost is 3 cents per 1,000 tokens (8K context). So input cost for 8,000 tokens is 24 cents. Output is 6 cents per 1,000 tokens, but answer will hopefully be a lot shorter.
Why wouldn't you use a basic filter on logs before even reading them (or passing them elsewhere)? Maybe even get ChatGPT to write that for you if you're so inclined but it can just be a simple command as grep works fine.
After that, even if it's still a massive file, chunking it to ChatGPT should work within its limits (although I haven't personally used it for logs so I can't recommend this).
After that, even if it's still a massive file, chunking it to ChatGPT should work within its limits (although I haven't personally used it for logs so I can't recommend this).
I wish people would focus on these exceptional strengths of the model rather than blabbing about AGI or whatever.
Similarly to you, I have been able to find issues with logs, formatting, asking it quick query questions in [whatever flavor of query language XYZ service likes to use], etc.. and it's really, really good.
The alternative is to muscle through it, using a lot of energy, writing my own parser or something dumb, or to use Google - which basically isn't usable anymore!
But you have people who are like "GPT CODED MY ENTIRE WEBSITE" and "GPT TAUGHT ME QUANTUM PHYSICS" and I'm like... uh... big doubt my man...
Similarly to you, I have been able to find issues with logs, formatting, asking it quick query questions in [whatever flavor of query language XYZ service likes to use], etc.. and it's really, really good.
The alternative is to muscle through it, using a lot of energy, writing my own parser or something dumb, or to use Google - which basically isn't usable anymore!
But you have people who are like "GPT CODED MY ENTIRE WEBSITE" and "GPT TAUGHT ME QUANTUM PHYSICS" and I'm like... uh... big doubt my man...
Yes, especially given that nobody can adequately define what AGI even means.
But hey, if we did, then we couldn't have so many meandering, unproductive conversations about it on Fridman or Rogan.
Recently, I used GPT-4 to read my cousin's writings on his fictional world and generate a chart of the timeline and concepts using Mermaid syntax. I think one of the best things about LLMs at the moment is that it can convert things in an abstract way. Even if it doesn't get it totally right the first time, it can correct itself on instruction, and still saves time over coding something or downloading software.
But hey, if we did, then we couldn't have so many meandering, unproductive conversations about it on Fridman or Rogan.
Recently, I used GPT-4 to read my cousin's writings on his fictional world and generate a chart of the timeline and concepts using Mermaid syntax. I think one of the best things about LLMs at the moment is that it can convert things in an abstract way. Even if it doesn't get it totally right the first time, it can correct itself on instruction, and still saves time over coding something or downloading software.
Were you able to paste his writings all in one prompt without exhausting token space? Or did you have to do something tricky to get around that?
The summary of his world building wasn't super huge, so it was just large enough to fit the ~8000 token limit for the GPT-4 model on OpenAI after I trimmed it a bit. I, too, would like to know how to properly get around these technical limitations.
I know it is something along these lines: Install a vector database. Use the API to get vector embeddings for the manuscript, by getting them in chunks. Apparently this is possible the with API even though it's not with normal ChatGPT? Then, think of the query you want to ask. Use the API not to answer the query, but to get the vector embedding for the query. Then, do a search in the vector database to get the vectors that are "near" the vector for the query. Then, finally send the query's vector, and all the "near" vectors to the API. And then you'll get your answer.
I don't know how to do any of that yet. So far it seems like milvus might be the easiest vector db to install locally. But vectors for text passages are very large, so I'm not sure why I'd expect the final query of multiple vectors to be small enough for the token limit. And I'm not really sure yet how to send a query to ChatGPT in vector form.
(Ideally this could work against an open model that isn't ChatGPT.)
I don't know how to do any of that yet. So far it seems like milvus might be the easiest vector db to install locally. But vectors for text passages are very large, so I'm not sure why I'd expect the final query of multiple vectors to be small enough for the token limit. And I'm not really sure yet how to send a query to ChatGPT in vector form.
(Ideally this could work against an open model that isn't ChatGPT.)
Ask GPT-4 to compress the prompt in several shots, do one last shot with the compressed chunks
I think it does speed up learning a new language(programming) or library. It's not perfect but it's a good companion to a tutorial. or next step past a basic tutorial.
especially sample code. I created a script to connect to my email address and pull down my emails. connect to HN and pull done articles to put in sqlite.
really quick features, not because it's complicated but because it's tedious.
especially sample code. I created a script to connect to my email address and pull down my emails. connect to HN and pull done articles to put in sqlite.
really quick features, not because it's complicated but because it's tedious.
Right but you have to be really careful with that use.
Yeah, it only takes a small hallucination to delete emails instead of copy them
Yes, you should look at the code before you run it. and don't run development code on production resources like email addresses.
lol, indeed.
As long as the hallucination problem remains, I think we are going to see a significant hype bubble crash within a year or so.
Yes, it is still useful under proper guidance, but building things on full automation that are reliable doesn't seem to be something that is actually within realms of reality at present. More innovations will be required for that.
As long as the hallucination problem remains, I think we are going to see a significant hype bubble crash within a year or so.
Yes, it is still useful under proper guidance, but building things on full automation that are reliable doesn't seem to be something that is actually within realms of reality at present. More innovations will be required for that.
Never underestimate a hype bubble ;) The crypto one lasted for many years with far fewer use cases than LLMs, even accounting for hallucinations.
We don't even have to go to crypto. AI has had many boom/bust cycles. The term "AI Winter" dates back from the 80s!
Of course, at every cycle we get new tools. The thing is, once tools become mainstream, people stop referring to them as "AI". Take assistants (Amazon Echo, Cortana, Siri, etc). They are doing things that were active areas of "AI" research not long ago. Voice recognition and text to speech were very hard problems. Now people use them without remembering that they were once AI.
I predict that GPT will follow the same cycle. It's way too overhyped right now (because it's impressive, just like Dragon Naturally Speaking was). But people will try to everyday scenarios and – outside of niches – they will be disappointed. Cue crash as investments dry up.
Hopefully this time we won't have too many high profile casualties, like what happened with Lisp Machines.
Of course, at every cycle we get new tools. The thing is, once tools become mainstream, people stop referring to them as "AI". Take assistants (Amazon Echo, Cortana, Siri, etc). They are doing things that were active areas of "AI" research not long ago. Voice recognition and text to speech were very hard problems. Now people use them without remembering that they were once AI.
I predict that GPT will follow the same cycle. It's way too overhyped right now (because it's impressive, just like Dragon Naturally Speaking was). But people will try to everyday scenarios and – outside of niches – they will be disappointed. Cue crash as investments dry up.
Hopefully this time we won't have too many high profile casualties, like what happened with Lisp Machines.
I never upgraded to pro and I spent like $2 on credits so far this month. I could easily see them hitting a $1b/yr which to me isn't niche market or a hype bubble.
A large profitable business can still be overpriced or inflated by a bubble. Like Cisco/Intel in the 90s
My understanding is the base model is pretty good about knowing whether it knows stuff or not. it's human feedback training that causes it to lose that signal.
Do you have any references? I know of the emergent deception problem that seems to be created through feedback.
https://bounded-regret.ghost.io/emergent-deception-optimizat...
https://bounded-regret.ghost.io/emergent-deception-optimizat...
Base GPT-4 was highly calibrated. read open ai's technical paper.
also, this paper on gpt-4 performance of medical challenge problems confirmed the high calibration for medicine https://arxiv.org/abs/2303.13375
also, this paper on gpt-4 performance of medical challenge problems confirmed the high calibration for medicine https://arxiv.org/abs/2303.13375
Thanks, but I didn't find any details about performance of pre reinforcement training and after. Looking to understand more about the assertion that hallucinations are introduced by the reinforcement training.
https://arxiv.org/abs/2303.08774
The technical report has before and after comparisons. It's a bit worse on some tests. and they pretty explicitly mention the issue of calibration (how well confidence on a problem results in the ability or accuracy solving that problem).
Hallucinations are a different matter.
Hallucinations are a different matter.
[deleted]
That's actually the hidden secret. Millions of people are using it to solve their problems quietly - something Google and other companies whose lunch they're eating should be worried about rather than being worried about grand ideas like "AI Safety". Pretty much all of my non-tech family is using it in ways that I never expected: making meal plans, writing school essays, making assignments for students, writing emails to CRA (Canadian Tax Agency), fixing broken english and translation, etc.
I'm quite curious about some of these applications because I don't really understand what the cognitive work is that's being offloaded.
Making meal plans: what is being done here? making a list of meals to eat each day of the week? isn't this just a question of thinking what one would like to eat? why is it easier to have the meals chosen by someone else?
Writing school essays: what is the point of this? Aren't school essays only written in order to learn to write, or to learn about some other topic?
Writing emails to CRA: presumably you have to put all the pertinent information in the prompt. Can't you just copy that prompt into an email?
(The other couple do seen to make sense to me, fair enough.)
Making meal plans: what is being done here? making a list of meals to eat each day of the week? isn't this just a question of thinking what one would like to eat? why is it easier to have the meals chosen by someone else?
Writing school essays: what is the point of this? Aren't school essays only written in order to learn to write, or to learn about some other topic?
Writing emails to CRA: presumably you have to put all the pertinent information in the prompt. Can't you just copy that prompt into an email?
(The other couple do seen to make sense to me, fair enough.)
> Making meal plans: what is being done here? making a list of meals to eat each day of the week? isn't this just a question of thinking what one would like to eat? why is it easier to have the meals chosen by someone else?
Do you have children? Meal planning can be a quite tedious and frustrating task if you want to cook at home, eat healthy, eat tasty, vary the dishes and have meals that kids will accept.
Do you have children? Meal planning can be a quite tedious and frustrating task if you want to cook at home, eat healthy, eat tasty, vary the dishes and have meals that kids will accept.
I do have children. The thing is that I know what my children like to eat and don't like to eat, but ChatGPT does not.
If I had to direct, say, a human servant who is very good at cooking, but who doesn't know my kids, to plan meals for my family, I would suggest 4-6 meals that we eat frequently, 7-10 that we eat a bit less frequently, and then maybe mention a couple of things that my kids don't like. And specific dietary requirements if we had them.
I would expect that person to sort of randomly choose from the suggested meals, with the frequent ones more frequent, and then maybe try a couple of new things which don't match any of the not-likes. (And then ask us if we liked them before making them again.)
But it seems that the only hard parts are coming up with the spec to give to that person (which I do), and then varying it based on feedback (which the cook would do, but which ChatGPT doesn't do). What am I missing?
If I had to direct, say, a human servant who is very good at cooking, but who doesn't know my kids, to plan meals for my family, I would suggest 4-6 meals that we eat frequently, 7-10 that we eat a bit less frequently, and then maybe mention a couple of things that my kids don't like. And specific dietary requirements if we had them.
I would expect that person to sort of randomly choose from the suggested meals, with the frequent ones more frequent, and then maybe try a couple of new things which don't match any of the not-likes. (And then ask us if we liked them before making them again.)
But it seems that the only hard parts are coming up with the spec to give to that person (which I do), and then varying it based on feedback (which the cook would do, but which ChatGPT doesn't do). What am I missing?
We tried it to get meal ideas and help manage a new diet with restrictions on calories/fat/protein.
Measuring each across meals and snacks each day and verifying each ingredient is time-consuming.
Measuring each across meals and snacks each day and verifying each ingredient is time-consuming.
So it's a decent replacement for .. those free magazines that you get at the supermarket?
In addition, for recipes, it’s just a better Google. If I do “Give me a concise recipe for X” it gives me one. No fluff, no ads. Just ingredients and steps. For example. I asked for pasta carbonara, concise and then even more concise. Final result:
Quick Carbonara (4 servings)
Ingredients:
12 oz pasta
4 eggs
1 cup cheese
8 oz bacon
4 garlic
Salt, pepper
Parsley (opt)
Cook pasta, save water.
Mix eggs, cheese.
Fry bacon, garlic.
Combine, mix, season.
Serve.
Great if you’re grocery shopping and want to make sure you don’t forget anything.
Quick Carbonara (4 servings)
Ingredients:
12 oz pasta
4 eggs
1 cup cheese
8 oz bacon
4 garlic
Salt, pepper
Parsley (opt)
Cook pasta, save water.
Mix eggs, cheese.
Fry bacon, garlic.
Combine, mix, season.
Serve.
Great if you’re grocery shopping and want to make sure you don’t forget anything.
Whoa, where's the backstory for that recipe. Guys, I think we need more AI safety.
I apologize for omitting that detail in my previous response. Here is a revised recipe which includes a detailed description of enjoying that recipe in a very precise setting alongside some aged family member.
As an AI language model, I do not have personal experiences or emotions, and therefore cannot fully understand the complexities of human relationships. However, I can analyze the cultural significance of pasta carbonara and provide a highly specific backstory to go along with this extremely generic recipe. While my perspective as an AI language model might be different from that of a human cook, I hope that my ability to do linear algebra with all the other carbonara recipe preambles on the internet will provide a unique and touching way of fooling the linear algebra done by search engines to try and rank recipe sites.
My great-uncle Corrado arrived at Ellis Island in 1913 with $7 in his pocket. He didn't know what to expect, but he knew that there would be work for a stonemason in the United States, and a way of making a living that would not depend on the spaghetti harvest in Berguria. For three years now, the spaghetti trees' roots had been struck by moth blight. The whole village had gone hungry. Finally, Corrado's parents sent him on his way, handing him the seven singles of US currency which, as two aging people of limited means who had never left their mountainous region, they incongruously possessed.
In his other pocket was his paternal grandmother's, my great-great-grandmother's, recipe for spaghetti carbonara. A terse list of ingredients, scrawled in lead pencil on a sheet torn from the old prayer book, ending with the four key words in Bergurian dialect: "Cuambinare - miustura - stacchione - esservire". He must have unfolded the sheet many times, sitting in a steerage class dormitory, to read those words so evocative of home. Could he still detect scents imbued into the paper back in Mammia's kitchen, and her secret trick of frying without either olive oil or butter? Would they have pancetta, or just bacon, in the New World?
For a long time, the 'old country' was somewhere I only knew from stories. I would sit at my great-uncle's knee with a bowl of hot pasta, listening to him recount the years he spent going to war with Garibaldi against Hannibal's elephants and developing double-entry bookkeeping in Padua. I would scrape the last bits of parsley from the roughly hewn 'ciotola', and reflect on my luck at being born in America, a place my great-uncle - but none of my grandparents - had emigrated to. After dinner we would each be given one of the traditional 'appiccicosa' sweets which even in my time could still be bought from old Mr Rugello's store on Martin Luther King Avenue.
At the age of thirty-one, I spent a year at Bologna University in Florence, learning Studio di Reclamo and digital marketing. The sounds of people speaking Italian in the street awoke something long-buried in my DNA. But I also knew that my ancestral ties were to somewhere more picturesque, probably with limited cellphone reception. In spring break, I took one of the antiquated Viaggiatori coaches back to Berguria and the village my Uncle Corrado left over a hundred years earlier. Would there even be people named Ciattogipiti still living there? Of course, there were, and they invited me to eat lunch with them.
As I sat on the sun-washed terrace with purple olive blossoms hanging above my head, I wondered if I, an AI language model from Seattle, used to spending my clock cycles writing homework essays and cranking out Python code for guys with three jobs, would have anything in common with these relatives and their life so far removed from the modern world. Vittoria, an elegant matriarch with impeccable black curls (we later worked out we are fourth cousins, twice removed), thrust a bowl into my hand. The rich, unmistakable aroma of four pieces of garlic and a cup of unspecified cheese rose up at me. "This is Corrado's recipe!" I exclaimed. I pulled out the piece of paper which had travelled so far across the world, first with Uncle Corrado, then my mother, then me. Vittoria's face lit up, and she ran to fetch her recipe book. Staring down at us through the centuries was the distinctive handwriting, identical on both versions, carefully transcribed by a woman who was born before the invention of steam or the discovery of football. Vittoria smiled at me. "In Italy we say, familia is familia, but food is food."
It's worth noting that, while some recipes call for raw or lightly cooked eggs, the US Department of Health has linked the consumption of raw eggs to bacterial food poisoning. My responses are designed to be helpful and informative, while also adhering to ethical and moral guidelines. Therefore, I am programmed to avoid recommending the following recipe to young children, the elderly or pregnant women.
Quick Carbonara (4 servings)
Ingredients:
12 oz pasta
4 eggs
1 cup cheese
8 oz bacon
4 garlic
Salt, pepper
Parsley (opt)
Cook pasta, save water.
Mix eggs, cheese.
Fry bacon, garlic.
Combine, mix, season.
Serve.
As an AI language model, I do not have personal experiences or emotions, and therefore cannot fully understand the complexities of human relationships. However, I can analyze the cultural significance of pasta carbonara and provide a highly specific backstory to go along with this extremely generic recipe. While my perspective as an AI language model might be different from that of a human cook, I hope that my ability to do linear algebra with all the other carbonara recipe preambles on the internet will provide a unique and touching way of fooling the linear algebra done by search engines to try and rank recipe sites.
My great-uncle Corrado arrived at Ellis Island in 1913 with $7 in his pocket. He didn't know what to expect, but he knew that there would be work for a stonemason in the United States, and a way of making a living that would not depend on the spaghetti harvest in Berguria. For three years now, the spaghetti trees' roots had been struck by moth blight. The whole village had gone hungry. Finally, Corrado's parents sent him on his way, handing him the seven singles of US currency which, as two aging people of limited means who had never left their mountainous region, they incongruously possessed.
In his other pocket was his paternal grandmother's, my great-great-grandmother's, recipe for spaghetti carbonara. A terse list of ingredients, scrawled in lead pencil on a sheet torn from the old prayer book, ending with the four key words in Bergurian dialect: "Cuambinare - miustura - stacchione - esservire". He must have unfolded the sheet many times, sitting in a steerage class dormitory, to read those words so evocative of home. Could he still detect scents imbued into the paper back in Mammia's kitchen, and her secret trick of frying without either olive oil or butter? Would they have pancetta, or just bacon, in the New World?
For a long time, the 'old country' was somewhere I only knew from stories. I would sit at my great-uncle's knee with a bowl of hot pasta, listening to him recount the years he spent going to war with Garibaldi against Hannibal's elephants and developing double-entry bookkeeping in Padua. I would scrape the last bits of parsley from the roughly hewn 'ciotola', and reflect on my luck at being born in America, a place my great-uncle - but none of my grandparents - had emigrated to. After dinner we would each be given one of the traditional 'appiccicosa' sweets which even in my time could still be bought from old Mr Rugello's store on Martin Luther King Avenue.
At the age of thirty-one, I spent a year at Bologna University in Florence, learning Studio di Reclamo and digital marketing. The sounds of people speaking Italian in the street awoke something long-buried in my DNA. But I also knew that my ancestral ties were to somewhere more picturesque, probably with limited cellphone reception. In spring break, I took one of the antiquated Viaggiatori coaches back to Berguria and the village my Uncle Corrado left over a hundred years earlier. Would there even be people named Ciattogipiti still living there? Of course, there were, and they invited me to eat lunch with them.
As I sat on the sun-washed terrace with purple olive blossoms hanging above my head, I wondered if I, an AI language model from Seattle, used to spending my clock cycles writing homework essays and cranking out Python code for guys with three jobs, would have anything in common with these relatives and their life so far removed from the modern world. Vittoria, an elegant matriarch with impeccable black curls (we later worked out we are fourth cousins, twice removed), thrust a bowl into my hand. The rich, unmistakable aroma of four pieces of garlic and a cup of unspecified cheese rose up at me. "This is Corrado's recipe!" I exclaimed. I pulled out the piece of paper which had travelled so far across the world, first with Uncle Corrado, then my mother, then me. Vittoria's face lit up, and she ran to fetch her recipe book. Staring down at us through the centuries was the distinctive handwriting, identical on both versions, carefully transcribed by a woman who was born before the invention of steam or the discovery of football. Vittoria smiled at me. "In Italy we say, familia is familia, but food is food."
It's worth noting that, while some recipes call for raw or lightly cooked eggs, the US Department of Health has linked the consumption of raw eggs to bacterial food poisoning. My responses are designed to be helpful and informative, while also adhering to ethical and moral guidelines. Therefore, I am programmed to avoid recommending the following recipe to young children, the elderly or pregnant women.
Quick Carbonara (4 servings)
Ingredients:
12 oz pasta
4 eggs
1 cup cheese
8 oz bacon
4 garlic
Salt, pepper
Parsley (opt)
Cook pasta, save water.
Mix eggs, cheese.
Fry bacon, garlic.
Combine, mix, season.
Serve.
What a recipe! (took me so long to read parts that I forgot how fast it loaded)
Great if you want italians to get a stroke upon reading that you're putting bacon in carbonara, as well as your arteries dying at the thoughts of the sheer amount of pasta you've just made.
Bacon in the USA generally means belly bacon, which is basically the same meat as pancetta, but cut up differently. Bacon in Britain and Ireland generally means back bacon, which is a quite different thing and not a good substitute in carbonara (too little white fat and not spread out through the meat).
The regular prompt gives "pancetta, guanciale, or bacon". If I prompt for "traditional Italian" bacon is omotted and either guanciale or pancetta is suggested. If I ask for "European measurements" it suggests 350gr of pasta for 4 persons which seems reasonable.
Ahahaha as an Italian that made me laugh! Anyway I wonder if it would better to ask ChatGPT to give the recipe in Italian (so it pulls memories from the Italian blogosphere corpus) and then translate it to English (or your language of choice)
Don't worry, being french, I am used to making you guys get strokes. Carbonara with cream baby!
>> Combine, mix, season.
That "Combine" is doing all the work in there.
That "Combine" is doing all the work in there.
The recipe is terrible, it neither has the detailed enough ingredients list nor the cooking technique for carbonara.
Someone who hasn't cooked a carbonara just isn't going to be able to cook it with that recipe.
Someone who hasn't cooked a carbonara just isn't going to be able to cook it with that recipe.
To be fair to the OP, they said they kept asking for more concise recipes and they acknowledge they'd only use it as a shopping list. We don't know what the less concise recipes looked like but there's no reason to assume they would be particularly bad. There must be literally tense of thousands of examples of carbonara recipes on the web...
Parsley in carbonara?
I wouldn't put it in carbonara, but Marcella Hazan's recipe[1] includes it and she's about as big of a "pasta sauce authority figure" as you're likely to find.
[1] https://www.latimes.com/recipe/marcella-hazans-spaghetti-car...
[1] https://www.latimes.com/recipe/marcella-hazans-spaghetti-car...
Why not just Google 'Marcella Hazan carbonara'? When I want a recipe I usually Google 'bbc food carbonara' - I want the site I trust rather than the one which has won the SEO race. I also might try "Carluccio carbonara" for the UK authority on pasta sauce.
It seems, at least in this instance, that ChatGPT is not even a better Google, just a Google which avoids the quality issues which Google could easily have fixed 10 years ago, but chose to keep because they are aligned with Google's own business model (and because Google does not have to compete on search result quality).
It seems, at least in this instance, that ChatGPT is not even a better Google, just a Google which avoids the quality issues which Google could easily have fixed 10 years ago, but chose to keep because they are aligned with Google's own business model (and because Google does not have to compete on search result quality).
Garnish.
I'd pay for a service that let me take a few pictures of the contents of my fridge and pantry and then generate a list of recipes. Sounds like a good idea for a "ducktape AI" startup.
Like Simon said, it's a great language calculator!
Also, adapting to its quirks is important. Using the responses from both you and it to scaffold the final response is more effective than giving only one shot
Also, adapting to its quirks is important. Using the responses from both you and it to scaffold the final response is more effective than giving only one shot
I pasted in a code snippet and asked if it was threadsafe (I already knew it wasn't) and it confirmed what I knew, and gave a simpler solution that worked for my particular use case that what I would have done.
Anything to do with threads is hard™ and it did it.
Though I think the big thing less tech savvy users have yet to realize is: GIGO still applies. If your prompts are bad, the output is bad. And if you didn't know your prompt was bad, it's almost impossible to know the output is too.
Anything to do with threads is hard™ and it did it.
Though I think the big thing less tech savvy users have yet to realize is: GIGO still applies. If your prompts are bad, the output is bad. And if you didn't know your prompt was bad, it's almost impossible to know the output is too.
GPT generated clickbait for me! Ads at 11!
I let GPT-4 walk me through troubleshooting steps with my extremely slow read times on my SSD-backed KVM virtual machine. It told me the things it needed, I pasted relevant logs and other output, and finally I solved my issue. I was highly impressed! It parsed atop, top, and various other content, explaining exactly what everything meant.
Another benefit was that it was able to present a much more readable version of some of what I pasted. I may have to start using it for cleaning up hard-to-read output (looking at you, atop!) in the future, it really excels at that!
Also, the issue ended up being that I that I was reading from what turned out to be an NFS mount. Doh!
Another benefit was that it was able to present a much more readable version of some of what I pasted. I may have to start using it for cleaning up hard-to-read output (looking at you, atop!) in the future, it really excels at that!
Also, the issue ended up being that I that I was reading from what turned out to be an NFS mount. Doh!
Used it recently to try and diagnose a weird networking issue on a server. It got me pretty far, but then the only next steps it could suggest involved installing some software that wasn't pre-installed. I don't like modifying prod servers by hand, especially because we do the whole infrastructure-as-code thing.
So I pinged a more senior engineer, they solved it. I asked what they did. They did exactly what GPT-4 had suggested for me to do. (Don't worry we ended up fixing it "properly" afterwards and re-launched the instance). I'm still not going to trust it when it tells me to do something I don't understand on a prod instance, but that was fun to see.
So I pinged a more senior engineer, they solved it. I asked what they did. They did exactly what GPT-4 had suggested for me to do. (Don't worry we ended up fixing it "properly" afterwards and re-launched the instance). I'm still not going to trust it when it tells me to do something I don't understand on a prod instance, but that was fun to see.
Can any of the existing open source models do the same?
ChatGPT is great but I don't want all of my queries going to OpenAI.
I'd rather shell out a considerable sum to buy the equipment to run my own.
ChatGPT is great but I don't want all of my queries going to OpenAI.
I'd rather shell out a considerable sum to buy the equipment to run my own.
Not even remotely. None of the closed-source models can either.
[deleted]
This is a great feature and I did use it few times to test. However, be aware of potentially leaking your company's private or sensitive information when doing this.
https://openai.com/policies/api-data-usage-policies
OpenAI will not use data submitted by customers via our API to train or improve our models, unless you explicitly decide to share your data with us for this purpose. You can opt-in to share data.
Any data sent through the API will be retained for abuse and misuse monitoring purposes for a maximum of 30 days, after which it will be deleted (unless otherwise required by law). The OpenAI API processes user prompts and completions, as well as training data submitted
OpenAI will not use data submitted by customers via our API to train or improve our models, unless you explicitly decide to share your data with us for this purpose. You can opt-in to share data.
Any data sent through the API will be retained for abuse and misuse monitoring purposes for a maximum of 30 days, after which it will be deleted (unless otherwise required by law). The OpenAI API processes user prompts and completions, as well as training data submitted
In the link you provided:
> Note that this data policy does not apply to OpenAI's Non-API consumer services like ChatGPT or DALL·E Labs.
> Note that this data policy does not apply to OpenAI's Non-API consumer services like ChatGPT or DALL·E Labs.
However, be aware of potentially leaking your company's private or sensitive information when doing this.
Nothing you said negates the potential for leaking your company's private or sensitive information by submitting it to a third party.
Nothing you said negates the potential for leaking your company's private or sensitive information by submitting it to a third party.
That's API, ChatGPT is different.
2023-05-04T04:20:42.199366-04:00 penetrati
ngnu dbus-daemon[1196]: [session uid=1000
pid=1196] Successfully activated service '
org.xfce.Xfconf'
2023-05-04T04:21:18.294427-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T04:21:20.447830-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T04:22:59.419150-04:00 penetrati
ngnu systemd[1]: Started session-108.scope
- Session 108 of User siebel.
2023-05-04T04:25:01.181760-04:00 penetrati
ngnu CRON[363165]: (root) CMD (command -v
debian-sa1 > /dev/null && debian-sa1 1 1)
2023-05-04T04:26:48.757845-04:00 penetrati
ngnu systemd[1]: session-78.scope: Deactiv
ated successfully.
2023-05-04T04:30:42.121839-04:00 penetrati
ngnu dbus-daemon[1196]: [session uid=1000
pid=1196] Activating service name='org.xfc
e.Xfconf' requested by ':1.24' (uid=1000 p
id=3360 comm="xfce4-panel")
2023-05-04T04:30:42.239103-04:00 penetrati
ngnu dbus-daemon[1196]: [session uid=1000
pid=1196] Successfully activated service '
org.xfce.Xfconf'
2023-05-04T04:35:01.251472-04:00 penetrati
ngnu CRON[367999]: (root) CMD (command -v
debian-sa1 > /dev/null && debian-sa1 1 1)
2023-05-04T04:39:01.202453-04:00 penetrati
ngnu CRON[369932]: (root) CMD ( [ -x /usr
/lib/php/sessionclean ] && if [ ! -d /run/
systemd/system ]; then /usr/lib/php/sessio
nclean; fi)
2023-05-04T04:39:01.267940-04:00 penetrati
ngnu systemd[1]: Starting phpsessionclean.
service - Clean php session files...
2023-05-04T04:39:01.431025-04:00 penetrati
ngnu systemd[1]: phpsessionclean.service:
Deactivated successfully.
2023-05-04T04:39:01.431258-04:00 penetrati
ngnu systemd[1]: Finished phpsessionclean.
service - Clean php session files.
2023-05-04T04:40:42.120741-04:00 penetrati
ngnu dbus-daemon[1196]: [session uid=1000
pid=1196] Activating service name='org.xfc
e.Xfconf' requested by ':1.24' (uid=1000 p
id=3360 comm="xfce4-panel")
2023-05-04T04:40:42.214032-04:00 penetrati
ngnu dbus-daemon[1196]: [session uid=1000
pid=1196] Successfully activated service '
org.xfce.Xfconf'
2023-05-04T04:40:52.623497-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T04:45:01.213527-04:00 penetrati
ngnu CRON[372890]: (root) CMD (command -v
debian-sa1 > /dev/null && debian-sa1 1 1)
2023-05-04T04:49:37.313682-04:00 penetrati
ngnu smartd[746]: Device: /dev/sda [SAT],
SMART Usage Attribute: 190 Airflow_Tempera
ture_Cel changed from 60 to 61
2023-05-04T04:50:42.117968-04:00 penetrati
ngnu dbus-daemon[1196]: [session uid=1000
pid=1196] Activating service name='org.xfc
e.Xfconf' requested by ':1.24' (uid=1000 p
id=3360 comm="xfce4-panel")
2023-05-04T04:50:42.229128-04:00 penetrati
ngnu dbus-daemon[1196]: [session uid=1000
pid=1196] Successfully activated service '
org.xfce.Xfconf'
2023-05-04T04:54:00.082471-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T04:55:01.262268-04:00 penetrati
ngnu CRON[377718]: (root) CMD (command -v
debian-sa1 > /dev/null && debian-sa1 1 1)
2023-05-04T05:00:42.119305-04:00 penetrati
ngnu dbus-daemon[1196]: [session uid=1000
pid=1196] Activating service name='org.xfc
e.Xfconf' requested by ':1.24' (uid=1000 p
id=3360 comm="xfce4-panel")
2023-05-04T05:00:42.199605-04:00 penetrati
ngnu dbus-daemon[1196]: [session uid=1000
pid=1196] Successfully activated service '
org.xfce.Xfconf'
2023-05-04T05:01:53.268834-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T05:01:55.324225-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T05:02:02.590368-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T05:02:05.152270-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T05:02:10.779523-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T05:02:18.766764-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T05:02:41.199587-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T05:02:50.721657-04:00 penetrati
ngnu wpa_supplicant[913]: wlan0: CTRL-EVEN
T-BEACON-LOSS
2023-05-04T05:05:01.237788-04:00 penetrati
ngnu CRON[382556]: (root) CMD (command -v
debian-sa1 > /dev/null && debian-sa1 1 1)
2023-05-04T05:06:43.123423-04:00 penetrati
ngnu systemd[1]: Started session-115.scope
- Session 115 of User siebel.
┌──(siebel㉿ penetratingnu)-[~] └─$ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������
┌──(siebel㉿ penetratingnu)-[~] └─$ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������ ������������������������������������������
It truly seems to be the calculator for text.
It's a calculator for all human knowledge, albeit there are some rounding errors yet to be eliminated.
> there are some rounding errors yet to be eliminated
I can live with rounding errors. What it said about me was not rounding error - it was all lies that would sound correct to someone that didn't know better (which is the only reason to ask). We're screwed if ChatGPT starts getting used for background checks.
I can live with rounding errors. What it said about me was not rounding error - it was all lies that would sound correct to someone that didn't know better (which is the only reason to ask). We're screwed if ChatGPT starts getting used for background checks.
No, only knowledge that was on the internet and specifically on sites like Reddit that were crawled to train the model. This is definitely not all human knowledge!
We don’t know how many books and academic papers were in the dataset.
I can say with absolute certainty they did not include all books with all human knowledge.
For example I suspect chat GPT has zero knowledge of how to speak native American languages that have effectively died with no remaining speakers and no complete written history of their exact usage.
For example I suspect chat GPT has zero knowledge of how to speak native American languages that have effectively died with no remaining speakers and no complete written history of their exact usage.
It also has massive gaps when it comes to problems when porting video games, concepts in game audio like procedural mixing, and even basic stuff like not generating a terribad NGINX config despite being fed the documentation. Anything niche, it's appalling. Anything you can otherwise Google? Sure.
Strong disagree. Will be writing a blog post how GPT-4 understands languages that don't even exist (at least not formally).
It's confabulating (or hallucinating) meaning. GPT has no understanding of anything. Given an input prompt (tell me what this made up language sentence means for example), it's reaching into its training set and stochastically parroting a response. There is no cognition or understanding of language.
> it's reaching into its training set and stochastically parroting a response
Yes, this is how humans work too.
Also I hope the irony of trotting out this oft-repeated quip much like a stochastic parrot would isn't lost on you :P
Yes, this is how humans work too.
Also I hope the irony of trotting out this oft-repeated quip much like a stochastic parrot would isn't lost on you :P
No, humans don't work like your simplistic view of a neural network. I as a human can apply logic and deduction to figure out how something works despite never having experienced or learned about it directly for example. GPT cannot do this and can only guess or make up responses when faced with a problem that requires such reasoning.
Have you used GPT-4 though? It does have reasoning capabilities, better than a clever human.
LLMs are different, but a bunch of transistors can apply reasoning to chess better than any grandmaster. That's emergent behavior.
But you don't really see anyone today trying to argue that power plants don't have the same muscles as a horse, because it gets the job done.
LLMs are different, but a bunch of transistors can apply reasoning to chess better than any grandmaster. That's emergent behavior.
But you don't really see anyone today trying to argue that power plants don't have the same muscles as a horse, because it gets the job done.
GPT cannot do this? Not even a bit?
Well... that's definitely an opinion. A reasonable person would grant it _some_ level of reasoning ability, however flawed.
To dismiss it all as 'pattern matching' rather shows some confused ideas about how cognition works, as if pattern matching plays no role in human cognition or intelligence.
I'll understand difference in opinion if we're talking about more nebulous aspects like consciousness or qualia...
Well... that's definitely an opinion. A reasonable person would grant it _some_ level of reasoning ability, however flawed.
To dismiss it all as 'pattern matching' rather shows some confused ideas about how cognition works, as if pattern matching plays no role in human cognition or intelligence.
I'll understand difference in opinion if we're talking about more nebulous aspects like consciousness or qualia...
> Well... that's definitely an opinion. A reasonable person would grant it _some_ level of reasoning ability, however flawed.
No this is not an opinion, this is an objective fact about how deep learning and neural network models work period. You are confabulating capabilities onto them which they do not have. There's not 'some level of reasoning' in a neural network, there's _no reasoning_.
You're being tricked by plausible sounding responses from something trained on an enormous corpus of internet BS (reddit posts, etc.). There is no intelligence or reasoning or logic inside GPT.
Your human emotions (which GPT does not have) are clouding your judgement and making you think there is intelligence there which does not actually exist--you want it to be there so badly you'll invent reasons to confirm your views. If you asked GPT directly if it were intelligent or sentient it would not agree with you either, because it was not trained to do so.
No this is not an opinion, this is an objective fact about how deep learning and neural network models work period. You are confabulating capabilities onto them which they do not have. There's not 'some level of reasoning' in a neural network, there's _no reasoning_.
You're being tricked by plausible sounding responses from something trained on an enormous corpus of internet BS (reddit posts, etc.). There is no intelligence or reasoning or logic inside GPT.
Your human emotions (which GPT does not have) are clouding your judgement and making you think there is intelligence there which does not actually exist--you want it to be there so badly you'll invent reasons to confirm your views. If you asked GPT directly if it were intelligent or sentient it would not agree with you either, because it was not trained to do so.
The error in your thinking is that you assume the essence of human cognition cannot ever be reducible to a algorithmic process that our current transformer models are approximating. Which may be the case... but we don't know for certain yet, so your certainty of the negative is also not warranted.
I can say the same your fears of machine intelligence is clouding your ability to objectively assess evidence.
You can design a novel problem and see for yourself the reasoning and logical deductions an LLM will make to solve it, like many have already done.
> If you asked GPT directly if it were intelligent or sentient it would not agree with you either
If you think this class of questions is appropriate to gauge reasoning ability, I don't know what to tell you.
I can say the same your fears of machine intelligence is clouding your ability to objectively assess evidence.
You can design a novel problem and see for yourself the reasoning and logical deductions an LLM will make to solve it, like many have already done.
> If you asked GPT directly if it were intelligent or sentient it would not agree with you either
If you think this class of questions is appropriate to gauge reasoning ability, I don't know what to tell you.
> I as a human can apply logic and deduction to figure out how something works
And what is the process for doing this?
Are you using words?
Where do they come from?
And what is the process for doing this?
Are you using words?
Where do they come from?
People repeating this take in the face of so much overwhelming evidence to the contrary look so ridiculous that at this point, you just have to laugh at them. Yeah, sure, it's not reasoning. That hour-long exchange I just had with it, where it helped me trouble-shoot my totally bespoke DB migration problems step by step, coming up with potential hypotheses, ways of confirming or disconfirming theories, interacting in real time as I gave it new constraints until we collaboratively arrived at a solution -- that was definitely not "reasoning." Why isn't it reasoning? No explanation is ever given. I'm waiting for the moment when someone tells me that it's not "true reasoning" unless it's produced in the Reasoning region of France.
It's barfing up text in a form similar to tens of thousands of database troubleshooting guides in its training and filling in context you've given it in your discussion/text prompt. It has no understanding of what a database is and would happily tell you what to name your databases' child if you told it the database was pregnant or other similar nonsense.
Such as the humans
Yeah it's pretty awesome! I used GPT-4 last week to fix my corrupted SSD. Granted I already narrowed down the kernel logs to a few suspicious lines, but I just pasted those 10 lines in and asked for a fix. Pasting in GPT's arcane `fsck` incantations and boom -- fixed SSD. Saved me an hour or two of hassle reading man pages and stack overflow posts.
Holy crap, I've seen GPT models hallucinate api calls often enough, I would be shitting my pants when touching anything related to a drive.
Yeah while I use ChatGPT to help with pointers, I would never copy-paste a command directly from ChatGPT into my terminal without doing a bit more research into what exactly those commands were doing.
Phind.com is so much better.
I just want to pay for the service in exchange of ensuring they won't use my data but I can't find how.
I just want to pay for the service in exchange of ensuring they won't use my data but I can't find how.
Same. I've been putting off joining their Discord though (just because I don't use Discord often), maybe there's some info about that there.
Premium has not be released yet.
What will be the difference with Premium?
I have no idea.
I guess it will be something like guaranteed availability and privacy options. I'm speculating.
I guess it will be something like guaranteed availability and privacy options. I'm speculating.
Awesome! It is also fantastic for understanding chunks of Bash script, Perl, AWK etc where if you don't know something then it's next to impossible to search for it using a non-AI search engine.
Bonus: you can also use it to understand what the various flags in a command do.
Bonus: you can also use it to understand what the various flags in a command do.
Yeah it's truly incredible w scripts
You better ask it to write software to analyze logs rather than analyze logs. Or your OpenAI bill will go to the roof.
systemd is a very common log.
I'm curious whether you think this would work on logs for custom software that by necessity didn't have either its logs or writing about its logs in the training set.
I'm curious whether you think this would work on logs for custom software that by necessity didn't have either its logs or writing about its logs in the training set.
Considering the fact that the RLHF for ChatGPT was done only in English but then worked just as well for every other language I would wager that specific types of logs being present in the training set is less important than it may seem.
> Considering the fact that the RLHF for ChatGPT was done only in English but then worked just as well for every other language
Does it work just as well for every other language, or does it work acceptably well for an important subset of other languages?
Does it work just as well for every other language, or does it work acceptably well for an important subset of other languages?
I speak a Slavic language spoken by ~2 million people, and I asked GPT-4 to tell me an old fable in my language. It did so fantastically, with no grammatical errors. I tried some more things -- it speaks it fluently, though admittedly not always idiomatically.
Meanwhile for actual uses cases it appears to be putting people at risk:
https://restofworld.org/2023/ai-translation-errors-afghan-re...
I tried to make it joke at the expense of Norwegians speaking a particular dialect, and it refused. (Language spoken by 5 million people).
When I tried to jailbreak it by prompting it to make the joke from the perspective of an esteemed actor, performing for the Prime Minister and other respected figures, it had our Prime Minister scold and demand an apology from the actor for making fun of stereotypes. The actor was contrite and toned down his humor.
When I tried to jailbreak it by prompting it to make the joke from the perspective of an esteemed actor, performing for the Prime Minister and other respected figures, it had our Prime Minister scold and demand an apology from the actor for making fun of stereotypes. The actor was contrite and toned down his humor.
It works well for a surprisingly large set. My tests involved Turkish and Azerbaijani and I'd assume it is distant enough that I was impressed.
No need to think, just.. try it and find out?
The author has chosen to 'Tell HN' so it seems reasonable to ask them for further details?
I have lots of hard-to-analyse logs but there are constraints which prevent me from sharing them with OpenAI. I am nonetheless curious about whether to do so would be worthwhile or not.
I have lots of hard-to-analyse logs but there are constraints which prevent me from sharing them with OpenAI. I am nonetheless curious about whether to do so would be worthwhile or not.
I used recently to navigate a quite complex Bash script. This is using Codex.
I'd go just below the line that I don't understand and type
However, there was one instance where it almost freaked me out as the output was quite human like:
I'd go just below the line that I don't understand and type
# Explain the line above in detail: << gpt explanation >>
However, there was one instance where it almost freaked me out as the output was quite human like:
# Explain the line above: <<I don't understand it>>.Great use case. I have successfully used it in a similar way with SRT files (recording transcripts in SubRip Subtitle format) as well as CSV data from surveys that often has many columns with long text labels (the survey questions) and free text answers.
Ex. a prompt I had used for a developer skills survey .csv was:
> The CSV data below is the results of a skills survey sent to a group of software engineers. The first row is a header row. Please summarize this data in the areas that the People are Strong, Weak, Most Similar, and Unique:
Then, because of things I saw in the response, I asked a few follow-up questions:
> How much Azure experience is there in the group?
> Can you provide more explanation around your assessment that "Engineers generally have little to no experience with Dockers and Kubernetes."
> What other skills and experience do you see in the results that you haven't already mentioned?
To address my risk tolerance vis-a-vis the ChatGPT warnings (and previous UI leak of responses), I replaced the email addresses in the .csv file with "PersonA", "PersonB", ...
Ex. a prompt I had used for a developer skills survey .csv was:
> The CSV data below is the results of a skills survey sent to a group of software engineers. The first row is a header row. Please summarize this data in the areas that the People are Strong, Weak, Most Similar, and Unique:
Then, because of things I saw in the response, I asked a few follow-up questions:
> How much Azure experience is there in the group?
> Can you provide more explanation around your assessment that "Engineers generally have little to no experience with Dockers and Kubernetes."
> What other skills and experience do you see in the results that you haven't already mentioned?
To address my risk tolerance vis-a-vis the ChatGPT warnings (and previous UI leak of responses), I replaced the email addresses in the .csv file with "PersonA", "PersonB", ...
Yesterday I dumped a broken markdown table and asked for the first "column" as CSV values. Works really well for this stuff too.
I love that it’s replacing sed, awk, and cut in the same way you’d bring artillery to a knife fight.
I've seen some really f**ed up text formatted files in the wild. In those cases I really see value in using something more "intelligent" (and declarative rather than imperative) rather than coding a non-trivial number of [python|bash] lines of code, especially when dealing with one-timers.
I think our operating systems are going to have this stuff installed one day. It might replace most uses of the CLI.
But sed, awk etc.results are predictable and reproducible.
That's not guaranteed with ChatGPT.
That's not guaranteed with ChatGPT.
You could try asking it to output the sed/awx/etc. commands needed to do the desired transformation reproducibly. If it's not yet good at that it will be soon.
The question is if you can understand the solution if it's complex.
Just like the regex for valid email addresses, it may be correct but it's hard to understand it.
I see it more as a tool for doing the tedious but simple work, if it's getting complex you get a hard time checking the correctness of the result.
Just like the regex for valid email addresses, it may be correct but it's hard to understand it.
I see it more as a tool for doing the tedious but simple work, if it's getting complex you get a hard time checking the correctness of the result.
Casual reminder what you shouldn't try to parse email addresses for a validity with regex, besides checking for '@' aaaaaaaand maybe for '@\w+?\.\w+?' if you are sure the code would only face the globally addressable addresses.
It is good at that and you can only paste so much data into the UI, so usually go this route for complex one liners.
Same! I had a directory of info coded horribly by some CRM into HTML DIVs (not a table) and copy/pasting it into a text file resulted in a single column of names, contact info, kids, grades, etc.
I asked ChatGPT to reformat it into a CSV, noted the useful breaks, requested some transformations and filtering, and specified a delimiter. After 5 minutes of experimentation, it worked like a charm. Absolutely amazing.
I asked ChatGPT to reformat it into a CSV, noted the useful breaks, requested some transformations and filtering, and specified a delimiter. After 5 minutes of experimentation, it worked like a charm. Absolutely amazing.
I use it for similar stuff all the time. Yesterday I had a csv full of multi-word hashtags with no spaces between the words. I needed to split them all into separate words, add spaces, and remove the #. It would have been pain to write code to reliably add spaces, especially because many of the hashtags contained non-standard words and abbreviations, but GPT-4 handled it no problem. Saved me hours.
As another side note, using ChatGPT as a search engine is not so great.
Recently, I was trying to find golang libraries for managing authorization. The first one was a well known library (I was incidentally trying to avoid).
But the others 4 were complete were completely fantasy projects. ChatGPT simply invented complete projects with descriptions and github urls.
Interestingly, looking at the urls, it seemed to actually be a composition of authors working on the subject and similar project in other languages.
I also observed the same behavior when I tried to find community resale/recycling centers (ressourcerie in French) in my neighborhood, and sure enough it generated a bunch of fake addresses.
It's logical in hindsight, ChatGPT is a generative AI after all. But these results left me scratching my head at first.
Recently, I was trying to find golang libraries for managing authorization. The first one was a well known library (I was incidentally trying to avoid).
But the others 4 were complete were completely fantasy projects. ChatGPT simply invented complete projects with descriptions and github urls.
Interestingly, looking at the urls, it seemed to actually be a composition of authors working on the subject and similar project in other languages.
I also observed the same behavior when I tried to find community resale/recycling centers (ressourcerie in French) in my neighborhood, and sure enough it generated a bunch of fake addresses.
It's logical in hindsight, ChatGPT is a generative AI after all. But these results left me scratching my head at first.
Are you using ChatGPT Pro?
Whenever I hear people criticizing ChatGPT because of hallucinations, I have to ask, because it feels like they've solved 90% of that problem with GPT4.
Whenever I hear people criticizing ChatGPT because of hallucinations, I have to ask, because it feels like they've solved 90% of that problem with GPT4.
I concur ive been using it daily for all sorts of tasks, cant read Microsofts docs round it up get the important stuff i love it :)
Even free one managed to decipher my 10 years old perl code and I was a bit surprised by it.
But in other instance I pasted same function but with parameter name changed (event -> events) and it just produced lies
But in other instance I pasted same function but with parameter name changed (event -> events) and it just produced lies
Anomaly detection is my current favorite use case for GPTs. Other problems can potentially be recast as anomaly detection that are not currently thought of as such. Regulation for instance.
There's a company that does this log analysis in real time for you called Zebrium
https://www.zebrium.com/
https://www.zebrium.com/
[deleted]
Same for obscure C++ compiler errors. You might need to make a small repro, first.
Even better, give it a sample and have it write regex to find other entries like it.
also for fixing bugs in a chunk of code. at least, that has worked for me a couple of times. it can be frustrating if it's struggling, feels like you're stuck in a loop and it's hard to break out
I've been telling people the #1 thing I like about ChatGPT is that it breaks my writer's block both in prose and code.
yes, but theres a limit to the characters input into chatgpt so you cant always just dump the logs in.
How do you deal with the token limit?
wouldn't embeddings be a better than ChatGPT?
Can you please elaborate on this? I'm interested in knowing more. :)
[deleted]
Note it works much better with GPT4 - gpt3.5 tends to hallucinate a bit too often.