Does anyone have any insight (or informed guesses) that might explain the strange downward "spike" that was consistently observed at dimension 196 in OpenAI's text-embedding-ada-002 model?
Can anyone comment on how the limits on GPT-X’s token space translate to limits on its vocabulary (with corresponding limits on understanding input and generating output)?
For example, is GPT-4’s list of ~100k tokens sufficient to understand and generate every non-obsolete word in the English language (per, say, a standard dictionary)? Or even every word in the training data?
If not, do we have examples of ordinary words that it is impossible for GPT-4 ever to understand or generate? What happens when it encounters those words and is unable to tokenize them; are they simply ignored (eg omitted from the input vector, or set to 0 or some sort of null token)?
Why was my Realtime Database reported bandwidth lower than average between September 2016 and March 2017?
For our bandwidth calculations, we normally include SSL encryption overhead (based on layer 5 of the OSI model). However, in September 2016, we introduced a bug that caused our bandwidth reporting to ignore encryption overhead. This might have resulted in artificially low reported bandwidth and bills on your account for a few months.
We released a fix for the bug in late March 2017, returning bandwidth reporting and billing to their normal levels.
This analysis from Ben Thompson (Stratechery[1]) just yesterday would be a great place to start:
"Imagine a Twitter app that, instead of a generic Moment that is little more than Twitter’s version of a thousand re-blogs, let you replay your Twitter stream from any particular moment in time. Miss the Oscars gaffe? Not only can you watch the video, you can read the reactions as they happen, from the people you actually care enough to follow. Or maybe see the reactions through someone else’s eyes: choose any other user on Twitter, and see what they saw as the gaffe happened.
What is so powerful about this seemingly simple feature is that it would commoditize “live” in a way that is only possibly digitally, and that would uniquely benefit the company: now the experience of “live” (except for the shock value) would be available at any time, from any perspective, and only on Twitter. That such a feature does not exist — indeed, that the company’s stated goal is to become more like old media, instead of uniquely leveraging digital — is as good an explanation for why the company has foundered as any."
It was a great episode and is worth the listen for those interested.
On Swift adoption at Apple:
"The Swift team itself has specific goals they need to achieve before there can be truly, across-the-board adoption at Apple. ABI stability is the number-one thing [35:30] that prevents framework developers, for example, from adopting Swift. That's a really important thing. That's one of the reasons it's always a really high priority. Swift has been adopted by application developers and other things. The Dock is public. Swift Playgrounds app is public. The Music app in iOS is publicly known. So there are definitely some big adopters.
More broadly though, the big problem is that I think, I won't speak for everybody but many, many people doing [36:00] Objective-C development at Apple are chomping at the bit. They want to be using Swift. It's really just a matter of getting the technology problems solved and checking off the things that are holding people back. It's not about people dragging their feet and not wanting to use it."
On whether to adopt Swift now:
"I don't [1:14:30] think Objective-C is going to go away anytime soon. Apple still supports C and C++ and there's no obvious benefit of dropping Objective-C, and obviously they have a ton of Objective-C code themselves."
He also described Apple's approach to some of the strategic questions that arose early on, such as whether to just invest in making Objective-C better instead of introducing Swift, and the various trade-offs involved.
"It is our goal to train a trillion-parameter model on a trillion-word corpus. We have not scaled our systems this far as of the writing of this paper, but it should be possible by adding more hardware."
In telling the other side of the story, the Dash developer published his (apparently secret) recording of a 7-minute long telephone call he had with a person at Apple who purports to be speaking on behalf of Phil Schiller.[1]
Leaving aside whatever inferences might be drawn from the fact that the developer saw fit to record and publish the recording in the first place, here's a brief summary of what was said:
- There was at least one other developer account "linked" to the Dash developer account. In this context, "linked" means that the accounts "shared the same details": they were enrolled in the Apple Developer Program "with the same credit card number", and "used the same test devices".
- Apple says that at least one of those other developer accounts "definitely had fraudulent activity": "It was not your direct account but it was a linked account." Warnings about fraudulent activity were sent to the linked account. No warnings were sent to the Dash account.
- The Dash developer asked: "Why didn't you notify me beforehand though, and let me know that an account that's linked to mine is doing fraudulent activity, so I can do something about it?" The answer is "because they were linked"; "we see them as the same entity". "If we have accounts that have enrolled in our program using the same credit card, they are the same legal entity; they are the same." So Apple believed they had notified the Dash developer because Apple believed the same person was behind the linked accounts. (Prudence would dictate notifying all accounts at risk of termination; we'll see what happens in future.)
- Apple's position is that no mistakes were made. The Dash developer account was linked to an account with fraudulent activity based on the facts known to Apple.
- The Dash developer says (in his blog post) that he "helped a relative get started by paying for her Apple's Developer Program Membership using my credit card" and "handed her test hardware that I no longer needed".
- Apple says they are "working with" the Dash developer to "unlink the accounts", which (I speculate) may involve some attempt to verify the Dash developer's claim that the linked account was used only by a relative and not by him. If the accounts are unlinked, there would be no reason for the Dash account to remain closed.
> If I had an algorithm that produced halfway decent results in an automated fashion, I wouldn't use it to get into the legal technology market. I'd set myself up as a litigation funder and make boatloads of money with accurate valuations of potential investments.
Legalist is a recent YC startup aiming to do exactly that:
"In some Australian cities, in the central business district, the traffic light push buttons don't work at certain times — for example, 7 am to 7 pm, Monday to Wednesday, and 7 am to 9 pm, Thursday to Saturday.
The rationale is that between these busy hours, the road traffic is relatively constant, and also, the pedestrian crossings are in continuous use. So the lights do their own thing.
However, the traffic light push buttons do work outside these hours, including all day Sunday."
The public beta of OpenAI Gym, a toolkit for developing and comparing reinforcement learning algorithms, was released last month: https://openai.com/blog/openai-gym-beta/
Some of these statements are incorrect or incomplete in a way that might hinder the casual reader.
The families on Nauru are free to leave and return to their countries of origin any time they choose. But all of the members of those families claim to have fled their countries of origin because they face a real risk of persecution or significant harm in those countries, and they claim not to wish to return because of that risk. It is not possible to assert whether the members of those families are or are not refugees until their claims have been processed. Papua New Guinea found at least 56% of claimants to be refugees in need of protection[1], and is still processing some claims. As at 8 October 2015, Nauru had found 75% of claimants to be refugees in need of protection.[2] Nauru has not finished processing the balance of the claims.
While those on Nauru are technically free to participate in the Nauruan community (and the article does not assert otherwise)—in the sense that they can now leave the detention compound and wander around the island—they have no rights other than those granted to them by virtue of their "regional processing centre" visas (which are granted upon being transferred to Nauru from Australia). Nauruan law requires that asylum seekers be provided with, amongst other things, adequate food, clean and sufficient clothing, and access to medical facilities,[3] all of which in practice can only be obtained at the regional processing centre (funded by Australia). They are not allowed to take food or water with them and they are subject to searches upon entry and exit. Until 5 October 2015, it was a condition of the RPC visa that the visa holder reside at the centre and not leave the centre without approval. Most of the asylum seekers on Nauru never consented to being taken to Nauru or being detained in those circumstances. The condition was lifted when the High Court of Australia heard a challenge to the legality of Australia's participation in the detention of those at the centre.[4] (Some retrospective laws were passed and the challenge failed.)
It is true that the measures taken by the Australian government to provide for offshore processing in Nauru and Papua New Guinea have deterred people from taking a perilous voyage to Australia by boat, whether to claim protection or otherwise. It is true that deaths at sea have reduced from several hundred a few years ago to almost nil.
The controversy behind these laws is whether that end is sufficient to justify the means, namely, Australia's involvement in bringing about the circumstances in which asylum seekers now find themselves on Nauru and Papua New Guinea.
It is not correct to say that "the only people who have been harmed are those that have self-harmed". Independent bodies have conducted their own investigations and found that many asylum seekers who have not self-harmed are suffering from inhuman and degrading treatment in Nauru and Papua New Guinea.[5] There are many other credible and significant reports of harm.
It is an oversimplification to say that "[t]he laws have broad support amongst Australian voters". The policy is divisive, complicated, and there are many different views about its merits.
Many refugee advocates who are qualified as Australian lawyers choose to act for some of these people on a pro bono basis and do not "prosper" from it. In some instances, where an Australian court finds that the Australian government has engaged in unlawful conduct in relation to a particular matter, the court may order the Australian government to pay costs. There are vanishingly few other instances in which any lawyer for a refugee would receive any money or material benefit.
While it is true that Australia "maintains a sizeable immigrant intake", that has nothing to do with Australia's humanitarian or refugee intake, which is treated separately to its migration program. The intake for the migration program has been at about 190,000 for the past few years, whereas the humanitarian intake is typically about 12,000 (or about 6% of the migration program).[6] The refuge offered to an additional 12,000 Syrian refugees last year is commendable.[7] The fact that the United States may not have done the same is not to the point.
I would have found it easier to review these Apply HN threads if each applicant had completed the full YC Fellowship application and posted that in their thread as well. For many of the existing threads, there's just not very much to go on, but you have to read each thread before you can work that out. You can feel the inefficiency begin to drag at you as you go through the review process.
Question: Why don't you require all Apply HN applicants to complete the full YCF application, and open those applications to the HN community? Each YCF application includes an option to generate a corresponding Apply HN thread. Comments from the HN community are made available to YC reviewers. The applicant consents to disclosure by ticking the Apply HN box.
Anyone who posted an Apply HN thread could have just lodged their own YCF application anyway, which the YC team would have had to review and determine in the usual way—but completing the application mandates a baseline level of information about the applicant's team and startup. So why not leverage the existing process and help the HN community help you to review them.
Unfortunately none of that will help you choose applicants from among the existing Apply HN threads (sorry), but it might be worth considering for the next stage of the experiment.
That isn't the case he ran before the Working Group on Arbitrary Detention though.[1]
Judging from his written submissions, he says that the UK has caused him to face "an impossible dilemma", namely, choosing between confinement in the embassy or giving up his (inalienable) right to non-refoulement as a refugee. Apparently the WGAD has previously held that putting someone in that position involves unlawful detention. So it turns out that is the significance of the fact he is in the Ecuadorian Embassy.