150M at 5B revenue is not great: that's 3% margin!
The bigger issue is that console manufacturer revenue is highly cyclical. This is hard to see in e.g. Xbox and Sony since they are both part of a larger conglomerate, but really obvious for nintendo.
You generally have a cycle of
- "Launch": high marketing costs, low/negative HW margins
- "Mid-cycle": lowering manufacturing costs, large game sales, high margin DLCs
- "end-of-cycle": falling HW sales, fewer exclusives (-> preparation for next gen), fewer consumers (-> waiting for next gen). Here you usually have maximum profits since you don't subsidize HW and marketing is minimal (platforms are already locked in)
Generally you have to establish a big userbase during the mid-cycle such that you can levarage it during the late-cycle to be able to afford next-gen. Xbox has the big issue that their mid-cycle was catastrophic, which means they now don't have the console base to get into the next generation:
If they have 3% margin _right now_ in the end-of-cycle where marketing and development costs are at their lowest, this does not bode well for the overall health of the business.
The issue with having a "no answer" option is that you implicitly add a decision problem into your test that depends on the "cost" of answering wrong.
Specifically, your model now has two "correct" classes p(class=y|x) and p(class=⊥|x). This makes the results ambiguous.
The way you resolve this is by adding in a cost of missclassification and a cost of answering wrong.
L(y, y') =
0 if y=y'
l_err if y≠y' and y'≠⊥
l_⊥ if y' = ⊥
You can then estimate the expected error over your dataset.
Notice that this now gives you additional degrees of freedom: Depending on how expensive answering wrong is compared to not answering at all, your predictor might be really bad or really good.
This means when benchmarking with a "no answer" action, you are often not actually benchmarking whether the model works well or not, but rather are benchmarking how well the model _happens_ to agree with the class-error weight you (implicitly) chose in your model.
Because the size of the attention matrix depends on the number of tokens (this is what makes attention N^2).
If you don't care about having a flexible number of input tokens (e.g. in image processing) you can learn a fixed routing matrix. This is known as an MLP mixer https://arxiv.org/pdf/2105.01601 : you have one layer that processes each token in isolation ("vertical MLP") but ignores the inter-token connections, followed by a layer that combines between tokens ("horizontal MLP") that treats the internals of every token identically.
Nothing is stopping them, it's just not worth it: Have a look at e.g. vast.ai's pricing (https://vast.ai/pricing).
The V100 (2017 -> 9 years old) can be rented from $0.02 to $0.37/h (right now I can find a V100 with a Xeon Gold 6140 and 48GB RAM for $0.165/h). Let's assume the guy you rent it to pins it at its 250W TDP and let's ignore the running costs of CPU/RAM/etc...
Then you draw 1/4 kwh for that compute hour. The industrial electricity prices in the US vary between 7.5 and 25 ct per kwh (depending on state, time of day, etc...), so at 100% efficiency, assuming nothing ever breaks, and the CPU consumes 0W you earn about 14ct/h.
And remember: V100s hours are sometimes sold at 1/10th the price.
If I pick average conditions you need to start thinking of whether it is worth it to rent them out: Usually it isn't unless you have them anyways and just sell idle capacity.
It's barely worth it to run them in a pure "is it profitable" sense, if we also account for the opportunity cost of taking up a slot in your datacenter it seizes to be worth it really quickly.
Regarding specifically depth anything: You're not running this on a microcontroller.
In general, CNNs still reign supreme on microcontrollers since you have a way lower peak memory demand which is what usually kills you. Here in this case you have a couple of _kilobytes_ of SRAM, potentially extendable to a couple of megabytes of PSRAM.
Even for small CNNs you often need to do some quite complex interleaving of layers (i.e. running parts of layer 1 and layer 2 in parallel interleaved to take advantage of the downsampling of CNNs) to keep performance and memory impact reasonable (see e.g. https://openreview.net/pdf?id=2O8qbyxH6X).
Think more "image classifier" less "run an image to image transformer". For depth anything, a single layer's activation is probably significantly larger than the available SRAM (I think it is (224/16)^2 patches each with activations [48, 96, 192, 384] for depth anything small: You aren't running this.)
It works the same way:
softmax is essentially just applying the normalization to the vector exp(x).
From an "engineering" POV this effectively ensures that the vector you normalize has strictly positive entries, so the result ends up being a proper distribution.
From a theory POV you get softmax like distributions (Gibbs distributions) by trying to balance following some energy E(x) and the entropy of the distribution.
In essence the softmax is the answer to "I try to follow the maximum of a function E(x) but I need to maintain some level of uncertainy".
The balancing coefficient between entropy and picking the maximum of the function is called "temperature" (following the behavior of particles in a physical system: The colder the system, the lower the chance of having particles randomly walk away from the minimal energy state).
specifically, the temperature is
softmax(x/temp)
if you draw temp->0, your softmax slowly becomes an argmax (with temp=0 being a literal argmax). If you increase the temperature, you are closer to the "random fluctuations" leaving more room for sampling x values that are not the maximum of x. (this is why e.g. LLMs become deterministic as you decrease temp->0)
Using a different base other than e implicitly changes the temperature:
N^x = exp(ln(N) x)
The normalization works the same since you are still dividing a positive value N^x by the sum of all alternatives sum(N^x_i), which is a normalization by design
I'm what way is that different?
You return early and the call Cascades up the call chain until you handle it (otherwise it's always an "either" results)
In practice you use something like an exception monad, which makes this a lot more ergonomic since you don't need to carry a case distinction around for every unwrap: an exception monad essentially has an implicit passthrough that says "if it's a value, apply the function, if it's an exception just keep that".
You only need to "catch" the exception if you actually need the value.
I'm this case the exception monad is not that different from annotating a function with "throws": your calling function either needs it's own throws (=error monad wrapper) in which case exceptions just roll through, or you remove the throws, but now need to handle the exception explicitly (=unwrap the monad).
Of course you can: you just have to define it in your type.
The output set becomes a union type of the normal output and whatever you want as an exception.
If you write this as a monad, your get very similar syntax to procedural code.
Effectively none. The US has a huge trade deficit with Germany/Europe so there is practically never a case where the US receives gold from Germany: It's always more then offset by the deficit.
The equivalent for the US would be the consumption goods that are already flowing into the US. I.e. US gets goods but doesn't sell enough to Germany, so the difference to maintain the total exchange rate is the Gold.
That's also why it was trivial for france to repatriate its gold compared to germany: Germany holds about 10x the amount of gold in the US compared to France (France was ~120 tons, Germany is roughly 1200 tons: France earned its gold through different trade).
That's also why it is such a complex thing to repatriate German reserves: France took almost 1 year to repatriate its gold. For Germany, the efforts would be decade spanning (though maybe with recent changes there is a little more urgency).
Germany already repatriated about half of its gold reserves between 2013 and 2017 from paris and new york to frankfurt.
There has been a recent (as in "18th of march" recent) petition to the Bundestag to repatriate the gold.
The reason not to repatriate the remaining gold back then is because Germany has substantial trade with the US, which is why Germany held gold in new york to begin with: It's the easiest way to resolve USD-Euro currency exchange at the central bank level (this is also why germany got rid of the paris gold reserves: with the euro you don't need currency exchange anymore).
Also, as you mentioned, the idea of "officially" repatriating gold with the current administration is quite dicey. It is very possible that the correct way of resolving this is to just stop buying gold in new york and let the currency exchange flux deal with the slow unwinding of the reserves without explicit repatriation.
Whether the US is capable of hiding their maleficence or not should not be an indicator of whether it is safe to deal with them. If your indicator for the US being a good partner in _anything_ is that "well we did corrupt things in the past, but people didn't use to care about it", then the US is still not a good partner.
It's not like the US has never e.g. openly threatened NATO allies with war: There is quite literally a standing law that allows the US president to invade the netherlands if any US military personnel is ever detained by the International Criminal Court.
This law has been on the books for over 20 years and has the publically announced intention to prevent the US from being prosecuted for all the other atrocities committed in e.g. Iraq. This bill was supported by both democrats and republicans.
The reality is that the US' stance towards the rest of the world has not changed with the recent administrations (nor would I expect it to: Trump does not happen in a vacuum). What did change was willingness of the rest of the world to act on the US' actions.
There were plenty of models the size of gpt3 in industry.
The core insight necessary for chatgpt was not scaling (that was already widely accepted): the insight was that instead of finetuning for each individual task, you can finetune once for the meta-task of instruction following, which brings a problem specification directly into the data stream.
Assuming this is real: Why do you think anthropic was put on what is essentially an "enemy of the state" list and openai didn't?
The two things anthropic refused to do is mass surveillance and autonomous weapons, so why do _you_ think openai refused and still did not get placed on the exact same list.
It's fine to say "I'm not going to resign. I didn't even sign that letter", but thinking that openai can get away with not developing autonomous weapons or mass surveillance is naive at the very best.
It might be that they pay less for anthropic depending how many tokens are generated by each model: total cost is token cost times number of tokens. I haven't checked gpt5, but it is not impossible that price wise they might be very comparable if you account for reasoning tokens used.
> keep in mind that he is literally the only one who talks about the company that way from everyone whose ever worked there (many, many have left for various reasons since).
That's not true.
There's been lots of criticism coming at Valve's management style over the years: There are many people who _hate_ it, but also many people who love it. It really depends on the type of person you are. If you really really dislike bureaucracy Valve's a great place to be, but also don't expect them to enact any quality-of-life systems: If your coworker wants you dead, then that's something you yourself have to deal with.
This is also why you see a lot of different opinions when looking through e.g. glassdoor: Some people that do the murdering love it there and thrive, but that system only works if people are there that can be murdered. Some people I know have described the atmosphere as "prison yard style": You trade rigid bureaucracy against a "you have to know who you have to know" bureaucracy. Richard Geldreich's account lines up pretty well to what I have heard about valve's emergent self-organization system.
The bigger issue is that console manufacturer revenue is highly cyclical. This is hard to see in e.g. Xbox and Sony since they are both part of a larger conglomerate, but really obvious for nintendo.
You generally have a cycle of - "Launch": high marketing costs, low/negative HW margins - "Mid-cycle": lowering manufacturing costs, large game sales, high margin DLCs - "end-of-cycle": falling HW sales, fewer exclusives (-> preparation for next gen), fewer consumers (-> waiting for next gen). Here you usually have maximum profits since you don't subsidize HW and marketing is minimal (platforms are already locked in)
Generally you have to establish a big userbase during the mid-cycle such that you can levarage it during the late-cycle to be able to afford next-gen. Xbox has the big issue that their mid-cycle was catastrophic, which means they now don't have the console base to get into the next generation: If they have 3% margin _right now_ in the end-of-cycle where marketing and development costs are at their lowest, this does not bode well for the overall health of the business.