Everything is a linear model(danielroelfs.com)
danielroelfs.com
Everything is a linear model
https://danielroelfs.com/blog/everything-is-a-linear-model/
89 comments
In the 70's in CS grad school at USC, I wrote an adaptive least squares cubic spline fit routine. Kept subdividing intervals and fitting a least squares cubic spline in each interval until a criteria was met.
Fun times.
Don't have the code or writeup any more.
Fun times.
Don't have the code or writeup any more.
Can you recommend some resources to go from step 3 to 4?
So far I have been successfully using GLMMs where appropriate, but then jumped to implementing completely arbitrary models by fitting them to the data (plus bootstrapping).
But that takes ages for most problems as you say
So far I have been successfully using GLMMs where appropriate, but then jumped to implementing completely arbitrary models by fitting them to the data (plus bootstrapping).
But that takes ages for most problems as you say
GLMs are a non linear transformation on an output followed by linear modelling. They are referred to as “linear models” but you might as well then also consider NNs as linear models, or any model at all which ends with addition as the final step.
> Splines can easily take care of that.
within set boundaries.
within set boundaries.
"You can make linear model non linear by adding non linearity" is kind of vacuous statement, isn't it
No no, linear just means some expression of the form
This linear combination "core" of the model can be used directly (in which case it is geometrically a hyperplane), but you could also use non-linear transformation of the inputs or the outputs.
E.g. x1 could be a spline basis function, or log-transformed data, or anything.
Similarly, the output y can be passed though some non-linearity to force a particular output (like in logistic regression). Hence the statement is not vacuous
y = b1*x1 + b2*x2 + ... + bp*xp
This linear combination "core" of the model can be used directly (in which case it is geometrically a hyperplane), but you could also use non-linear transformation of the inputs or the outputs.
E.g. x1 could be a spline basis function, or log-transformed data, or anything.
Similarly, the output y can be passed though some non-linearity to force a particular output (like in logistic regression). Hence the statement is not vacuous
Well yeah, but what is the non trivial statement here? Linear model is linear, and if you add to your model non-linear steps (such as pre- or post- processing) then it is not longer linear and can handle non-linear data. In the same spirit, you could say that GPT4 is a linear model (token logits) with a very large preprocessing (transformer).
Maybe what OP meant is that particular non linearities (splines) allow to keep some goodies from the linear model in non-linear settings. Still it's not clear for me in which settings exactly such models are good enough
Maybe what OP meant is that particular non linearities (splines) allow to keep some goodies from the linear model in non-linear settings. Still it's not clear for me in which settings exactly such models are good enough
This sounds really cool but was hard to digest for me as a ML Engineer who came into work just around deep learning and DNNs.
Is there some go-to practice material I could look at?
Splines I haven't touched since numerical computing exercises in school.
Is there some go-to practice material I could look at?
Splines I haven't touched since numerical computing exercises in school.
Simon Wood's Generalized Additive Model book.
Or, for a more approachable treatment, Semiparametric Regression with R by Harezlak, Ruppert, and Wand. A middle ground between Wood's book (which is comprehensive but can dip into math that's way over my head at times) and H/R/W is Semiparametric Regression by Ruppert, Wand, and Carroll.
I have also heard great things about Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it. His other writing is fantastic though.
I have also heard great things about Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it. His other writing is fantastic though.
> Frank Harrell's Regression Modeling Strategies which uses a slightly different approach (still spline-based though), but I haven't read it.
Very very good indeed, with the exception that he basically ignores compute time and efficiency. I learned a bunch, but applying his approach to the kinds of datasets I deal with (much larger and with pretty strict compute budgets) was very difficult.
Very very good indeed, with the exception that he basically ignores compute time and efficiency. I learned a bunch, but applying his approach to the kinds of datasets I deal with (much larger and with pretty strict compute budgets) was very difficult.
If you know DNNs, you'll find the introduction of splines in Semiparametric Regression in R very intuitive - splines are introduced using what the authors call a "truncated line basis" but you already know it as the RELU function, just with a bias. Indeed, even the penalization of splines will look extremely familiar: it's basically just L2 regularization to induce smoothness.
You might also enjoy reading the "Neural Additive Model" paper from Hinton's lab, which is basically GAMs using a separate DNN as a "spline basis" for each input variable.
You might also enjoy reading the "Neural Additive Model" paper from Hinton's lab, which is basically GAMs using a separate DNN as a "spline basis" for each input variable.
[deleted]
Widely known in the controls engineering world for the last hundred years as 'anything is linear if you zoom in far enough'
Well, subject to certain constraints.
You aren't guaranteed that your equilibria behave the same in the linearization of a nonlinear system if your Jacobian has any eigenvalues with real part of 0.
https://en.wikipedia.org/wiki/Hartman%E2%80%93Grobman_theore...
You aren't guaranteed that your equilibria behave the same in the linearization of a nonlinear system if your Jacobian has any eigenvalues with real part of 0.
https://en.wikipedia.org/wiki/Hartman%E2%80%93Grobman_theore...
It's amazing how many ideas I've seen being "rediscovered" these days from control theory.
Alternatively, anything is linear if you add enough dimensions.
Statistics is more than hypothesis testing, but you'll get surprisingly far without straying too far from linear models - I remember a Stats prof saying 'most of classical Statistics is GLM [0]'
[0] https://en.wikipedia.org/wiki/Generalized_linear_model
[0] https://en.wikipedia.org/wiki/Generalized_linear_model
It's true, until it's not. It's easy to make these claims, but would you bet your money on it?
Which means it really all just finding hyperplanes that are near the data.
"Classification of mathematical problems as linear and nonlinear is like classification of the Universe as bananas and non-bananas. "
and everything turns around the same principles. For example dynamical models and PID controls.
yet solving a banana, is the only thing we really know how to do. So we end up fitting everything in our banana models.
and everything turns around the same principles. For example dynamical models and PID controls.
yet solving a banana, is the only thing we really know how to do. So we end up fitting everything in our banana models.
I disagree with the implication that linearity is an unnatural concept, it appears whenever the changes being studied are small relative to the key parameters that determine the system. Every system is linear for small perturbations. Even logic gates; in negative feedback they can form passable inverting amplifiers. In a place as big as the universe it is rather common for two things to be very different in scale and yet interacting.
> Every system is linear for small perturbations.
every smooth system, sure, but even continuity is no guarantee of locally linearity.
every smooth system, sure, but even continuity is no guarantee of locally linearity.
I've never actually seen a physical example of a system without a continuous first derivative. For example phase transitions, commonly touted as an example of discontinuity, don't actually occur until the matter has gone a bit over the point and a transition nucleates somewhere. The probability of a phase transition is a continuous function of temperature, with continuous derivatives.
I'm skeptical that discontinuities can exist because, if they did, they'd serve as infinitely powerful microscopes. If there's a discontinuity in nature, it must exist at absolute zero. I don't have a similarly good argument for continuous first derivatives but I do think it's interesting that there are no examples AFAIK.
I'm skeptical that discontinuities can exist because, if they did, they'd serve as infinitely powerful microscopes. If there's a discontinuity in nature, it must exist at absolute zero. I don't have a similarly good argument for continuous first derivatives but I do think it's interesting that there are no examples AFAIK.
Any physical system that makes/breaks contact, such as walking robots. Sure, the foot is not perfectly rigid and technically is a stiff spring. But from a computational perspective, problems still bear all the hallmarks of a discontinuous system such as requiring a very short integration step.
Yes, that's another example of the discontinuity existing in an idealized model, but not real life.
Quantized space is absolutely discontinuous, and tunneling is a discontinuous system. In fact assuming the universe is quantum it’s discontinuous in reality but the appearance is continuous. But these distinctions aren’t super useful unless you’re dealing with these sorts of effects. Continuity is the approximation, discontinuity is the reality. But depending on what’s useful we use the mathematics that help us.
Tunneling currents are continuous in every parameter, although I admit that when you're dealing with particles you have continuous probability distributions with continuously varying means, rather than continua of matter. (But that should count, because all macroscopic variables are expectation values.)
Specifically at quantized space and time levels everything is discrete even distribution functions. There’s no sense in having a continuous spacial distribution sub Planck lengths.
If I wanted to understand and obtain your intuition about linearity, what would you recommend?
I question how strongly I would recommend the years of working with it. :-)
Like pure math? Engineering?
Study it in school?
There are a lot of linear problems that I'm interested in.
Do you use a computer algebra system?
Do you have a few books?
Study it in school?
There are a lot of linear problems that I'm interested in.
Do you use a computer algebra system?
Do you have a few books?
Isn’t the whole point of chaos sensitivity to initial conditions? Where the output even when minuscule differences varies wildly with input?
In chaotic systems, time becomes one of the parameters that has to be small for a linear model to make predictions, but they are still linear for brief times.
yeah, we use a lot of small linear systems to model a non linear system.
We just break the problem in small bananas because that's what we can solve, and then solve for those lil'bananas and call it done.
We just break the problem in small bananas because that's what we can solve, and then solve for those lil'bananas and call it done.
So, Lady Finger, not Cavendish. Got it.
Interesting aspect to see the world from, thanks
My favorite moment in university was in the first class of semester 2, where a prof said "lets look at a really small part of our thing, and assume we apply some force to it. This will make it stretch, lets assume the stretching is linear relative to the applied force". I raised my hand and asked "is this assumption supported empirically?" and he said "no, we know it's not always true, but if we don't make it we can't calculate anything".
At the time I was mad at engineers for being non-scientific, but after a few years I understood the deep wisdom in that. Nonlinear materials exist, and materials we use have nonlinear ranges. We just don't build things from those, because the math is too unwieldy. (except in very very specific edge cases where we spend a lot of money building a very limited thing)
At the time I was mad at engineers for being non-scientific, but after a few years I understood the deep wisdom in that. Nonlinear materials exist, and materials we use have nonlinear ranges. We just don't build things from those, because the math is too unwieldy. (except in very very specific edge cases where we spend a lot of money building a very limited thing)
It's funny: I had the EXACT SAME PROBLEM !!!! Like "how is it science when a prof says: "this equation is too complicated too solve so let's erase these third and forth derivatives and pretend that it's the same - even if it's not - because that's all that we can do"
So I left the "physics" courses and went the "math" courses... only to learn 3 years later how to prove that this kind of approximation is mathematically sound indeed :-D
So I left the "physics" courses and went the "math" courses... only to learn 3 years later how to prove that this kind of approximation is mathematically sound indeed :-D
Should have taken one more course about chaos theory which is the special case when this approach fails.
Can you give some examples of linear and non-linear materials?
Practically all materials behave nonlinearly when stretched or compressed a visible amount. For certain structural applications, though, if that happens we've already failed. Linear models work really well for designing big concrete structures and certain metal structures. Sometimes we try to apply linear models to other things, but that's always kind of fishy.
[deleted]
Linear algebra was the first math class I took in undergrad. I thought the next one was going to be non-linear algebra! But it wasn't.
I knew the linked-in-the-article https://lindeloev.github.io/tests-as-linear/ which is also great. A bit meta on the widespread use of linear models: "Transcending General Linear Reality" by Andrew Abbott, DOI:10.2307/202114
Here is the Python port of test-as-linear developed by George Ho (eigenfoo):
https://github.com/minireference/tests-as-linear/blob/bugfix...
I'm linking to my fork of it because I've added some fixes and filled in some of the missing parts (e.g. Welch's t-test).
I'm linking to my fork of it because I've added some fixes and filled in some of the missing parts (e.g. Welch's t-test).
I read this article when I was in grad school 5 years ago. Absolutely love it and talk about it to this day.
It really makes me frustrated about the ways I was introduced to statistics: brute force memorization of seeming arbitrary formulas.
It really makes me frustrated about the ways I was introduced to statistics: brute force memorization of seeming arbitrary formulas.
You can state and prove theorems with linear models. You can do inference and testing. This means papers and academics naturally love them. And therefore, they are everywhere.
Not the case with non-linear models. We need to throw computers at them.
Not the case with non-linear models. We need to throw computers at them.
I think that's an overstatement. You can consider the nonlinearity as a perturbation of the linear theory, study the topology of the solutions or their symmetries, or sometimes even find exact solutions, like exist for a handful of transistor circuits.
If the model is y = a + b * x + e, then we can precisely state probability distributions of a and b given x and y for reasonably general assumptions about error e.
If the model is y = f(x) + e, where for example f is a neural net, there is not much we can say about the “quality” of parameters of f. That is, we can’t attach confidence interval. We will have to resort to expensive simulations and for most practical applications this is not workable (size of dataset).
What you say is useful in a different context. Local linearization is a very powerful idea.
If the model is y = f(x) + e, where for example f is a neural net, there is not much we can say about the “quality” of parameters of f. That is, we can’t attach confidence interval. We will have to resort to expensive simulations and for most practical applications this is not workable (size of dataset).
What you say is useful in a different context. Local linearization is a very powerful idea.
Maybe not if there are many more parameters in f than there are samples in the dataset, but if there are a handful of parameters in f then a typical step is to take the inverse of the derivative of goodness-of-fit with respect to each fit parameter, to determine the precision with which each parameter was determined by the fit.
Another fun stats X is really Y:
estimating the Area Under the Curve metric (AUC) is equivalent to the Wilcoxon-Mann-Whitney test!
https://rmets.onlinelibrary.wiley.com/doi/abs/10.1256/003590...
estimating the Area Under the Curve metric (AUC) is equivalent to the Wilcoxon-Mann-Whitney test!
https://rmets.onlinelibrary.wiley.com/doi/abs/10.1256/003590...
I find it irritating that the article mentions repeated measures, but does not try them, much less a mixed effects model. Yes, they are linear models with more parameters, but doing it in lm would be a special kind of madness
A cool thing about linear models is that they can be used to model non-linear correlations by using transformations. For example, an exponential function can be made linear if you just take the logarithm.
See the topic of GLMs: https://en.wikipedia.org/wiki/Generalized_linear_model
Not all functions have a trivial transformation.
Doesn't have to be 'trivial'. It could be a trillion-parameter MoE LLM spitting out 10,000-large embeddings. But a lot of that will act linearly and can, in theory, be put into your linear model easily & profitably.
I thought nonlinearity was very important to be able to make a larger model better than a smaller one? Like so important that tom7 made a half-joke demo with it: https://yewtu.be/watch?v=Ae9EKCyI1xU
Well, you can create a non-linear model by piece-wise combining multiple linear models.
The famous ReLU non-linearity is just that - two linear functions joined.
The famous ReLU non-linearity is just that - two linear functions joined.
same thing with any feed forward network too. They are all piece-wise linear in respect to inputs.
Layers reduce resource requirements and make some patterns easier or even practical to find, but any ANN that is a FNN supervised learning could be represented as a parametric linear regression.
Unsupervised learning, that tends to use clustering is harder to visualize but is the same thing.
You still have ANNs, which have binary output, which can be viewed through the lens of deciders. They have to have unique successor and predecessor functions.
Really this is just set shattering that relates to a finite VC dimensionality being required for something to be PAC learnable.
But the title of this is confusing the map for the territory. It isn't that 'Everything is a linear model' but that linear models are the preferred, most practical form.
The efforts to leverage spikey neutral networks, which is a more realistic model of cortical neurons, and which have continuous output (or more correctly the computable reals) tend to run into problems like riddled basins.
https://arxiv.org/abs/1711.02160
Obviously setting rectified linear unit at 0 = 1 resolves to differentiation problem, but many functions may not be so simple
Perhaps a useful lens is how TSP with a discreet Euclidean metric is in NP-complete while the continuous version is in NP-hard.
But it isn't that everything is linearizable, but rather that linearized problems tend to be the most practical.
Layers reduce resource requirements and make some patterns easier or even practical to find, but any ANN that is a FNN supervised learning could be represented as a parametric linear regression.
Unsupervised learning, that tends to use clustering is harder to visualize but is the same thing.
You still have ANNs, which have binary output, which can be viewed through the lens of deciders. They have to have unique successor and predecessor functions.
Really this is just set shattering that relates to a finite VC dimensionality being required for something to be PAC learnable.
But the title of this is confusing the map for the territory. It isn't that 'Everything is a linear model' but that linear models are the preferred, most practical form.
The efforts to leverage spikey neutral networks, which is a more realistic model of cortical neurons, and which have continuous output (or more correctly the computable reals) tend to run into problems like riddled basins.
https://arxiv.org/abs/1711.02160
Obviously setting rectified linear unit at 0 = 1 resolves to differentiation problem, but many functions may not be so simple
Perhaps a useful lens is how TSP with a discreet Euclidean metric is in NP-complete while the continuous version is in NP-hard.
But it isn't that everything is linearizable, but rather that linearized problems tend to be the most practical.
I see what you mean. Though in my mind, and this is clearly subjective, piece-wise linear is at least less strict than a linear model. (With enough ReLUs, you could get arbitrarily close to a lookup table, which I think would be best described as a nonlinear model.)
Nonlinear things start looking like linear things again in very high dimensions
Only when you dimensions are truly independent and that is a stretch. Really what you are saying is that you are more likely to find a field for your problem, and fields don't exist in more than 2 dimensions.
Consider Predator Pray with fear and refuge, which is indeterminate, and not due to a lack of precision but a topological feature where ≥3 open sets share the same boundary set.
https://www.sciencedirect.com/science/article/abs/pii/S09600...
General relativity, with 3 spacial and one temporal dimension is another. One lens to consider this is that rotations are hyperbolic due to the lack of independence from the time dimension.
Quantum mechanics would have been much more difficult if it didn't have two exit basins. Which is similar to ANNs and linear regressions being binary output.
(Some exceptions will orthogonal dimensions like EM)
Consider Predator Pray with fear and refuge, which is indeterminate, and not due to a lack of precision but a topological feature where ≥3 open sets share the same boundary set.
https://www.sciencedirect.com/science/article/abs/pii/S09600...
General relativity, with 3 spacial and one temporal dimension is another. One lens to consider this is that rotations are hyperbolic due to the lack of independence from the time dimension.
Quantum mechanics would have been much more difficult if it didn't have two exit basins. Which is similar to ANNs and linear regressions being binary output.
(Some exceptions will orthogonal dimensions like EM)
Linear models don’t need everything to be linear.
I presume you are implying that linear model only mandates linear relationship between predictor and regression coefficients?
Linear models are a linear combination of possibly non-linear regressors. The linearity is strictly in the parameters, not in whatever you're adding up.
A neural network can be pedantically referred to as a linear model of the form y = a + b*neural_network, for example. Here, y is a linear model (even though neural_network isn't).
A neural network can be pedantically referred to as a linear model of the form y = a + b*neural_network, for example. Here, y is a linear model (even though neural_network isn't).
IMO it's not very pedantic... It's pretty much exactly what it is! (I'm not too sure about the example equation you give, however)
A linear relationship between any transformation of the outcome and any transformation of the predictor variables — so the function is linear but the relationship between predictors and outcome can take on almost any shape.
Ah, I missed 'the transformation of outcome' in my mind. Thanks for clearing it up.
This is why I feel like I couldn't become a credible AI/ML consultant. I would just throw everything into a linear model, make some progress, and call it a day.
99% of AI / ML consultants are just throwing everything into a linear model, make no progress, slap a big AI sticker on the box and call it a day.
You would have fitted right in.
You would have fitted right in.
Wasn’t there an OpenAI paper where they showed that MM multiplication with the numerical imprecision of floats was enough to get general function learning?
You might not even need anything else…
You might not even need anything else…
One almost certainly does not need anything else ... unless what you want is big piles of investor cash, a Scrooge McDuck swimming pool quantity of investor cash - then you need to call whatever maths you do AI.
Maybe it's because to scientists and statisticians comfortable with linear algebra, everything looks like a linear model.
Everything is linear if plotted log-log with a fat magic marker (Mar's Law) [1].
[1]: https://spacecraft.ssl.umd.edu/akins_laws.html
[1]: https://spacecraft.ssl.umd.edu/akins_laws.html
I thought everything was a power function.
I thought everything was a graph.
Nah, it's all just ifs and for-loops:
https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F4...
https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F4...
(2022)
isn't every non linear model fundamentally linear if it is run on a von neumann computer?
My series of lm/glm/gam/gamm revelations was:
1. All t-tests and ANOVA flavors are just linear models
2. Linear models are just a special case of generalized linear models (GLMs), which can deal with binary or count data too
3. All linear models and GLMs are just special cases of generalized linear mixed models, which can deal with repeated measures, grouped data, and other non-iid clustering
4. Linearity is usually a bad assumption, which can easily be overcome via splines
5. Estimating a spline smoothing penalty is the same thing as estimating the variance for a random effect in a a mixed model, so #3 and #4 can be combined for free
And then you end up with a generalized additive mixed model (GAMM), which can model smooth nonlinear functions of many variables, smooth interaction surfaces between variables (e.g. latitude/longitude), handle repeated measurements and grouping, and deals with many types of outcomes, including continuous, binary yes/no, count, ordinal categories, or survival time measurements.
All while yielding statistically valid confidence intervals, and typically only taking a few minutes of CPU time even on datasets with hundreds of thousands / millions of datapoints.