Want Better Forecasting? Silence the Noise(knowledge.wharton.upenn.edu)
knowledge.wharton.upenn.edu
Want Better Forecasting? Silence the Noise
https://knowledge.wharton.upenn.edu/article/want-better-forecasting-silence-the-noise/
14 comments
Skimming the transcript, I can’t help but feel like this field needs to study some signal processing ideas. They use a word like noise, but never use a word like filter, or attentuation. Even their use of the acronym BIN is unfortunate, since binning is a very important concept when dealing with the frequency domain.
Can you say more about what type of attenuation would be possible?
The noise models in physical systems are fairly simple/characterizable (y = f(x) + e, where e ~ P and P is some stable distribution, or e = a z-transform model), whereas in social sciences, the "noise" component is actually a catch-all/residual for whatever is unknown (e is unknown or unstable). It seems to me that it would difficult to apply any kind of signal processing techniques but I could be wrong.
The noise models in physical systems are fairly simple/characterizable (y = f(x) + e, where e ~ P and P is some stable distribution, or e = a z-transform model), whereas in social sciences, the "noise" component is actually a catch-all/residual for whatever is unknown (e is unknown or unstable). It seems to me that it would difficult to apply any kind of signal processing techniques but I could be wrong.
And this might be the very problem. Noise is a very well defined term in information theory. If political forecasting is throwing around the term without applying it correctly they’re gonna have a bad time.
At the end of the day though, any time you have time series data you can apply filters to smooth and shape your data. I don’t understanding how they’re modeling their data. There’s a good chance they’re doing some kind of frequency modeling where they’re counting correct predictions. It definitely sounds like they’re doing some stochastic modeling when they start talking about percentage predictions. You can definitely shape frequency domain as well with filters, though I havn’t quite thought through how the stochastic aspects might interact.
Keep in mind, filters are very basic, and even something as common as averaging data is a low-pass filter. As is fitting to a curve. This all acts to attenuate the signal we care about without also attenuating the noise. Though, again, if someone isn’t being rigorous about what constitutes noise, then no amount of filtering will actually help...
I’m also sorry if this thread isn’t very insightful. I’ve been having my nose rubbed in signal processing at work for the last 2 months, and it’s all I can see everywhere I look. I see parrallels everywhere that may not be there.
You’re also very correct about the simplicity of physical models versus social sciences. It may just be that trying too hard to apply basic information theory at models that are almost impossible to create in the first place is a fools errand.
At the end of the day though, any time you have time series data you can apply filters to smooth and shape your data. I don’t understanding how they’re modeling their data. There’s a good chance they’re doing some kind of frequency modeling where they’re counting correct predictions. It definitely sounds like they’re doing some stochastic modeling when they start talking about percentage predictions. You can definitely shape frequency domain as well with filters, though I havn’t quite thought through how the stochastic aspects might interact.
Keep in mind, filters are very basic, and even something as common as averaging data is a low-pass filter. As is fitting to a curve. This all acts to attenuate the signal we care about without also attenuating the noise. Though, again, if someone isn’t being rigorous about what constitutes noise, then no amount of filtering will actually help...
I’m also sorry if this thread isn’t very insightful. I’ve been having my nose rubbed in signal processing at work for the last 2 months, and it’s all I can see everywhere I look. I see parrallels everywhere that may not be there.
You’re also very correct about the simplicity of physical models versus social sciences. It may just be that trying too hard to apply basic information theory at models that are almost impossible to create in the first place is a fools errand.
> Noise is a very well defined term in information theory. If political forecasting is throwing around the term without applying it correctly they’re gonna have a bad time.
Class is a very well defined concept in software development. Are political science people going to have a bad time because they "throw around" the word class and use it in an economic sense?
There are only so many possible finite words that can be created from vocalizable letter combinations, there is bound to be overlap across disciplines.
Class is a very well defined concept in software development. Are political science people going to have a bad time because they "throw around" the word class and use it in an economic sense?
There are only so many possible finite words that can be created from vocalizable letter combinations, there is bound to be overlap across disciplines.
I'm not sure they're using "noise" any differently
Wikipedia says
> In signal processing, noise is a general term for unwanted (and, in general, unknown) modifications that a signal may suffer during capture, storage, transmission, processing, or conversion.
That sounds like what the interviewee is talking about. What distinction do you have in mind?
> In signal processing, noise is a general term for unwanted (and, in general, unknown) modifications that a signal may suffer during capture, storage, transmission, processing, or conversion.
That sounds like what the interviewee is talking about. What distinction do you have in mind?
I’m about to turn in my PhD dissertation for a PhD in one of the business disciplines. I’ve found this to be very common for academic researchers in finance and accounting. Most have no experience outside academia, and business silos itself.
For instance, we have econometrics instead of statistics. Which means we have different terms for just about anything in statistics. And, through discussing my work with my sibling, who is an accomplished statistician, our cutting edge econometrics is usually 20+ years old statistics models that are all but abandoned by the rest of the world.
For instance, we have econometrics instead of statistics. Which means we have different terms for just about anything in statistics. And, through discussing my work with my sibling, who is an accomplished statistician, our cutting edge econometrics is usually 20+ years old statistics models that are all but abandoned by the rest of the world.
It's definitely an issue across many fields and I don't have any other explanation than pride and "confidence" issues. A similar thing is AI and stats which have only really converged in the 1990s and are still quite separate communities, or reinforcement learning and control theory which investigate overlapping issues from different angles but haven't communicated for a long time.
Well then what holds someone back from learning state-of-the-art stats and making a name for themselves in econometrics?
Well then what holds someone back from learning state-of-the-art stats and making a name for themselves in econometrics?
Protip: an EWMA is a smoothing convolution whose deconvolution can be computed in closed form with a little algebra. To reduce the effects of noise, apply the EWMA, then use a forecasting method of your choice to predict the smoothed series, and finally apply the deconvolution to recover the original series. This technique can be useful for series with strong seasonality, where some of what may appear to be noise is actually useful signal, but signal which arrived slightly ahead or behind schedule within the season.
> How would you feel if an algorithm decided whether or not you should be charged with a crime? Whether an algorithm decided whether or not you had cancer? I think for many of these cases, it’s well known that the algorithms do much better than people. I was just reading Malcolm Gladwell’s new book, Talking to Strangers. He tells the story of a judge in Chicago who decided whether to keep detainees or release them on bail. He liked to look into the eyes of the detainee to decide whether he would skip bail. It turned out that information wasn’t nearly as valuable as other information that you can derive from machine learning, algorithms and so forth. Accuracy increased greatly with the algorithm.
I feel like this glosses over a lot of evidence that shows that using algorithms to determine guilt and innocence in the criminal justice system is incredibly fraught.
https://www.wired.com/2017/04/courts-using-ai-sentence-crimi...
https://www.propublica.org/article/how-we-analyzed-the-compa...
https://www.washingtonpost.com/business/2019/11/19/algorithm...
I feel like this glosses over a lot of evidence that shows that using algorithms to determine guilt and innocence in the criminal justice system is incredibly fraught.
https://www.wired.com/2017/04/courts-using-ai-sentence-crimi...
https://www.propublica.org/article/how-we-analyzed-the-compa...
https://www.washingtonpost.com/business/2019/11/19/algorithm...
Read the paper and still confused as to what the author means by “noise”?
And don’t you know whether something is noise or not after the fact? You may think some signal is useful when you first encounter it and you may not know it’s noise until after it produces a false prediction. So silencing it isn’t actually possible.
And don’t you know whether something is noise or not after the fact? You may think some signal is useful when you first encounter it and you may not know it’s noise until after it produces a false prediction. So silencing it isn’t actually possible.
As an example, before 9/11, there was some chatter about a possible terrorist attack in the USA. After 9/11 we know now this wasn’t noise. But if it hadn’t happen, we would classify it as noise. Telling the CIA to silence the noise is useless advice because we don’t know beforehand what is noise!
This "silence the noice" concept sounds like the Deming "common causes of variance". And "better forecasting" like "reduce variability to improve predictability"
Soo... signal processing?