Energy is a shorthand for L2 norm (in a probabilistic sense). This similar to the definition of the energy of a signal in signal processing. 'Information content' is an alternative for 'energy' here, but it can be mistaken for entropy.
In that PCA pre-processing step, nothing guarantees that principal components are better representations for your problem than original inputs; in fact PCA has nothing to do with your target, how could it guarantee principal components are better representation to predict it?
Similarly, understanding the covariance structure of your original inputs will not necessarily help you predict your target better.
Here's a simple example illustrating this. Take x a single feature highly informative about y, take z a (large) d-dimensional highly structured vector that is independent from both x and y. Now, consider using [x, z] to predict y.
In this case, x happens to be a principal component as x and z are independent; it is associated to eigenvector [1, 0,...., 0] and eigenvalue Var(x). All other eigenvectors are of the form [0, u_1, ..., u_d] where [u_1, ..., u_d] is an eigenvector of Cov(z).
All it would take for x to be the very last (i.e. 'least important') principal component is for Var(x) to be smaller than all eigenvalues of Cov(z), which is easily conceivable, irrespective of y! In your quest for a lower-dimensional 'bottleneck' using PCA you would end up removing x, the only useful feature for predicting y! This will certainly not reduce overfitting.
PCA and other autoencoders work well as pre-processing step when there are structural reasons to believe low energy loss (or low reconstruction error) coincides with low signal loss. In tabular data, this tends to be the exception, not the norm.
Feature selection ought to be model-specific. Because a feature wasn't selected by Lasso (in a linear model) does not mean it cannot be useful in a non-linear model.
I'm the author of the post. I'm slightly late to the party, but I'll try to clarify a few misunderstandings.
First and foremost, the post deals with the following scenario too many data scientists find themselves in: "I have (generated) a lot of features; let me do PCA and train my model using the top few principal components". This is a terrible idea and the post explains why.
Second, there seems to be a debate about 'feature selection' vs. 'feature construction' (or 'feature generation'), and whether PCA is of the former or latter type. Here are the definitions I use in the whole blog.
Feature Construction is the process consisting of generating candidate features (i.e. transformations of the original inputs) that might have a simpler relationship with the target, one that models in our toolbox can reliably learn.
E.g. a linear model cannot learn a quadratic function. However, because a quadratic function of x is linear in [x, x^2], the feature transformation x -> [x, x^2] is needed to make our quadratic function learnable by a linear model.
Feature Selection is the process consisting of removing useless features in the set of candidates generated by feature construction. A feature is deemed useless when it is uninformative about the target or redundant.
In the scenario the blog post deals with (i.e. "I have (generated) a lot of features; let me do PCA and train my model using the top few principal components"), data scientists do both feature construction (full PCA, i.e. projecting the original input onto eigenvectors to obtain as many principal components as the dimension of the original input) AND feature selection (only selecting the first few principal components with the highest eigenvalues).
When the goal is to predict y from x, using PCA for either feature construction OR feature selection is a bad idea!
For feature construction, there is nothing in PCA that will intrinsically guarantee that a linear combination of coordinates of x will have a simpler relationship to y than x itself. PCA does not even use y in this case! E.g. Imagine all coordinates of x but the first are pure noise (as far as predicting y is concerned). Any linear combination of x will just make your inputs noisy!
For feature selection, even assuming principal components make sense as features, principal components with the highest variances (i.e. corresponding to the highest eigenvalues) need not be the most useful for predicting y! High variance does not imply high signal.