If you have a mortgage, it ain’t half bad
def add(df1, df2, meta_cols, val_cols=None):
# join on meta cols
# add val cols (default to all non meta cols if None)
# return df with all meta and val cols selected
In theory I think that's fine. The problem is that in practice this will cause a lot of visual noise in your models, since for every operation you would need to specify, at least, your meta columns, and potentially value columns too. If you change the dimensionality of your data, you would need to update everywhere you've specified them. You could get around this a bit by defining the meta columns in a constant, but that's really only maintainable at a global module level. Once you start passing dfs around, you'll have to pass the specified columns as packaged data around with the df as well. There's also the problem that you'd need to use functions instead of standard operators. # pandas
prices_df.loc['2023-03'] *= 1.1
# polars
polars_df.with_column(
pl.when(pl.col('timestamp').is_between(
datetime('2023-03-01'),
datetime('2023-03-31'),
include_bounds=True
)).then(pl.col('val') * 1.1)
.otherwise(pl.col('val'))
.alias('val')
)
Add expected temperature offsets to base temperature forecast at the state county level: # pandas
temp_df + offset_df
# polars
(
temp_df
.join(offset_df, on=['state', 'county', 'timestamp'], suffix='_r')
.with_column(
( pl.col('val') + pl.col('val_r')).alias('val')
)
.select(['state', 'county', 'timestamp', 'val'])
)
Now imagine thousands of such operations, and you can see the necessity of pandas in models like this. import numpy as np
import time
vals = np.random.randn(1000000, 2)
point = np.array([.2, .3])
s = time.time()
for x in vals:
np.linalg.norm(x - point) < 3
a = time.time() - s
s = time.time()
np.linalg.norm(vals - point, axis=1) < 3
b = time.time() - s
print(a / b)
~296x faster, significantly faster than the solution in the article. import numpy as np
import time
vals = np.random.randn(1000000, 2)
point = np.array([.2, .3])
s = time.time()
for x in vals:
np.linalg.norm(x - point) < 3
a = time.time() - s
s = time.time()
np.linalg.norm(vals - point, axis=1) < 3
b = time.time() - s
print(a / b)
~296x faster, significantly faster than the solution in the article. And my assertion was supported by nearly 20 years of numpy being a leading tool in various quantitative fields. It’s not hard to imagine that a ubiquitous tool that’s been used and optimized for almost 20 years is actually pretty good if used properly.