> pull everything to my PC and use pandas, to take advantage of its ability to build up results piece by piece.
Most difficulties in data processing arise from the need to process multiple tables. It is highly difficult in SQL but it can be also difficult in pandas especially for complex analytical queries. Indeed, in pandas you still have to use the same join and groupby operations as in SQL.
An alternative to SQL, join-groupby, and map-reduce is developed in the Prosto library:
It is a layer over pandas and its main distinguishing feature is that it relies on functions and function operations for data processing as opposed to using only sets and set operations in SQL and other set-oriented approaches
is (strictly speaking) not part of the relational algebra because it is not using set operations (like join or union). It was added to the relational model because we hardly can process data without such expressions.
A better way to formally describe such calculated columns is to introduce functions and treat them as first class elements of the data model. In particular, function operations are better than set operations in these cases [1]:
- Calculating data using calculated columns. We describe a calculated column as a function without generating new relations (as opposed to select-from)
- Aggregating data. We can add an aggregate column without generating new relations (as opposed to groupby)
- Linking data. We can add derived link columns without generating new relations (as opposed to join)
This function-based approach to data modeling and data processing was implemented in Prosto [2] which demonstrates how many typical relational tasks can be solved using functions.
Just for comparison, a functional approach is a major alternative to relational and set-oriented models and query languages. The difference is that functions and operations with functions are first class elements of the model.
One version of it is implemented in this project:
Here is one possible implementation of the concept-oriented model of data for data processing. It heavily relies on functions and operations with functions and is an alternative to purely set-oriented approaches like map-reduce or join-groupby (sql):
For example, I have data from 1900 till 2000. I train ARMA using this data by storing the corresponding coefficients as model parameters. Now I get data from 2010 to 2020. My goal is to use these (AR and MA) coefficients in order to predict the value in 2021 (without using the historic data I used for training). I think it does make sense and it is precisely how typical ML algorithms work. So it is more a matter of how an algorithm is implemented and which usage patterns it follows.
In fact, there are two general approaches to unifying ML with time series:
o Unifying ML API and patterns so that time series can be analyzed in the same way as normal tabular data. Example: sktime
o Preprocessing libraries applying data transformations to the input time series by converting them to a (tabular) form which can be used by normal ML algorithms. These are typically general or specific libraries for feature engineering, feature learning, feature selection and generic transformations. Examples: https://github.com/blue-yonder/tsfresh or https://github.com/prostodata/prosto
When I see a new forecasting library, my first question is whether it can apply ARIMA in a sklearn manner by training a model using (large) train time series X, storing the model by discarding the train data, and then using this model for predictions by feeding a completely different (and shorter) time series. Importantly, the time series used for prediction is not a continuation of the time series used for training. Moreover, we do not store anywhere the original time series used for training.
So the question is whether sktime can apply ARIMA in this way.
Anomaly detection belongs to unsupervised learning while in time series analysis we normally think about future and future values are viewed as labels. One approach to think in terms of anomaly detection is to train a normal forecasting model. An anomaly is then viewed as large deviation from predicted values. Another approach to train an autoencoder on segments of the time series. Then anomaly is defined by the degree of deviation of the decoded segment from the real segment.
> No, the relational model is beautiful and consistent! SQL is messy...
What is beautiful and consistent is the relational algebra. The relation model relies on this formalism to model data by making some rather strong assumptions about how tuples and relations represent things and how relational operations are used to process them. And these assumptions are precisely what propagates to SQL and what some authors (see references in the article) consider controversial and messy. Then the question is whether and how these controversies can be fixed (or whether they are bugs or features).
A radically different approach to fix these controversies is to introduce a different formalism and different data model (as opposed to fixing only syntax) which is based on using functions. In other words, instead of using sets and set operations, we use in addition functions and function operations [1]. Here you can find an implementation of this approach:
https://github.com/prostodata/prosto Prosto is a data processing toolkit radically changing how data is processed by using both sets and functions and being a major alternative to map-reduce, join-groupby and other set-oriented approaches
I think both conceptions are important. On one hand, some code is naturally related to certain data and hence we should accordingly design the program. On the other hand, complex and distributed programs have code which must be dynamically bound to different data. OOP focuses on only one aspect but does it very well. How to adequately support the opposite aspect is not clear. At least there are many ways how this can be done.
> I am currently working on complex (JOIN and aggregates) queries without any SQL knowledge
One way to process data without joins and groupy in multiple tables is to use Prosto toolkit: https://github.com/prostodata/prosto It is an alternative to SQL, MapReduce and other classical approaches by radically changing how data is processed
In the context of computer science, matrices are used as a representation construct with different possible formal interpretations which frequently can be recognized depending on the supported operations. For example, some functions of NumPy assume that matrices represent elements of linear algebra while other functions treat a matrix as a collection of rows or columns. In the latter case, a square matrix is probably an exception because rows and columns have completely different semantics.
PyTorch is simpler, easer to use, consumes less memory and allows for dynamic dynamic computational graphs (dynamic operations during the forward pass).
Is there any (significant) difference between sharding and load balancing?
It seems that in both cases the idea is to distribute (supposedly independent) requests between workers and one of the main difficulties is that requests might not be independent either within one stream (say, in the case of sessions) or between different streams (say, if they need to use one common state).
Most difficulties in data processing arise from the need to process multiple tables. It is highly difficult in SQL but it can be also difficult in pandas especially for complex analytical queries. Indeed, in pandas you still have to use the same join and groupby operations as in SQL.
An alternative to SQL, join-groupby, and map-reduce is developed in the Prosto library:
- https://github.com/prostodata/prosto Functions matter! No join-groupby, No map-reduce.
It is a layer over pandas and its main distinguishing feature is that it relies on functions and function operations for data processing as opposed to using only sets and set operations in SQL and other set-oriented approaches