Stanislav, Seltser, Petacube
agree with Jake. Matlab still has some unique features to it which many open source projects (eg python) don't have - eg FPGA integration, System Modeling and Simulation. Combine that with decent language, nice debugger and many strong industry-specific solutions its good bet its will be around for a while.
The reasons people dump matlab is not that its overpriced but due to the lack of integration with big data systems and the fact that matlab license cost at large scale becomes untenable.Plus number of industries using these unique matlab features is relatively small.
Stanislav Seltser, Petacube
you are talking comparing structured workload(array-based TileDB) to unstructured one (JSON+Spark). Once you convert your JSON to sparce array structure (one time conversion) TileDB will beat Spark+JSON by several orders of magnitude. Caveat: assuming your spark+json workoad is a some heayy processing not a lightweight one.
Stanislav Seltser, Petacube
_pmf:
think of distributed persistent numpy arrays as opposed to key-value pairs in Redis. The idea you will not pull data to do computation, you will push your code in. pretty good for archiving due to compression
ingesting data into postgres makes sense for sparce data but not for dense data because it is waisting a lot of of space due to storage of coordinates with every data point and every weather variable. If you are using NOOA grib2 forecast files those are dense . Not to mention losing compression in postgres. TileDB will store data compressed, the dimension coordinates themselves will be compressed, plus column storage (each NetCDF variable) will make retrieval of dense weather data blazingly fast as oppose to postgres where you will have to scan the whole table
SciDB --- license: Affero ,software size: 5GB ,data model: ACID ,focus on: dense ,dimensions: integer ------
TileDB license MIT ,software size <1MB , data model: eventual consistency via fragments , focus on: dense, sparse ,dimensions are integer, floats