This brought me to dig deeper into Elasticsearch code, find out more about its code quality, deal with machine endiannes, deal with byte shifting, think how to structure code in Golang and overall enjoy the feeling of touching the bare metal again...
The clients actually acts as a cluster node and therefore has knowledge about the cluster state, its indexes and shards, because it receives notifications from it, once it joins.
This allows to execute operations on a specific shard of a specific index on a specific node of the cluster resulting in better performance than going through the HTTP interface.
It can be used to efficiently store big quantities of data, for instance logs, which then can be visualized with Kibana.
It's just unfortunate that Elasticsearch presents the problems mentioned in the article and which I also experience in production, because it has a series of plugins which makes it a good solution for specific use cases.
I am currently writing a Golang client for Elasticsearch which uses the native binary protocol and I have to say the lack of documentation about it is making the process really painful!
I tried to use the Elasticsearch thrift plugin but unfortunately it does not work for the version 1.1 and 1.2
So basically I have to inspect each and every byte of each and every request and response in order to be able to send or parse data.
While developing the client a managed several time to crash the Elasticsearch server by sending malformed packets. In addition, this, brought me to review the networking part of Elasticsearch code and I think it needs a refactoring and a better, deeper and cleaner usage of Netty.
I hope they will soon sort out this and the problems mentioned in the article, since I think that Elasticsearch is really an amazing product!
Looking for a company where code simplicity and performance matter! I do not like to waste CPUs cycles, memory and I/O in general.
I have more than 13 years experience as a developer and I have been designing and implementing scalable, distributed and resilient systems and software, to handle over 40 million unique users per month, for the last 7 years. I am proficient with Scala, Java, Netty and Akka and I really enjoy coding in Golang. I have a strong passion for cryptography and IT security in general.
On my free time, I am working on a Golang Elasticsearch client which uses the tcp transport.
I am an open minded, friendly and business driven person who loves technology. So for any questions drop me an email !
I am passionate about cryptography and IT security and I am looking for an opportunity to work on open source projects preferably in crypto and security in general.
I have also more than 5 years of experience in scaling web sites and native applications stack with more than 50 billion page views per month.
I love talking to others, so for any question feel free to ping me anytime!
> The judges who chose AES and SHA-3 as the "winners" of the global competitions are the NSA.
Sure, however this process creates alternatives and if the crypto community thinks the winner is backdoored I am pretty sure we will know it and additionally we will have a valid alternative ready to be implemented.
Additionally if the NSA/NIST modifies the specs for the crypto construction there is still the possibility to implement the original one. See SHA-3 for instance. It was about to be weakened, but the crypto community could still implement the original spec.
> That would include AES and SHA-3.
You cut the rest of the sentence and therefore changed completely the whole meaning. My original sentence included: "...without going through years of reviews by international cryptographers."
Take a look at this video of D.J.B.: https://www.youtube.com/watch?v=G-TM9ubxKIg
He makes a great example with the Dual_EC_DRBG, where many cryptographers told NIST that there could be a backdoor. NIST answer basically was: sorry too late, it has already been implemented !
So in other words, in case of Dual_EC_DRBG the standardization process was all in reverse. First NIST standardized it and then the crypto community started to review it and found problems.
In general you should prefer crypto constructions which are a result of global competitions.
For example AES and SHA3.
You should avoid at all costs anything that has been standardized by NIST without going through years of reviews by international cryptographers. Dual_EC_DRBG is a clear example of crypto construction which falls into this category.
This is my general rule of thumb.
However knowing which ciphers one should use is not enough! You absolutely need to know HOW to use them.
A basic and superficial example is AES in ECB mode, which is semantically secure as long as you use a key to encrypt one and only one single block.
Another one is, for example, after how many encrypted blocks a key should be rotated, based on the underlying cipher used.
Once you have learnt how to use the basic building blocks of crypto you are then NOT supposed to write your own implementation and instead use existing ones....there is a small problem with this....they are broken or they either not implement all the necessary crypto constructions you need.
OpenSSL is an example of broken crypto implementation, and instead NaCl does not have TLS implemented.
So this is a short summary and my personal opinion of why crypto is hard.
On top of all this there are not enough experts out there which have the time to review crypto implementations or new and old constructions, and we are living a historical period where we desperately need crypto to protect our privacy.
Unless I am mistaken US and most European countries are based on Democracy.
Wikipedia defines Democracy as a form of government in which all eligible citizens have an equal say in the decisions that affect their lives. Granted, wikipedia is not the oracle but it gives a good definition in my opinion. Does, the gathering of my personal data, affect my life ? Well in my personal opinion it does, therefore I should be informed about it.
