I have had some time off this week so decided to work on something a bit different. I have been working on a concept fuzzing framework for security testing. In the screenshot you can see some of the files produced by it - The bottom right is the configuration used to generate the file format (for this case Bitmap, although I have tested a few others like WAV)
Bottom left is a bitmap produced with no defects. The top shot is a bitmaps produced with some random changes - you can see the green bitmap is now corrupted due to a change somewhere in the format.
I've played with a variety of frameworks (Storm and IBM Infosphere Streams are probably the two I have worked with the most) - In my previous job I was in a team which designed and developed our own (and later integrated partly with IBM Streams) I took down many lessons from the initial creation, the later integration and the work following that. ghostream was born out of those lessons.
Mostly I found the follow problems with existing frameworks :
- Tied to an execution environment - Storm with ZooKeeper and Streams with their own custom one. This makes it very difficult to use either for small projects - and limits integration choices when scaling.
- Resource hogs - The IBM Streams environment is a huge resource hog, I never really got that far with Storm but the number of dependencies it required just to get something up and running provided a similar sense of dread.
- Easy to debug/optimization - Streams has the best tools for this at the moment but they are all heavily tied to the execution environment. I'm hoping with ghostream that structures can be built at the protocol level to provide a way to build tools on top - not tied to any particular environment.
This week as been all about getting a second version stamped and out (nowhere near a finished version or even 1.0, but a 0.2 version)
I've been working on https://ghostream.com a framework for constructing stream processing systems. Getting a first draft of the website (http://imgur.com/ujsmjJz) and the documentation (although there is still much to do)
And after that, lots of bug fixes and environment clean ups (code coverage pipeline / static analysis etc.) Managed to knock down quite a few little bugs that way.
I posted this on the last thread, but I am definitely still looking for volunteers.
I'm actually currently starting to look for a co-contributor or two, there is so much that I want to get done, and having an extra set of eyes and hands would be great.
ghostream is a stream processing framework, built in C++ - it is a data stream processing framework. Having used numerous commercial ones (most notably IBM's Streams), and building various other part-frameworks (i.e. for time and budget reasons focussed on one particular thing) I has an itch to build an open source one (while storm and samza are definitely related, I feel they fall into a slightly different category - I'm all about the streams :) )
It is still in very active development - but the base is solidified now and the new test framework is starting to shape up nicely. There are a whole list of issues on github + a roadmap waiting to be exploited.
There is a great episode of Black Mirror (All episodes of Black Mirror are great, but this one is more relevant - Season 1 Episode 3) titled The Entire History of Us, it turns the figurative mirror on a world where people do exactly this...
I'm actually currently starting to look for a co-contributor or two, there is so much that I want to get done, and having an extra set of eyes and hands would be great.
ghostream is a stream processing framework, built in C++ - it is a data stream processing framework. Having used numerous commercial ones (most notably IBM's Streams), and building various other part-frameworks (i.e. for time and budget reasons focussed on one particular thing) I has an itch to build an open source one (while storm and samza are definitely related, I feel they fall into a slightly different category - I'm all about the streams :) )
It is still in very active development - but the base is solidified now and the new test framework is starting to shape up nicely. There are a whole list of issues on github + a roadmap waiting to be exploited.
I am still amazed when I try to read a simple blog article and am greeted with a blank white page, it turns me right off the site and I go spend my time elsewhere.
I will happily shell out money for content that I like.
I have tried neither of those two...I have heard good things both though, maybe I will make the time to give the a try.
A Javascript IDE would be great, I remember working on some Javascript a couple of years ago and being frustrated at even getting basic autocomplete to work in any tool (Vim was the closest but lacked any context). And +1 Open Source. I look forward to giving it a try.
On my side projects I tend to use Vim for C++ which works fairly well after heavy customization with plugins.
At work I have tried to adopt Vim with Java but Java does not lend itself to simple developments (far too may 3rd party library imports). Eclipse is bulky and slow, doesn't customize well and constantly crashes.
I have heard emacs lends itself better to Java than Vim due to it's extensibility, but devoting the time to learning another application is not really something I want to do.
Vim is too basic for a complete IDE (and it should remain that way it is a text editor after all), Eclipse is too bulky, I want something that is just right.
I'm currently building ghostream (https://github.com/ghostream/ghostream) - it is a data stream processing framework. Having used numerous commercial ones (most notably IBM's Streams), and building various other part-frameworks (i.e. for time and budget reasons focussed on one particular thing) I has an itch to build an open source one (while storm and samza are definitely related, I feel they fall into a slightly different category - I'm all about the streams :) )
It is still in very active development - but the base is solidifying and the new test framework is starting to shape up nicely.
Probably not what you are looking for, but I would love to see more real-time data curation - I would certainly be willing to hand over cash for some decently priced real time data feeds - stock tickers, sports events, raw traffic data, headlines, currency, flights - many of these are available in pull formats, but it would be great to get a push feed.
I read GEB during my final year of university, during winter break. It took about 3 weeks of solid reading (wake up, read, pause and think and clear my mind, do other stuff, go back to reading, sleep, repeat) to get through it. By that point in my life I had already been exposed to many of the topics discussed, and even unwittingly read works that cited GEB, so had been also been exposed to some of the ideas presented as well - I don't however think that matters too much.
It really is, in my opinion, an amazing and clever book. If you can, I would recommend taking a few weeks to really digest it, it is not a book you can read for hours and hours on end - you will need to stop, clear your mind and reflect on some of the points made - at least that is what I had to do.
Be warned, the book will start to mess with you...but it warns you when it does...most of the time.
I really wish people wouldn't give this as much weight has it gets, there are so many problems with the methodology:
1. The raw data (answers, in particular the free response text) has never to my knowledge been published.
2. The algorithm used to select schools and pupils taking the test has never to my knowledge been published. (In fact it differs from country to country...or city..)
3. The test itself is taken in many different languages, which language version of the test is used in which country, has to my knowledge never been published.
4. They compare Shanghai, a relatively well off city with entire countries..without knowing the data in 1-3 we cannot know if this comparison is fair or makes any sense at all.
5. They administer the test at different times of the year relative to exam time...some teenagers would have just got back from summer break, others would have been deep revision, preparing for other exams - I would guess this has a statistically significant impact on performance.
Until they publish all the data so it can be analysed independently (and god forbid replicated), this isn't science or statistics...it's busy work.
Honestly, part of it is motivation. I keep getting distracted with other things.
I expected it to go the 80-20 way, the initial ideas are not new ones, I have been building similar frameworks for a few years.
The time is mostly spent on the more difficult problems like "How should operators hang up when the program is closed?" " Should the program be expected to close?" "Should there be a global context object, or a default one" "Should an operator have a window by default or not"
There is no right answer, so its just a case of sucking it up and implementing something, seeing if it works, and going from there.
Plus the side escapades with earthquakes and space weather and irc channels - although implementing these little things does drive the bigger project.
I am hoping before the end of the year I can hit something I feel comfortable slapping "1.0" on and move onto something else for a while and use it as a base for working on some other ideas I have.
An open source stream processing framework (https://ghostream.com) - it isn't really ready for the limelight yet, but you can find the code base on github (https://github.com/ghostream/ghostream) - final steps on the to do list are a few more example projects, a couple more functional operators and to polish up the documentation.
Check out this veritasium video https://www.youtube.com/watch?v=ZMByI4s-D-Y - It goes into a little detail on how the kilogram got it's name, it is also very interesting in it's own right.
I pulled the example from the air to make a point, but you are correct, the last 20% is 80% of the work...
See the comment by aaron695 who made a very good point about feedback.
I wonder though if there is some easy fruit at the bottom of the tree i.e. elementary school - where I gather grading is probably a lower burden and easier to formalise - although I am probably again greatly understating the problem.
A very good point that I did not consider, I remember getting very good feedback from my teachers - It would appear I underestimated the size of the problem.
I have had some time off this week so decided to work on something a bit different. I have been working on a concept fuzzing framework for security testing. In the screenshot you can see some of the files produced by it - The bottom right is the configuration used to generate the file format (for this case Bitmap, although I have tested a few others like WAV)
Bottom left is a bitmap produced with no defects. The top shot is a bitmaps produced with some random changes - you can see the green bitmap is now corrupted due to a change somewhere in the format.