I run the infrastructure (k8s+helm on GCP) for a PoS validator of a top 50 crypto project. My client is a big whale who bonded ~$15.6M at the projects all-time high. It’s about 1-2 hours of work per month and my 10% cut of the rewards nets me anywhere from $4k-$40k/mo. depending on the price. Given the fact that crypto is in the gutter now I haven’t been selling any to USD, but it’s a nice way to stack an asset with high upside potential while doing very little work.
Automox has seen double digit growth year over year, and this year is no exception. We recently raised $110 million Series C led by Insight Partners and appointed Dmitri Alperovitch, Crowdstrike co-founder as the Chair of our Board. We are modernizing the IT Ops market with our cloud native approach to automating and streamlining IT workflows.
With over 100 positions to hire for this year, we have something for everyone. From building data pipelines, scaling our infrastructure in AWS, developing distributed backend services, to simplifying a complex UI.
* Staff, Senior, Mid Level Engineers across the stack
* Senior Data Engineers
* SDETs (Python)
* Windows or Linux or Mac Systems Engineers
* UX - Researcher and Architect
* Technical Product Managers
* Senior Site Reliability Engineers
Can confirm that running Spark at scale is difficult. Not even necessarily talking about scale of data or scale of performance, but organizational scale. Getting dozens or hundreds of engineers aligned around best practices, tooling and local development for Spark is both challenging and extremely rewarding. When you have everyone buy into Spark as not just an execution environment but a programming paradigm, it really unlocks some cool potential. If anyone cares this is how I've found to best get Spark users riding on rails:
* Use a monorepo to "namespace" different projects/teams/whatever. Each namespace has its own build.sbt for Scala jobs and Conda/Pip requirements file for PySpark. This gives you package isolation so that different projects can bump requirements at their own pace. This is crucial in larger organizations where you might have more siloed development or more legacy applications.
* Build each project in the monorepo into a separate Docker image and tag it accordingly with some combination of the branch and namespace.
* Deploy applications onto Kubernetes by invoking the SparkOperator (https://github.com/GoogleCloudPlatform/spark-on-k8s-operator), This abstracts away a lot of the hassle of driver/executor configuration and gives you nice out-of-the-box functionality for scraping Spark metrics.
* For local development, use some type of CLI or Makefile to build/run the image locally. This is where the implementation diverges somewhat from using SparkOpelrator (unless you want to tell your employees that everyone needs to run Kubernetes on their local machine, which we thought would create too much friction).
* For orchestration, write a custom operator for Airflow that submits a SparkOperator resource to the Kubernetes cluster of your choosing. The operator should supervise the application state, since the SparkOperator doesn’t quite do that well enough for you. This is something I wish we had the opportunity to open source.
* Where it gets tricky is building Spark applications locally and running remotely, Say you built a job locally and tested it on a small subset of your data. Now you want to see what happens when you run across a full dataset, requiring more than 16gb of memory (or whatever the developer has on their laptop). You need some way to build your image locally but schedule it remotely. This could be done via the same CLI or Makefile, but you end up with a lot of images and it gets pretty costly. I’m sure we would have figured it out eventually if we didn’t all get laid off last month :P
* BONUS: Use Iceberg or Delta (https://iceberg.apache.org/) (https://delta.io/). These are storage formats that work with distributed file storage like HDFS or S3 to partition and query data using the Spark DataFrame API. You get time travel, schema evolution and a bunch of other sweet features out of the box. They are an evolution of Hadoop-era partitioned file formats and are an absolute must for organizations dealing with lots of data & ML infrastructure.
This post took up more time than I had wanted, but it actually feels good to write down before I forget. I hope it is useful for someone building Spark infrastructure. I'm sure others have a completely different approach, which I'd be curious to hear! As someone whose full time job was basically just to orchestrate Spark application development, I can say for certain products like this are needed in order for the ecosystem to thrive, and I would probably have given you my business had the circumstances been correct. Good luck to you and your team.
Now may be a good time to plug a project we worked on at my last gig. KeySpace uses IPFS to store PGP keys in a decentralized file system. We used a smart contract on the Ethereum blockchain to store an address-hash lookup. What this achieves is fully decentralized peer-to-peter encrypted communication. We used it to facilitate trustless OTC negotiation and trading.
Sure, we could move humans even further away from the means of production and deploy robots to do mega-scale monoculture using closed-source hardware / software, while continuing to further the monopolies held by Monsanto, John Deere, etc..
Or, we could try to shift agriculture back to a local scale, use open source hardware/software, and community-owned infrastructure to build more sustainable, polyculture food systems.
In particular, I am excited about the rooftop farming work being done in the Brooklyn Navy Yard here in NYC(1). Our Public Advocate has even discussed building-code mandated "green roof" legislation(2). CNC/Robotics & IoT are the key to unlocking urban micro-agriculture that can begin to offset some of our dependency on dirty food, and I applaud those(3) who are working on these very important problems.
Neo4j, and graph databases in general, are an excellent use case for IoT access management.
Our schema involved taking physical assets/personnel and representing them as different labels: machine, factory, production line, user, usergroup, etc. We then drew complex relationships between different user/groups in the organization and the assets they were responsible for.
At first, we used a relational database, but it soon became difficult to go more granular than simply: user belongs to usergroup, usergroup belongs to client, client has factories, factories have lines, lines have machines.
As many have pointed out here, it's not that you can't do this with non-graph databases, it just requires a more complex query layer. Neo4j allowed us to represent complex business relationships as natural language, and that really helped us as the business scaled.
I'm willing to bet money this was a cyber-terrorist attack. Unfortunately we'll never know. If a link were established, it would be the subject of a gag order on grounds of national security. But more likely, the true root cause will never be found because the authorities didn't do a deep enough forensic analysis. It's too easy to blame something this on mechanical failure, especially in America's aging infrastructure. They won't even think to look at the PLCs and control systems that control the gas pumps :/
Speaking of capacity, I just got this error trying to build an EKS cluster.
UnsupportedAvailabilityZoneException: Cannot create cluster because us-east-1b, the targeted availability zone, does not currently have sufficient capacity to support the cluster. Retry and choose from these availability zones: us-east-1a, us-east-1c, us-east-1d
But yeah, sure, keep telling yourself the AWS doesn't have a problem with power and compute capacity. Or maybe it's just poor product design?
Oden Technologies | New York, NY | Full Time, Onsite
We're a small but rapidly growing team focused on building products that allow manufacturers to improve their production processes using data. We’re working across a range of cutting edge disciplines including industrial Internet-of-Things, big data, and machine learning.
We have openings across the board:
- Frontend: help build the next iteration of our manufacturing analytics platform, a first of its kind suite of applications for analyzing real-time data, optimizing production processes, and modeling the factory of the 21st century.
- Backend: build highly available APIs in Python / Go that efficiently and reliably capture machine and human data.
- DevOps Engineer: tackle interesting problems with infrastructure in a hybrid cloud & IoT environment, such as quorum-based distributed systems and cloud/edge application deployment strategies.
- Data Scientist: build statistical and machine learning models that improve efficiency of manufacturing using the telemetry collected from machines in the field.
- Forward Deployed: integrate with different production machines, allowing for seamless transmission of data to our platform.
- Customer Success Manager: ensure that our clients are using the product to achieve the best possible outcomes for their business. This person is ideally an operations/logistics/industrial consultant, engineer, lean expert, or similar with a proven track record for demonstrating ROI.
I'll take a stab at it. I'm not an electrical engineer but I do have some experience with industrial control systems.
From Wikipedia: "Generally substations are unattended, relying on SCADA for remote supervision and control." These SCADA systems may or may not be connected to Internet, which could allow an attacker to remotely access and modify the code that controls transformers and other electrical equipment.
In another comment, someone mentioned the power company was, in this case, pumping C02 into the substation in order to contain smoldering electrical insulation. This means that, most likely, the copper conduit heat up beyond defined tolerances. This could be due to more current being carried than those conduits are rated for. Normally, the SCADA system would be responsible for keeping these currents within tolerances. What I am saying is that they could deliver a payload to the PLCs via the SCADA that could trick the transformers into taking more load than they could handle.
It'd be pretty easy to trigger a substation fire by feeding lower values into the voltage sensors than how much was actually flowing. Similar to how Stuxnet faked the values it was overriding so that everything seemed normal when in reality the centrifuges were spinning out of control. As another commenter pointed out, it's just keeping an open mind...
Crazy conspiracy theory: the Chinese and/or Russians have most likely already compromised our power grid and other infrastructure control systems and are waging covert economical war by disrupting important services.
Our company, https://oden.io, is currently hiring engineers in NYC to help tackle this problem. We are an IoT company helping to facilitate the 4th industrial revolution. Look out for our post in tomorrow's Who Is Hiring thread!
With JSON web tokens (JWT), the client or server must know the secret key used to sign the token in order to validate it, but anyone can view its payload.
Hi, I work at a manufacturing startup on pretty much the same exact problem (reducing scrape rates and downtime). I'd love to pick your brain if you have a moment to chat :) my email is [email protected]
Nice writeup! How are you finding the Apache Beam python bindings for Dataflow? We are a Python shop, but have resorted to writing our production pipelines in Java since Apache Beam doesn't yet support realtime jobs.
Oden Technologies | Data Engineering | New York, NY | Fulltime, Onsite | https://oden.io/
We are an IoT startup creating a hardware / software platform for Industry 4.0 [1] factories. We collect data from industrial machinery and analyze, aggregate and display it so that manufacturers can make more product with less material. There's a lot of exciting things happening at the company and now is a great time to get into a small (8-person) team working working on a lofty mission that will revolutionize an underserved industry.
We're looking for a data engineer with experience in building realtime and batch processing data pipelines. We ingest tens (soon to be 100s) of millions of data points daily and do complex aggregations and calculations that help our customers to hone their manufacturing processes. If you have experience with lambda architecture, timeseries / graph dbs and cutting edge data engineering technologies, we need your help ASAP.