From zero to Production Elixir in 1 month(medium.com)
medium.com
From zero to Production Elixir in 1 month
https://medium.com/onfido-tech/from-zero-to-production-elixir-in-1-month-1-2-a9d051191ad7
19 comments
The author seems to be unaware of being able to upload directly to S3 via JS eliminating the need of even the Phoenix app.
I'm not sure if it's a recent feature, but I was unaware of it until recently. However, you'd still need an app to sign the upload URL. Otherwise, you'd have to expose the AWS secret key to the client.
It's mostly a matter of setting up an endpoint in your app to sign manifests using the secret key. It's give or take a dozen lines of code on the backend (to do it in an "unsophisticated way"), and was one of the first things I wrote in Elixir! These two links should get you on your way:
1. http://bencoe.tumblr.com/post/30685403088/browser-side-amazo...
2. https://aws.amazon.com/articles/1434/?tag=vig-20
1. http://bencoe.tumblr.com/post/30685403088/browser-side-amazo...
2. https://aws.amazon.com/articles/1434/?tag=vig-20
I should make a service that handles this for people. That has to be in demand, no? :P
What type of security implications are there from this? Are things like rate-limiting (by time period, all-time) or access control handled properly?
I clicked on this article because I needed to do something similar this week, and ended up using the JS-based approach. I’ve built something like this with S3 before, but ended up doing it with Google Cloud Storage this time around. The process looked like this:
1) Client-side app receives an image (via drag-and-drop or file picker)
2) Client hits an authenticated endpoint on backend with the name of the file, which returns a “signed URL” (authenticated, with customizable expiration) from Google Cloud Storage
3) (This part is particular to GCS, and I think unneeded for AWS/S3) Hit this signed URL in the client, and receive a final, PUT-able URL
4) Upload the file using the final authenticated URL from Google Cloud Storage
Like I said, I think with AWS/S3 you can just request a signed URL and send data to that URL directly. Delegating as much communication with GCS to the backend was important for my project [1] but you might be able to just manage it with a JS lib.
[1] Streaming music, so trying to abstract the actual files from the client-side as much as possible to discourage downloading the raw audio files
1) Client-side app receives an image (via drag-and-drop or file picker)
2) Client hits an authenticated endpoint on backend with the name of the file, which returns a “signed URL” (authenticated, with customizable expiration) from Google Cloud Storage
3) (This part is particular to GCS, and I think unneeded for AWS/S3) Hit this signed URL in the client, and receive a final, PUT-able URL
4) Upload the file using the final authenticated URL from Google Cloud Storage
Like I said, I think with AWS/S3 you can just request a signed URL and send data to that URL directly. Delegating as much communication with GCS to the backend was important for my project [1] but you might be able to just manage it with a JS lib.
[1] Streaming music, so trying to abstract the actual files from the client-side as much as possible to discourage downloading the raw audio files
If we want to store the object's URL in a relational database, should the browser send another HTTP request to our service? Wouldn't we lose the object if there's a network error during the interval between the two requests?
Yes, one option is to do as you've described, for example, use JS to upload the file, then submit a form as normal with a file url / id in the form.
However, you can also create a Lambda that is triggered whenever a file is uploaded to a given bucket and directory. With this, you can react to the file being received and update your database without the need for a second request.
However, you can also create a Lambda that is triggered whenever a file is uploaded to a given bucket and directory. With this, you can react to the file being received and update your database without the need for a second request.
s3 requires a signature as well. Without requiring a signature, anyone could upload anything to your s3 account. With a signature, you choose when and where people can put their files.
There's also the fact that Puma has slow client support so this issue isn't even an issue.
as much as I love Elixir (and Erlang/BEAM) I get uncomfortable when people say "I heard it was a magic bullet for scalability and concurrency[...]. the language provides primitives that can, in certain circumstances, make it easier to scale horizontally, but there's no magic. and there are certainly tradeoffs (esp with regards to managing state) that can actually make your code harder to scale.
dig the post, just an observation. for anyone new to the language reading this please understand that Elixir isn't a super fast/concurrent Ruby replacement. but it is a hell of a lot of fun to program in.
dig the post, just an observation. for anyone new to the language reading this please understand that Elixir isn't a super fast/concurrent Ruby replacement. but it is a hell of a lot of fun to program in.
I feel like scala went through the same thing with akka, just with less blog hype. It's a panacea until someone asks how you manage state.
This feels like a bit... too much. If you’re really going for microservices, that means every service will be small and specialized. Do you really need to split it further into so many sub-apps? I feel like the S3 app is going to end up as one module with a handful of functions.
These are medium sized. Remember in elixir it's perfectly convenient to map across a list by spawning a process for each element in the list and sending the function you want to map to that process, and then they collapse down and get reduced.
A process (or application) in elixir is no big deal, just think of it as a module or object that lives in its own process, probably has a set of processes and all that...
If you were writing this in go you might call it a monolith because all of those "Apps" would be in one single binary executable.
A process (or application) in elixir is no big deal, just think of it as a module or object that lives in its own process, probably has a set of processes and all that...
If you were writing this in go you might call it a monolith because all of those "Apps" would be in one single binary executable.
My comment is about code organization, not process usage. You can have 6 separate apps and still do everything in a single process.
By splitting into lots of sub-apps, you end up with a deep folder structure like `my_service/apps/s3/lib/s3/query.ex`, each app having its own versioning, configuration, tests, etc. That makes sense for a sizable codebase, but for a microservice, I don't think the added complexity is worth the benefit. It can be organized very cleanly with just modules and folders.
By splitting into lots of sub-apps, you end up with a deep folder structure like `my_service/apps/s3/lib/s3/query.ex`, each app having its own versioning, configuration, tests, etc. That makes sense for a sizable codebase, but for a microservice, I don't think the added complexity is worth the benefit. It can be organized very cleanly with just modules and folders.
These aren't sub applications, they're more similar functionally to classes, although they tend to end up a little larger in a lot of cases.
They are applications. That's what an umbrella does, it groups multiple Erlang applications into one project.
Code?