Hey all,
Vedran here co-founder of Treblle (API observability platform). In short we've got SDKs that sits on top of your API and logs every single API request that happens. We then give you some really cool things like: auto-generated docs, analytics, quality scoring, real-time logging and monitoring, bug tracking...the works.
Right now we run our infrastructure on top of AWS. Basically the request goes from your API to AWS API Gateway then to a super simple Lambda function which simply stores the request body as JSON in S3. That's it. Afterwards we use SQS, more Lambdas and our Laravel based ETL to make sure we actually give you as the user rich and meaningful insights. More detailed blog about how it all works can be found here: https://blog.treblle.com/diary-of-a-wimpy-devops-engineer-monoliths/.
The thing I'd like to speed up and improve is the first part - let's call it the ingest part. I'd like to move it to the Edge : ) Using maybe AWS Lamda@Edge or Cloudflare Workers or even something else you all suggest.
Reducing latecy is key here so my idea was maybe to completely remove the need for the Gateway and have the SDKs directly call a Lamda function. In that case I would need to implement some sort of a usage limit check on the Lambda to see if the the user has reached a maximum of their plan. I could use Redis or Memcached to store an API and the number of API calls it made and have it expire in a month or something similar. Has anyone done something like that before?
Anyone running something similar on Workers? I think Workers has KVs which could be use to solve this as well. If we move to R2 things might be even cheaper.
Right now we run our infrastructure on top of AWS. Basically the request goes from your API to AWS API Gateway then to a super simple Lambda function which simply stores the request body as JSON in S3. That's it. Afterwards we use SQS, more Lambdas and our Laravel based ETL to make sure we actually give you as the user rich and meaningful insights. More detailed blog about how it all works can be found here: https://blog.treblle.com/diary-of-a-wimpy-devops-engineer-monoliths/.
The thing I'd like to speed up and improve is the first part - let's call it the ingest part. I'd like to move it to the Edge : ) Using maybe AWS Lamda@Edge or Cloudflare Workers or even something else you all suggest.
Reducing latecy is key here so my idea was maybe to completely remove the need for the Gateway and have the SDKs directly call a Lamda function. In that case I would need to implement some sort of a usage limit check on the Lambda to see if the the user has reached a maximum of their plan. I could use Redis or Memcached to store an API and the number of API calls it made and have it expire in a month or something similar. Has anyone done something like that before?
Anyone running something similar on Workers? I think Workers has KVs which could be use to solve this as well. If we move to R2 things might be even cheaper.
Thank you!