HackerTrans
TopNewTrendsCommentsPastAskShowJobs

MatthausK

no profile record

Submissions

Show HN: Automatically extract data from APIs with dlt and OpenAPI

colab.research.google.com
7 points·by MatthausK·2 वर्ष पहले·1 comments

Show HN: Dlt – Python library to automate the creation of datasets

colab.research.google.com
114 points·by MatthausK·3 वर्ष पहले·54 comments

comments

MatthausK
·2 वर्ष पहले·discuss
one of the dltHub founders here - we aim to address this in the coming weeks
MatthausK
·2 वर्ष पहले·discuss
one of the dltHub founders here - we aim to address this in the coming weeks
MatthausK
·3 वर्ष पहले·discuss
Pulling from and into production databases is one of the early favourites from our dlt user base. Some reasons explained here in this MongoDB example (https://dlthub.com/docs/blog/MongoDB-dlt-Holistics)
MatthausK
·3 वर्ष पहले·discuss
We hear a lot about the dlt & AWS Lambda. We have currently one user working on the use case (see our Slack https://dlthub-community.slack.com/archives/C04DQA7JJN6/p169...)
MatthausK
·3 वर्ष पहले·discuss
Thanks for your vote of confidence & support Max!
MatthausK
·3 वर्ष पहले·discuss
We took at least one immediate practical good piece of advice out of this which is that we should release a conda package and make sure that dlt works in it.
MatthausK
·3 वर्ष पहले·discuss
1) Yes. We support all the databases and buckets as data sources as well. Some examples: - get data from any sql database: https://dlthub.com/docs/dlt-ecosystem/verified-sources/sql_d... or https://dlthub.com/docs/getting-started#load-data-from-a-var... - do it super quickly with pyarrow: https://dlthub.com/docs/examples/connector_x_arrow/ - get data from any storage bucket:https://github.com/dlt-hub/verified-sources/tree/master/sour... 2) Strictly technical answer: on the code level sources and destinations are different Python objects so the answer is no:) but you as a user rarely deal with them directly when coding
MatthausK
·3 वर्ष पहले·discuss
You can use pydantic models to define schemas, validate data (we also load instances of the models natively): https://dlthub.com/docs/general-usage/resource#define-a-sche...

We have a PR (https://github.com/dlt-hub/dlt/pull/594) that is about to merge that makes the above highly configurable, between evolution and hard stopping: - you will be able to totally freeze schema and reject bad rows - or accept the data for existing columns but not new columns - or accept some fields based on rules'