HackerTrans
TopNewTrendsCommentsPastAskShowJobs

MatthausK

no profile record

Submissions

Show HN: Automatically extract data from APIs with dlt and OpenAPI

colab.research.google.com
7 points·by MatthausK·il y a 2 ans·1 comments

Show HN: Dlt – Python library to automate the creation of datasets

colab.research.google.com
114 points·by MatthausK·il y a 3 ans·54 comments

comments

MatthausK
·il y a 2 ans·discuss
one of the dltHub founders here - we aim to address this in the coming weeks
MatthausK
·il y a 2 ans·discuss
one of the dltHub founders here - we aim to address this in the coming weeks
MatthausK
·il y a 3 ans·discuss
Pulling from and into production databases is one of the early favourites from our dlt user base. Some reasons explained here in this MongoDB example (https://dlthub.com/docs/blog/MongoDB-dlt-Holistics)
MatthausK
·il y a 3 ans·discuss
We hear a lot about the dlt & AWS Lambda. We have currently one user working on the use case (see our Slack https://dlthub-community.slack.com/archives/C04DQA7JJN6/p169...)
MatthausK
·il y a 3 ans·discuss
Thanks for your vote of confidence & support Max!
MatthausK
·il y a 3 ans·discuss
We took at least one immediate practical good piece of advice out of this which is that we should release a conda package and make sure that dlt works in it.
MatthausK
·il y a 3 ans·discuss
1) Yes. We support all the databases and buckets as data sources as well. Some examples: - get data from any sql database: https://dlthub.com/docs/dlt-ecosystem/verified-sources/sql_d... or https://dlthub.com/docs/getting-started#load-data-from-a-var... - do it super quickly with pyarrow: https://dlthub.com/docs/examples/connector_x_arrow/ - get data from any storage bucket:https://github.com/dlt-hub/verified-sources/tree/master/sour... 2) Strictly technical answer: on the code level sources and destinations are different Python objects so the answer is no:) but you as a user rarely deal with them directly when coding
MatthausK
·il y a 3 ans·discuss
You can use pydantic models to define schemas, validate data (we also load instances of the models natively): https://dlthub.com/docs/general-usage/resource#define-a-sche...

We have a PR (https://github.com/dlt-hub/dlt/pull/594) that is about to merge that makes the above highly configurable, between evolution and hard stopping: - you will be able to totally freeze schema and reject bad rows - or accept the data for existing columns but not new columns - or accept some fields based on rules'