I like the YAML abstraction. This should make it easier to programmatically try and evaluate multiple configurations for the whole AI pipeline (not just the LLM) against a dataset or real users through an API deployment.
Some feedback: It would be great to see in one place all the supported fields and values for the YAML config.
Nice, thanks! I was reading https://pathway.com/developers/user-guide/deployment/persist.... If I understand correctly you persist both source data and internal state, including the intermediary state of the computational graph. And you only rely on the backend to recover from failures and upgrades. So if I want to clone a Pathway instance, I don't need to reprocess all source data, I can recover the intermediary state from the snapshot.
Interesting implementation! For complex stream and text processing, I also prefer processing data in memory with Python (ETL) rather than SQL in the warehouse (ELT).
I see the ingested documents in the data folder don't have an id field, only a doc field.
{"doc": "Using Large Language Models in Pathway is simple: just call the functions from `pathway.stdlib.ml.nlp`!"}
What if I pass two contradictory statements? Is there a way to remove (or better update) a document with a new version?
For example, if I am ingesting some public docs, and I update a doc page. How do I make so that it only takes the answer from the latest document version?
Know that there is also a CLI to manage configurations defined in YAML files. And a few options to deploy Airbyte in "one click". It's all in the README, sorry to hear you didn't find your way around our docs... There are a number of growing features and deployment options now.
Oh, declarative doesn't necessarily mean no-code. Airbyte data integration connectors are built with an SDK in Python, Java, and a low-code SDK that was just released...
You can then build custom connectors on top of these and many users actually need to modify an existing connector, but would rather start from a template than from scratch.
Nice article! I also tend to favor SQL for simple querying and data processing with dbt, but when I need to unit test some complex logic, I prefer Python.
I like how Prefect is positioned as an orchestrator for the modern data stack.
Airflow also started as an orchestrator, but then they tried to cover all sorts of other use cases like ETL/ELT pipelines with transfer and transformation operators...
I feel like Prefect focuses on doing one thing, orchestration, and then integrates with other data tools.
Airbyte CDC is based on Debezium, but Airbyte abstracts it away and make it easier to CDC from Postgres, MySQL, MSSQL to any supported destination (included S3). Here is the doc for CDC: https://docs.airbyte.io/understanding-airbyte/cdc
I guess one benefit is that you can use Airbyte for all your data syncs, CDC and non-CDC. You can give it a try with your own data, and see if it's easier for your team. You can run Airbyte locally with Docker Compose: https://docs.airbyte.io/quickstart/deploy-airbyte
Nice work there! I also think that the next challenge for data teams is all this data documentation and discovery work.
I still think that Airflow is great for power data engineers. Airbyte and dbt are positioned to empower data analysts (or lazy data engineers like me) to own the ELTs.
Some feedback: It would be great to see in one place all the supported fields and values for the YAML config.