> YAML, pivoting being done in the frontend, no symmetric aggregates
(one of the maintainers of Lightdash) You touched on some of our most interesting problems here! Would be especially interested to hear about what you liked / didn't like about symmetric aggregates in Looker and how you find dev with YAML. If you have an idea of how you'd like these to look in Lightdash, the team would be really open to making that a reality.
We're building support for dbt cloud users right now. Using dbt locally has allowed us to piggy-back on dbt's query runner and we're having to build that ourselves to support dbt cloud.
Thanks! With Hubble, we realised that while some companies wanted a separate tool to monitor data quality (e.g. data governance teams in financial integrations) most modern data teams want to test inside their existing code base (e.g. dbt). Also the majority of the company tends to interact with the data through their BI tool and we think that's where adding data quality makes most sense.
For example: flagging a dashboard as out of date, or showing that a report depends on data with failing tests.
There's rich metadata in the transform layer that just isn't getting pulled through to existing reporting/BI/viz tools.
We still have a lot of love for Hubble and data quality monitoring. By connecting dbt and Lightdash, we finally get some of those data monitoring features we always wanted.
Thanks for sharing Rakam, they always stood out for their choice of using dbt as their transform layer, it's really cool.
We're really excited to release the first public version of Lightdash!
Lightdash is an open source alternative to looker that lets analysts define data transformations and metrics in one place. Lightdash gives analysts a BI platform built on the open-source tools they already love (dbt).
We believe that the future of the modern data stack lies in having a single source of truth for all your metrics.
Tools like dbt have made it possible for analysts to manage their transformations using SQL. But existing BI tools still hide away lots of extra business logic, meaning that metrics get scattered across the company (you know those 5 different calculations of revenue XD) and data context gets lost between tools.
With Lightdash, your BI tool is fully integrated with your dbt project. This means:
- You define your metrics right beside the rest of your data transformations, in dbt.
- Developing metrics becomes lightning fast: change some SQL or a metric and immediately see your data viz update
- All your dbt metadata (column and table descriptions, lineage, freshness, test results) is kept in sync with lightdash so you don't have to try to maintain it in multiple places.
Lightdash is still in the early days and we've got lots of work to do. Today, Lightdash supports most popular databases and warehouses but is only tested with PostgreSQL and BigQuery - so, if you try it with another database, it'd be great to hear about your experience using it!
We'd love any feedback or to hear about how you're solving BI at your company today :)
Wrote a simple CLI tool that converts dbt models into looker view files. Once you've built your dbt project, run dbt2looker and copy the files over to looker.
Features:
- Auto-generates a Looker view per dbt model
- Supports dbt model and column-level descriptions
- Automatically maps raw column types to looker types
- Creates dimension groups for datetime/timestamp/date types
- Currently supports: BigQuery, Snowflake, Redshift (postgres to come)
Same experience but reverted to Mac because the hardware is unbeatable.
Had MacBooks since the white clamshells and eventually wanted to go full linux. So bought the X1 Carbon and really regretted it. Screen and trackpad were by far the worst by comparison.
So I’ve returned. Picked up a Mid-2015 15” MBP a few months back and haven't looked back.
I saw that but couldn't square it with the earlier sentence.
In my previous role we interviewed tens of small business owners and nobody knew what was in their docs. I hope insurance companies will be as bold to go beyond simplifying language but also simplifying terms (e.g. Lemonade's https://www.lemonade.com/policy-two)
> Most of the UK's biggest insurance companies produce policies that explain everything fully in plain English.
Is this an argument in favour of plain English? Insurance policy documents are incredibly hard to understand and full of bloat. They are a near-perfect example of how not to write an accessible, informative, and useful document for the intended audience.
Why is "doing software engineering" not "doing science"?
Anybody who has conducted experimental research will say they spent 80% of the time using a hammer or a spanner. Repairing faulty lasers or power supplies. This process of reliable and repeatable experimentation is the basis of science itself.
Computational experiments must be held to the same standards as physical experiments. They must be reproducible and they should be publicly available (if publicly funded).
Yes we can run the whole stack on-prem. We realised very early that on-prem would be needed for many users. So we've made it easy to spin up Hubble in a k8s cluster in your cloud or on bare metal.
Metadata is a valuable place for finding information like load times, rows inserted / updated. Currently we just rely on read-access and raw SQL. A common way users are doing this now (and we are internally for our analytics data) is using, for example, the Fivetran logs table to monitor ingestion times and inserted rows, rather than querying the raw tables.
For CICD, absolutely we want to support this as well as stopping/conditional execution in DAGs (e.g. airflow). We’re launching webhooks very soon
Yes, we store the historical value of each test so you can always scroll back through time and see the state of the data warehouse at any given point.
For example, if you have a test that counts the number of rows "COUNT(*)" - that value will be recorded. So you can look back an hour/day/week and see how many rows the table had without executing any SQL. These values are stored in a time series db, so querying history is fast.
Our tech stack: monolith backend in python + postgres + react. The test themselves are all SQL queries and run in the data warehouse.
Yeah we called this project hubble long before we were worried about SEO.
Actually, the name does relate back to Edwin Hubble. We previously worked together on an internal data tool called Telescope (it was used for annotating medical images for computer vision). The telescope project slowly evolved into the product we have today. So we changed the name to our favourite telescope. I have a fondness for the Hubble telescope: there was a huge poster of it on the way into the computational physics dept. and takes me back to the grad school days!
Thanks! We love DBT and take a lot of inspiration from their work. We’re putting a lot of effort into suggesting the right tests based on the data types, sources, and field names. A lot of these tests are pretty repetitive to write so we want to make it easy to spin them up.
We’ve also found that keeping a history of the state of the warehouse over time is really useful context for determining whether a test has failed (example: this table tends to update every 30-40 minutes so we’ll set a threshold at an hour).
We also handle the scheduling, which is surprisingly annoying to manage (we built a couple of internal tools for this in the past). That’s something we really missed with great expectations (you get this with DBT cloud).
Testing files is an interesting use case, to an extent we support this using Athena or Bigquery external tables for json/csv/parquet. We’re intentionally limiting it to SQL for now.
Full table scans can get expensive. We’re adding support for incremental tests so for append-only tables you’ll only test the recent rows. This is especially useful if you use partitioned tables in bigquery.
Actually in the first version of the product we automatically tested every column in every table. The tests are more selective now, which is partially due to cost and partially because nobody wants to navigate through 10,000 tests.
Redshift will be supported this week! We have a list of new sources to get through and it’s right at the top. We’ve been emailing over the IP for whitelisting but we’ll add it to the connection page too.
As for pricing, we’re experimenting. Our costs do scale with number of tests (more scheduled tasks, more historical results stored). At the moment we retain the last month or so of test results, which is manageable for pretty large workloads.
> This is the crux of the AI business dilemma. If the economics are a function of the problem – not the technology per se – how can we improve them?
The article focusses on the costs of resources to build a model (annotated data + compute) but the economics are also affected by the ongoing cost of making a prediction error. False positives and false negatives usually have a different cost and each user might have their own preferences:
e.g. "show me all the content that's a bit relevant" vs "show me just the content that's really relevant".
If you can write out the loss function in $$$ terms not just accuracy, then you're closer to either abandoning the problem or finding a profitable AI model.
Ok I admit, the blank keycaps are just to look cool.
> I have kind of un-learned Qwerty. When I have to go to someone else's machine at work (game dev studio) to diagnose a crash, Qwerty feels very foreign and I look like an idiot when I go to type (looking up and down, making lots of mistakes, and generally typing like an old uncle who has never used a computer).
This 100%. Was helping a colleague to debug, started typing... "you know what, I might ask someone else"
After 20 years of QWERTY I could sort-of touch type but it’s very hard to unlearn bad habits and position your hands properly.
The best decision I made was to switch to DVORAK. I was very unproductive for about a month.
Training involved blank key caps and a small print out of the layout above my monitor. The result is I type much faster, with fewer errors, and I never look down (usually pointless since your keyboard will be qwerty labelled anyway!)
EDIT: doesn’t have to be Dvorak but the point is switching layouts forces you to learn from first principals: feeling out the keyboard notches (F and J keys on standard US) and going from there
(one of the maintainers of Lightdash) You touched on some of our most interesting problems here! Would be especially interested to hear about what you liked / didn't like about symmetric aggregates in Looker and how you find dev with YAML. If you have an idea of how you'd like these to look in Lightdash, the team would be really open to making that a reality.
For pivoting in the backend, this is coming! Issue here: https://github.com/lightdash/lightdash/issues/2907