I like the idea of supporting multi cloud, not cloud vendor lock-in, but I still doubt providing a tool can solve this.
We have been building a DBaaS(database as a service) for many years. Until now, we have only supported AWS and GCP, have not supported Azure. It is really hard to let your product run stably on multi clouds.
For example, not all the cloud vendors provide the same infrastructure at the same time. We wanted our database run on K8s, but several years before, only GCP provided its managed K8s - GKS, so we had to provide our DBaaS on GCP at first. We also wanted to let our customer use PrivateLink to access our service, but GCP had supported too late than AWS, so we had to provide the VPC mechanism for a long time.
Even all the cloud vendors provide the same service, there are still some differences in the details above. E.g, one customer complained us that they met a backup problem on Google Cloud Storage, they found that the speed of our backup was not the same as our product spec statement. Then we found that we only fully tested this feature on AWS S3.
So Now even our DBaaS runs well both on AWS and GCP, we still don’t have the confidence to deploy the service on Azure. We plan to devote a quarter to testing to run DBaaS on Azure in the end of this year.
IMO, I still recommend handling the interfacing with the cloud directly to let us know the clouds better. Of course there are some awesome tools helping us to do some functional encapsulation, hiding the underlying implementation, but we must ensure these tools are easy enough to maintain.
IMO, Using AI for image generation is the trend, whether we like it or not. But for me, as a software engineer, I still insist on creating pictures for my own blogs or presentations, etc.
I'm not a painter, of course I paint badly. But I find that it is not so difficult to create a picture that expresses my ideas. Mostly I try to use https://excalidraw.com/ for sketching hand-drawn, and find the free stock pictures from https://www.freepik.com/.
I think this is another joy of creation, like programming :-)
Using map or array instead of switch-case is a common optimization way, but please do a benchmark if you have no confidence whether you need do or not.
When we started to build the distributed database TiDB, we had the similar discussion - whether we need to provide a so strong consistence for the customers? Inspired by Google Spanner, we also wanted to let the application programmers to take easy and not struggle with the consistent problems in applications layer by themselves.
So we decided to provide the strong consistency. But unlike Google Spanner, we couldn’t use TureTime API, so we introduced a standalone timestamp service named TSO to allocate timestamp. Using TSO is simple to guarantee all transactions’ order, but this introduces another problem - if the TiDB is deployed crossing different regions, the transaction latency is high.
Another example for consistency is from our realtime analytical processing design. TiDB is a HTAP(hybrid transaction/analytical processing) database. It is normal to keep consistency in the TP part, but for the AP part, is it necessary to keep consistency? If there is an update in the table, does the customer really need to get the latest data from the AP part? Here we still choose “YES”, we want our customers to focus on their business and not worry whether the query data result is consistent of not.
Of course, keeping consistency in the database comes at a cost. Now we have been trying our best to optimize the latency and performance problems :sob:
IMO, we choose a right way. We now have supported strong consistency at first, and then we can provide a loose consistency for performance too, but on the other hand, if we only build a loose consistent database at first, it is hard to provide a strong consistency for the customers.
We build our binary first with one image as the builder image, then use `copy` to copy the binary from the builder to the final executable image like alphine.
an example Dockerfile likes:
FROM golang:1.18.1-alpine as builder
# RUN apk add, wget, etc, and build the binary
FROM alpine
# or FROM scratch
COPY --from=builder builder/binary /binary
ENTRYPOINT ["/binary"]
As a chief engineer, I started to work with my colleagues to build an open source distributed, relational database from scratch 7 years ago. At that time, MySQL Sharding, or NoSQL was the popular solutions for scalability. So the first challenge for us is how to design our system architecture? Followings are something we do:
- Paper. We learned a lot from the paper, like Google Spanner, Google Percolator, Raft Algorithm, etc.
- Learn from the open source projects and Leverage the power of community. E.g, we learned storage engine from RocksDB, ClickHouse, etc. to build our own engine. We also let the community contribute ideas and codes to our product.
- Try to link people as many as you can and learn from them. E.g, when I read a paper, mostly I will try to contract the authors and have a discussion about the paper with them, at the same time, I also ask them to introduce their friends to me for further communication.
- Sharing. The more you go out and share like Meetup, Blog, Webinar, etc., the more you get.
IMO, multi cloud is the future, just as eggs can’t be in one basket. I’ve been building a distributed database for more than 7 years, and find that many of our customers had been using different databases in different cloud vendors before they migrate to our product. E.g, one customer used Azure MySQL, Cloud SQL for MySQL, Aurora for MySQL.
Another thing I find vendor lock-in is not nice, for example, one customer, which is also a big customer of one cloud vendor, has suffered a lot from the cloud vendor’s own database. The customer finds the database it is not stable when its data size grows, has to spent a long and tough time to upgrade to the new version regularly, couldn’t get a good-enough service it expected before.
IMO, not only for our database, but also for other applications on the cloud, supporting multi cloud is mostly a MUST. And we also need to build an easy-to-use product, provide a good-enough service for our customers.
I've been building a distributed database product for more than 7 years, managing a team of nearly 300 R&D staff and serving hundreds of customers.
Customer Support is hard for us. Not only the number of OnCalls, but also the complexity of the product causing that we can't solve the problem easily and fast. We are under a lot of pressure.
To solve this, I have tried following ways:
- Build an independent team (may name Escalation Engineering or L3 in other companies, here let's name L3) to focus on the customer problem solving.
- Establish the OnCall duty mechanism, some engineers (maybe 10 one time) must join the L3 for two weeks only focusing on OnCall. Why it is so important, because I believe the quick way to learning our product is from OnCall.
- Get to know our customers better. We are a database product, if we can know our customers' business workloads, we can give them more advices to let the business application run more better in our product.
- Build many Diagnostic tool to solve the OnCall easily.
Finally, I would like to say that although customer support is sometimes painful, it allows us to better understand our customers and know the strengths and weaknesses of our products.
Very sad to hear this news, but not so unexpected.
As a distributed database engineer, we have been working with some universities and exploring the optimization of database on Intel Optane for a long time, and even have published some papers.
But everything was only in the experimental stage, it was difficult for us to use the research successfully in production, because nearly all of of our customers hadn't bought Intel Optane. They thought Optane was too expensive and didn't find suitable scenarios. As far as I know, the only successful story was from one of our customer, he used Optane for Redis which had an acceptable latency and better cost-effective.
Things go worse when the cloud is raising. We find that we can't use Optane on the cloud, so our later priority is to explore the optimization on the cloud disk like EBS GP3, etc.
I joined a startup 7 years ago as the first employee to build an open-source distributed database and have been working as a chief engineer since then. I write a lot of code, but most of the time I'm actually doing something other than code:
I visit customers to show our product proposition and listen to their requirements, write articles online to introduce our product, participate in meetups to talk about the technical architecture, operate the community to attract contributors, talk to ISVs or other companies about product integration...
Many of them are not programming, but why I still have to do these? I think even we have a good enough product, if nobody knows about it, it is nothing. We must try our best to improve the awareness of our product.
By the way, doing these things also make me grow faster, become more focused and even better at programming. So if you are a Software Engineer, please have a try, this is another kind of fun :-)
For me, programming not only brings me pleasure, it also brings me peace of mind. Programming can quickly put me in a state of mind flow.
My most recent experience was when I was learning Next.js. As a distributed database engineer, I nearly have no knowledge of web development, but I really want to show some database demos on the Web. So one night, I began to learn Next.js from scratch - I tried to connect to the database, query the result and then render the chart. After I deployed the application on Vercel, two hours had passed.
I felt tired but very excited. I believe this is the pleasure of programming :-)
As a back-end engineer focusing on distributed database, I nearly have no knowledge of web development. I rarely write JS, can't tune CSS, etc.
But when I met Next.js and Vercel, I found that they are very friendly to beginners. I can build the demos on the web, more beautiful and intuitive (Previously, I had only to build demo on the console).
Interesting feature, I think we can learn from this in our product.
IMO, it is still not safe even we know there are no queries running for this table. You may still meet a scenario that when you type `drop table`, another guy begins to run the query at the same time.
As the maintainer of another database, We have been trying our best to improve this scenario too.
Early on, we have provided a feature called `recover table` to recover table immediately after you wrongly drop the table. But this still can't avoid affecting current running queries on this table.
We call this problem `DDL affects DML`, and now we try to introduce a Table meta lock to guarantee that no any query is running when the DDL executed. We hope we can release this feature in the end of this year.
I am very agree with some options of this blog. As the maintainer of the open source distributed database TiDB https://github.com/pingcap/tidb, we also face the same problem of choice. We have a community version, an enterprise version(of course, we must sell it to our customers to earn money) and also a cloud service named TiDB cloud.
Seven years before, we started to build TiDB to solve MySQL sharding problem, yes, "I Don’t Want to Shard (MySQL)" https://news.ycombinator.com/item?id=32041656 too. At that time, we only had a community version of TiDB.
After we had developed TiDB for more that one year, we needed to consider earning money because we (PingCAP) are a company. So we built an enterprise version to sell to our customers. Unlike other companies do mostly, they separate the community and enterprise version a lot, there are even some killing features that only exist in the enterprise version. But we believe that all our customers, whether they pay for us or not, must get the benefits, nearly the same value from TiDB. So we decide to keep the same between the community and enterprise. The only difference for the enterprise version is that the version contains another two tiny features - audit log and IP white list. As far as I know, no community user asks us for these two feature util now.
Things were going well, and a few years later, we met the same dilemma for TiDB community version and TiDB cloud service https://tidb.cloud. Of course, at this time, we still insist on that the TiDB must be open source, any improvement for the cloud will be contributed to the community version at first, then we deploy to our own service later. This mean anyone can deploy TiDB easily on the cloud too.
This is our open source decision. Thanks to this decision, our products are now increasingly well known, you can check the insights from https://ossinsight.io/analyze/pingcap/tidb/.
Gitea is very easy to use, but I find the Activity feature is a little slow.
I experienced the "Try Gitea" service and migrated our TiDB repo https://github.com/pingcap/tidb to it. When I clicked the Activity tab and selected "1 year" period, I found the page loading was so slow, nearly 90s. And I also found that this Activity doesn't have a Cache, I re-selected "1 year" again, and the page loading was nearly the same time.
I guess Gitea uses git command to traverse all the logs for the period every time. Maybe it can use a database to speed up, or like Github only provide at max "1 month" period.
We have been building a DBaaS(database as a service) for many years. Until now, we have only supported AWS and GCP, have not supported Azure. It is really hard to let your product run stably on multi clouds.
For example, not all the cloud vendors provide the same infrastructure at the same time. We wanted our database run on K8s, but several years before, only GCP provided its managed K8s - GKS, so we had to provide our DBaaS on GCP at first. We also wanted to let our customer use PrivateLink to access our service, but GCP had supported too late than AWS, so we had to provide the VPC mechanism for a long time.
Even all the cloud vendors provide the same service, there are still some differences in the details above. E.g, one customer complained us that they met a backup problem on Google Cloud Storage, they found that the speed of our backup was not the same as our product spec statement. Then we found that we only fully tested this feature on AWS S3.
So Now even our DBaaS runs well both on AWS and GCP, we still don’t have the confidence to deploy the service on Azure. We plan to devote a quarter to testing to run DBaaS on Azure in the end of this year.
IMO, I still recommend handling the interfacing with the cloud directly to let us know the clouds better. Of course there are some awesome tools helping us to do some functional encapsulation, hiding the underlying implementation, but we must ensure these tools are easy enough to maintain.