wsh91 · HackerTrans

Hi there! One thing you might consider is using a function to replicate your Firestore data to BigQuery, depending on the value the latter would add. (Very fast OLAP; I'm a big fan.)

(Cloud Firestore eng. here.)

wsh91

7 lat temudiscuss

Hi there. :) I'm an engineer on the Cloud Firestore team. I can't speak to if or when we'll be able to remove this restriction, but please know that the folks working on this (like me) are very aware of what a pain in the butt this is. I'm sorry that I can't give you a more satisfying answer, but I hope we'll be able to change this soon. Thanks for being a customer!

wsh91

7 lat temudiscuss

Cloud Spanner is also relational. :)

(Fellow GCPer.)

wsh91

7 lat temudiscuss

An export is one component of a backup solution. :) Scheduling is another (https://firebase.google.com/docs/firestore/solutions/schedul...). (Checking restores is critical, too; that's also feasible [managed import into a new cloud project] but requires a bit more legwork.)

(I work on Cloud Firestore.)

wsh91

8 lat temudiscuss

Is there anything we could be doing to serve you better?

wsh91

8 lat temudiscuss

a) Cloud Firestore supports transactions across the entire database. You can learn more about them here: https://cloud.google.com/firestore/docs/manage-data/transact....

b) Given that the primary use case for namespaces was/is multitenancy, it's not clear to me why you'd want to transact across them. Nevertheless, you can. What's leading you to draw this conclusion?

wsh91

8 lat temudiscuss

Cloud Firestore (the next generation of Cloud Datastore) removes those limitations. https://cloud.google.com/firestore/docs/manage-data/transact...

It also supports the Cloud Datastore API.

(I work on it!)

wsh91

8 lat temudiscuss

Hey there--I'm an engineer on the Cloud Datastore team. I'd love to know more about what your needs are if you're willing to share.

wsh91

9 lat temudiscuss

This (fantastic) paper is cited in the Google Megastore paper [1] and we actually use the term "entity group" in our product [2] indirectly because of it. Pat Helland is always worth reading; thanks to the submitter for the reminder.

[1] https://static.googleusercontent.com/media/research.google.c...

[2] https://cloud.google.com/datastore/docs/articles/balancing-s...

wsh91

9 lat temudiscuss

Have a look at https://cloud.google.com/compute/docs/load-balancing/ipv6.

(Disclaimer: I work on GCP, albeit not on networking stuff.)

wsh91

9 lat temudiscuss

Only because this question has been asked multiple times at TGIF (regular all-hands meeting).

wsh91

9 lat temudiscuss

That's not true. Anyone has access to 20% time, the question is just whether or not it makes sense for individual people.

Source: I work here and take advantage of 20% time.

wsh91

9 lat temudiscuss

Heh. :)

One technicality: DynamoDB's pricing is based on throughput, as you know, but it's provisioned throughput. (You manage your capacity unit allocations yourself, at least that was the case a little while ago.) You're charged money for how much you provision regardless of whether or not you actually consume the capacity units. Our pricing model (like Fauna's) only takes ops and storage into consideration.

wsh91

9 lat temudiscuss

I can't speak to the SDK parts, but take a look at Firebase's suite more broadly. :) Viz a viz Cognito, check out Authentication and the Realtime Database in particular.

wsh91

9 lat temudiscuss

Firebase's Realtime Database does this, too.

https://firebase.google.com/pricing/

(Disclosure: I work on Cloud Datastore, which is another Google Cloud product; it, too, does pricing based on ops and storage rather than provisioning, as my colleagues note.)

wsh91

9 lat temudiscuss

As you noted earlier, Megastore has a schema and we don't, so we have our own index implementation, yes. :)

wsh91

9 lat temudiscuss

Perhaps I've misunderstood you, but I'm pretty sure cross-partition (I assume you mean cross-entity-group in our terms) transactions are in fact consistent (not totally sure what you mean by transactions being consistent, per se; if you're talking about serializability, at least, we are). Explicitly (from [1]):

  Queries that participate in a transaction are always strongly consistent.

And the consistent write throughput to which you refer means sustainable write throughput per entity group. We can burst much higher.

It would be much easier to assess the relative consistency models of our products if FaunaDB had documentation with respect to its claims. We have a litany of pages about ours (e.g. [2] and [3]).

[1] https://cloud.google.com/datastore/docs/concepts/structuring...

[2] https://cloud.google.com/appengine/articles/transaction_isol...

[3] https://cloud.google.com/datastore/docs/concepts/transaction...

wsh91

9 lat temudiscuss

Again, neat stuff, but that's not what this link claims:

  * The first serverless database
  * The first active-active multi-cloud database
  * The first strongly-consistent multi-region database available to the public

None of these are firsts. I don't know if our (GCP) services are themselves the first of their kind (it's an ambitious claim and, as an engineer, I try to be careful about those), but Datastore meets at least two of those three and predates FaunaDB by several years.

wsh91

9 lat temudiscuss

I'm familiar with both; what disappoints me is the claim of novelty here with respect to autoscaling. That's just not true. To quote you:

"A serverless system must scale dynamically per request. Current popular cloud databases do not support this level of elasticity—you have to pay for capacity you don’t use. Additionally, they often lack support for joins, indexes, authentication, and other capabilities necessary to build a rich application."

That first criterion we absolutely meet, today. Cloud Datastore has been doing that for eight years now. We don't have joins, but we do have indexes, auth, multi-region replication and a whole lot more.

wsh91

9 lat temudiscuss

:)

Here's a paper with which you might already be familiar, but it's one of the citations for the Megastore paper: http://adrianmarriott.net/logosroot/papers/LifeBeyondTxns.pd.... You'll probably enjoy it (if you haven't already!).

comments