dlsspy

13 lat temudiscuss

Should also note that the "arrays are values" thing doesn't actually matter in practice. I've written a wide variety of go code in the last > 4 years and I can only think of two places I've used an array.

dlsspy

13 lat temudiscuss

"<-c", regardless of what's on the left side of it will block in every case until there's a messages available in the channel (note that close is a message).

I just notice this section has been removed. I'll be less angry now. :)

dlsspy

13 lat temudiscuss

This is very much false:

While this will return immediately:

    v, ok := <- c

dlsspy

13 lat temudiscuss

> Your modified Go server doesn't return "Pong" for "Ping".

The program doesn't read the result, so it doesn't matter. Returning Pong isn't harder, but why write all that code if it's going to be ignored anyway?

> It's fundamentally different. - you're firing off all your requests before waiting for any replies, and so hiding the latency in the more common RPC style request-response chain, which is a real problem.

As I said, the program isn't correlating the responses with the requests in the first place -- or even validating it got one. I don't know scala, but I've done enough benchmarking to watch even less sophisticated compilers do weird things with ignored values.

I made a small change that produced semantically the same program (same validation, etc...). It had similar performance to the scala one. If you don't think that's helpful, then add further constraints.

dlsspy

13 lat temudiscuss

It's not obvious to me what that does. Does that only service one of the 100 clients or one of their 10,000 pings each? Or both?

dlsspy

13 lat temudiscuss

Did you consider running the go client against the scala server and vice versa?

Also, that's kind of a lot of code. Here's my rewrite of the server: http://play.golang.org/p/hKztKKQf7v

It doesn't return the exact same result, but since you're not verifying the results, it is effectively the same (4 bytes in, 4 bytes back out). I did slightly better with a hand-crafted one.

A little cleanup on the client here: http://play.golang.org/p/vRNMzBFOs5

I'm guessing scala's hiding some magic, though.

I made a small change to the way the client is working, buffering reads and writes independently (can be observed later) and I get similar numbers (dropped my local runs from ~12 to .038). This is that version: http://play.golang.org/p/8fR6-y6EBy

Now, I don't know scala, but based on the constraints of the program, these actually all do the same thing. They time how long it takes to write 4 bytes * N and read 4 bytes * N. (my version adds error checking). The go version is reporting a bit more latency going in and out of the stack for individual syscalls.

I suspect the scala version isn't even making those, as it likely doesn't need to observe the answers.

You just get more options in a lower level language.

dlsspy

13 lat temudiscuss

We've done a few of them here. Notably:

* http://cbgb.io/

* http://dustin.github.io/2012/09/09/seriesly.html

* https://github.com/couchbaselabs/sync_gateway

cbgb is an API compatible Couchbase implementation in go. We use it in place of couchbase when we need something tiny to play around with.

seriesly is a time series database for storing and aggregating sample data and doing things like this: http://bleu.west.spy.net/~dustin/seriesly/

sync_gateway is how our mobile team synchronizes data across all your phones and tablets and your central DB.

dlsspy

13 lat temudiscuss

I have no idea what my response was about. As I said, I was pretty sick that day. Sorry you had to type all this stuff to explain to me that I'm a moron. :)

dlsspy

13 lat temudiscuss

Then why isn't that the default?

dlsspy

13 lat temudiscuss

> but I think "<- true" looks nicer than "<- struct{} {}"

And what looks even better:

    defer close(doneChannel)

(it's also syntactically correct -- you ca't just have a defer block without a function invocation)

dlsspy

13 lat temudiscuss

Or turning that around, do you think it requires all 8 of my cores to copy data over the network? Do you think the two lines of code + justification text provides sufficient value for this application to distract from the point of it in order to show why someone should override the default behavior?

Do you believe users shouldn't have any control over the number of cores any particular application consumes?

Have you measured the CPU contention of the application and determined that using more cores is worth the overhead of increased overhead of multi-thread exclusions (vs. more simple things happening directly in the scheduler)?

Overall, it has nothing to do with this article and now even more people are going to copy it in more unnecessary places as a cargo-cult "turbo button" for their programs.

If you are going to use an idiom like that, the least you could do is check for the GOMAXPROCS environment variable and only do this as a default when the user hasn't specified otherwise.

dlsspy

13 lat temudiscuss

Here's the version I threw together in (what it says here), 12 minutes: https://gist.github.com/dustin/5478818

dlsspy

13 lat temudiscuss

There's so much code here.

If I weren't sick, I'd submit a new version that:

1. Didn't reimplement io.Copy

2. Didn't avoid io.TeeReader

3. Didn't do weird things to avoid regular channel ranges.

4. Didn't do non-standard date formatting.

5. Didn't reinvent the log package.

6. Didn't try to convince anyone runtime.GOMAXPROCS(runtime.NumCPU()) was a good idea (it's not)

In fact, maybe I will anyway. brb

dlsspy

13 lat temudiscuss

goroutines are not threads because thread means something and goroutine means something else.

dlsspy

14 lat temudiscuss

What else would I use? I've been writing tons of go code for nearly three years now. I get fast, concurrent, parallel code in very little time.

I was pretty much production-ready with seriesly in two weeks by myself (though I'm starting to get contributions from other users). Last night, I closed my last open issue "bulk interface" by making an optional memcached binary protocol interface with custom packets for database selection and streaming data in.

Today, I started a new project with a new guy, and got some pretty impressive internal demos working after a couple hours of work.

I get lots of things done fast and reliably. This doesn't happen as much for me in other languages. I went into more details in the follow-up post where I described how I built seriesly (and keep in mind, I wrote this after it had only been alive for two weeks): http://dustin.github.com/2012/09/13/inside-seriesly.html

dlsspy

14 lat temudiscuss

This is great feedback. You seem to get what I'm going for.

Thoughts on your specific items:

1. I could probably prioritize the query/doc processing and get most of this out of the way, or something like what I've been thinking about for #4.

2. I've thought about this one for sure. It's actually possible to do externally already, just not very magically. I'll learn more when I get more internal people pushing it.

3. I've been tempted to add replication -- not because I need it, but because it's just really easy. master-slave is completely trivial. master-master isn't hard, but requires a tiny bit of state to be tracked I don't have an easy way to do yet. It'd be worth it just for fun.

4. I have a lot of infrastructure for this. To be efficient, I need something like _all_docs that doesn't include the values and/or something like get that evaluates jsonpointer. Then you could pretty well round-robin your writes and have a front-end that does this last-step work. Harvest the range from all nodes concurrently while collating the keys. Once you find a boundary from every node, you have a fully defined chunk and can start doing reductions on it. A slightly harder, but more efficient integration is to have two-phase reduction and let the leaves do a bunch of the work while the central thing just does collation. You wouldn't be able to stream results in that scenario, though.

5. Is this as simple as disabling DELETE and PUT (where a document doesn't exist)?

dlsspy

14 lat temudiscuss

What kinds of advanced things are you wanting to accomplish?

dlsspy

14 lat temudiscuss

There are two hard things in computer science. The next blog post I'm planning is related to cache management.

dlsspy

14 lat temudiscuss

Whisper (backend for graphite), cube, ganglia... possibly more. I used to build things directly on top of rrdtool, but the schema definition can be a pain when you've got a lot of dynamic data.

dlsspy

14 lat temudiscuss

I think there are some good ones. I wrote this because I could get it running faster than I could get the data I've got adapted to existing ones. That doesn't mean they're bad as much as it means I don't understand the data I've got. :)

The way I like to think about document-oriented databases is that you store what you have when you have it, and worry about what it was later when you need to get things back out of it.

e.g. the big bag of stuff I mentioned in the blog post contains a few things I know I don't need, a few things I think I probably need, and a lot of stuff I just don't want to think about (I might need it later, maybe after some manipulation, etc...). Lob it all in.

The downside of a system like seriesly vs. a system like rrd (or any modern equivalent) is the same as the downside of any nosql database vs. a sql database. By planning up front, I can keep the size down and get more performance by incrementally computing stuff from the beginning. In the meantime, I'll just buy more disk. :)

I'm reasonably happy with the performance, though. There's a good number of visitors on the page right now and this is what they're seeing:

    2012/09/11 12:28:50 Completed query processing in 82.54ms, 6,266 keys, 1,280 chunks

That means that for that query, it scanned through 6,266 keys in the on-disk b-tree, grouped them into 1,280 separate result "rows" to be reduced and did the necessary computation to emit all of them in under a tenth of a second while lots of other queries were in flight. My "extreme" cases right now are taking under 3 seconds on over half a million keys. I consider that acceptable for two weeks of side-project.

Submissions

Hacking on the GoPro Plus API

Redis VS Memcached (slightly better bench)

Hello World in Go -- A Memcached Server

comments