SaveTheRbtz

3 anni fadiscuss

"An In-Depth look at Lerp, Smoothstep, and Shaping Functions"[1] by SimonDev is the best 8min introduction to Lerp for a software engineer that I saw.

[1] https://www.youtube.com/watch?v=YJB1QnEmlTs

SaveTheRbtz

3 anni fadiscuss

We're mostly on the Grafana stack, so we tend to favor their open-source solutions for metric backends. For client libraries, we're currently using Prometheus for metrics and OpenTelemetry (OTEL) for tracing, but we're looking to migrate fully to OTEL once metrics are stable there. As for logging, it depends on the project, but generally we use either Zerolog or Zap, with either Loki Docker plugin or Promtail.

SaveTheRbtz

4 anni fadiscuss

Thanks for creating this proposal! Would love to see it committed upstream. For me it worked just fine -- I'll need to play a bit more with it and maybe put it into a real-world production setup to see how well it preforms on real-world data.

SaveTheRbtz

4 anni fadiscuss

I did PoC experiments with compression, chunking, and IPFS here: https://github.com/SaveTheRbtz/bazel-cache

If you need a mature compression implementation for bazel I would recommend using recent bazel versions w/ gRPC-based bazel-remote: https://github.com/buchgr/bazel-remote

bazel nowadays supports end-to-end compression w/ `--experimental_remote_cache_compression`: https://github.com/bazelbuild/bazel/pull/14041

SaveTheRbtz

4 anni fadiscuss

I was recently playing[0] with the ZSTD seekable format[1]: a *valid* ZSTD stream format (utilizing ZSTD skippable frames, that are ignored by de-compressor) w/ an additional index at the end of the file that allows for random access to the underlying compressed data based on uncompressed offsets. This combined w/ a Content-Defined-Chunking[2] and a cryptographic hash[3] in the index allows for a very efficient content-addressable storage. For example I've successfully applied it to a bazel-cache, which gave me between 10x and 100x wins on repository size w/ negligible CPU usage increase.

[0] https://github.com/SaveTheRbtz/zstd-seekable-format-go

[1] https://github.com/facebook/zstd/blob/dev/contrib/seekable_f...

[2] e.g FastCDC https://www.usenix.org/system/files/conference/atc16/atc16-p...

[3] https://github.com/facebook/zstd/pull/2737

SaveTheRbtz

5 anni fadiscuss

The analysis itself is quite impressive: a very systematic top-down approach. We need more people doing stuff like this!

But! Be careful applying tunables from the article "as-is"[1]: some of them would destroy TCP performance:

  net.ipv4.tcp_sack=0
  net.ipv4.tcp_dsack=0
  net.ipv4.tcp_timestamps=0
  net.ipv4.tcp_moderate_rcvbuf=0
  net.ipv4.tcp_congestion_control=reno
  net.core.default_qdisc=noqueue

Not to mention that `gro off` that will bump CPU usage by ~10-20% on most real world workload, Security Team would be really against turning off mitigations, and usage of `-march=native` will cause a lot of core dumps in heterogenous production environments.

[1] This is usually the case with single purpose micro-benchmarks: most of the tunables have side effects that may not be captured by a single workflow. Always verify how the "tunings" you found on the internet behave in your environment.

SaveTheRbtz

5 anni fadiscuss

re. syncookies: Linux by default starts issuing syncookies when listening socket's backlog overflows, so it may be accidentally triggered even by a small connection spike. (This, of course, is not an excuse for a service misconfiguration but it is quite common: somaxconn on Linux before 5.4 used to be 128 and many services use the default.)

re: pacing: Awesome!! I would guess it is similar to Linux "internal implementation for pacing"[1]. Looking forward to it eventually graduating form being experimental! As a datapoint: enabling pacing on our Edge hosts (circa 2017) resulted in ~17% reduction in packet loss (w/ CUBIC) and even fully eliminated queue drops on our shallow-buffered routers. There were a couple of roadbumps (e.g. "tcp: do not pace pure ack packets"[2]) but Eric Dumazet fixed all of them very quickly.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

SaveTheRbtz

5 anni fadiscuss

> tcpdump ... linux TCP stack ... ebpf

> Networks are not my specialty

I wish all network non-specialist were like you!

SaveTheRbtz

5 anni fadiscuss

Thanks! Added it back to the blogpost (seems like we accidentally lost the `pktmon` reference during the editing process =)

SaveTheRbtz

5 anni fadiscuss

A couple of questions:

* What are the reasons for disabling TCP timestamps by default? (If you can answer) will they be eventually enabled by default? (The reason I'm asking is that Linux uses TS field as storage for syncookies, and without it will drop WScale and SACK options greatly degrading Windows TCP perf in case of a synflood.[1])

* I've noticed "Pacing Profile : off" in the `netsh interface tcp show global` output. Is that the same as tcp pacing in fq qdisc[2]? (If you can answer) will it be eventually enabled by default?

[1] https://elixir.bootlin.com/linux/v5.13-rc2/source/net/ipv4/s... [2] https://man7.org/linux/man-pages/man8/tc-fq.8.html

SaveTheRbtz

5 anni fadiscuss

Sadly, middleboxes are a real problem, esp. with our Enterprise customers. We had this problem even with HTTP/2 rollout so there is even a special HTTP/1.1-only mode in the Desktop Client for environments where h2 is disabled.

In the future we are planing on having an HTTP/3 support which will give us pretty much the same benefits as SCTP with a better middlebox compatibility.

SaveTheRbtz

5 anni fadiscuss

Oh, Windows 7? That does explain it. Windows TCP stack really improved in 8.1. The main change there is the auto send buffer tuning which allows automatic growing of SNDBUF. Let me see if we can put a dirty hack^W^Wworkaround for Windows < WIN2012R2SERVER that would unconditionally set SO_SNDBUF to something like 1Mb.

SaveTheRbtz

5 anni fadiscuss

We didn't mention RSS/RPS in the post mostly because they are stable. (Albeit, relatively ineffective in terms of L2 cache misses.) FlowDirector, OTOH, breaks that stability and causes a lot of migrations, and hence a lot of re-ordering.

Anyways, nice reference for TAPS! Fo those wanting to dig into it a bit more, consider reading an introductory paper (before a myriad of RFC drafts from the "TAPS Working Group"): https://arxiv.org/pdf/2102.11035.pdf

PS. We went through most of our low-level web-server optimization for the Edge Network in an old blogpost: https://dropbox.tech/infrastructure/optimizing-web-servers-f...

SaveTheRbtz

5 anni fadiscuss

Microsoft Devs explain this in their "Algorithmic improvements boost TCP performance on the Internet"[1] article.

TL;DR is that they had RACK (RFC draft) implemented as an MVP but w/o the reordering heuristic.

[1] https://techcommunity.microsoft.com/t5/networking-blog/algor...

SaveTheRbtz

5 anni fadiscuss

Interesting, can you try disabling upload limiter in settings? Also what is your RTT to `nsf-1.dropbox.com`?

PS. One known problem that we have right now is that we use a multiplexed HTTP/2 connection, therefore:

1) We rely on the host's TCP congestion. (We have not yet switched to HTTP/3 w/ BBR.)

2) We currently use a single TCP connection: it is more fair to the other traffic on the link but can become bottleneck on large RTTs.

SaveTheRbtz

6 anni fadiscuss

Alexey from Traffic Team is here. Traffic is definitely a non-negligible part of the budget. We try to reduce it as much as possible for both lower operational expenses and better user experience. Main drivers for that improvement (besides owning our own Edge infrastructure) on the client side are:

1) Brotli (Broccoli) compression.

2) Differential updates through librsync.

3) "LANSync" a P2P sync within a broadcast domain (secured through server issued short lived TLS certs.)

That said, Desktop Client is only 1/3 of the overall Dropbox traffic -- the rest 2/3 are split between Web and API.

SaveTheRbtz

6 anni fadiscuss

We've made a note about how inefficient our solution was and what was the plan to fix it. Sadly, to get proper stats in nginx we needed two things:

* C interface for stats, so we can would have access to from C code.

* Instrument all `ngx_log_error` calls so we would have access not only to per-request stats but also various internal error conditions (w/o parsing logs.)

That said, we could indeed just improve our current stat collection in the short term (e.g. like you suggested with a per-worker collection and periodic lua_shared_dict sync.) But that would not solve the longterm problem of lacking internal stats. We could even go further and pour all the resources that were used for Envoy migration into nginx customizations but that would be a road with no clear destination because we would unlikely to succeed in upstreaming any of that work.

SaveTheRbtz

6 anni fadiscuss

One of the senior engineers once said to me that "Bazel is like a sewer: you get back what you put in."

Bazel requires a lot of upfront effort but the power of (a programmatically accessible/modifiable) dependency graph and a common build/test system across all the languages is very hard to underestimate.

SaveTheRbtz

6 anni fadiscuss

It is easy enough for simple cases (and we used it for quite a while, until we moved to using Lua for that.) For more complex scenarios you will have new `server` blocks, certificates, tls tickets, log files / syslog endpoints, so the automation will end up interacting not with just a single dynamic upstream file but with rather large amount of system interfaces. Control-plane ends up being distributed between config generation, filesystem state, service configuration (e.g. syslog.)

On a more practical note, each nginx `reload` will double the number of workers, almost doubling memory consumption and significantly increasing CPU usage (need to re-establish all TCP connections, re-do TLS handshake, etc.) So there is only that many reloads that you can do in an hour.

SaveTheRbtz

Submissions

Migrating Dropbox from Nginx to Envoy

Evaluating BBRv2 on the Dropbox Edge Network

Datadog Log Management from Zero to One

comments