HackerTrans
TopNewTrendsCommentsPastAskShowJobs

hyc_symas

758 karmajoined 13 years ago

comments

hyc_symas
·7 days ago·discuss
The txnID was added to the page header to enable support for incremental backup. As a consequence, it's sufficient to compare a page's txnID to the current txnID to know if it's dirty or not, and spilled pages don't need a cleanup pass to clear their dirty bit on commit so commits of large txns are faster now.
hyc_symas
·7 days ago·discuss
write/fsync can be faster in a large dataset because writes let the filesystem know an explicit list of dirty pages, so fsync only needs to deal with them.

mmap/msync gives no hints about which pages are dirty (unless the app tracks them itself and msyncs them individually, which would completely defeat any reduced syscall advantage of using a writable mmap in the first place) so the entire map must be scanned for dirty pages.

In practice, the expected performance advantages of using a writable mmap just aren't there, and coupled with the ease of silent corruption, it's best to never use that approach.
hyc_symas
·8 days ago·discuss
By the way, you're wrong on both points - the cache of page mallocs is not infinite, and it does spill dirty pages to disk when necessary. And the latter is what bounds the number of malloc'd pages.
hyc_symas
·9 days ago·discuss
You're dreaming. None of your explicit memory control operations mean anything in practice, because today everything runs in VMs with no actual control of the underlying hardware. Probably co-resident with an unknown number of other tenants.

As for what you claim the paper's authors were saying - I quoted their text verbatim. Your interpretation is not what they said.

They claimed using mmap safely is impossible, and using it correctly requires more complexity than a traditional DB design. The safety claim was already disproven by multiple researchers. To prove their second claim they would have had to produce a DB that did traditional buffer management and was simpler and more performant than using mmap. They never did any such thing, nor could they.
hyc_symas
·9 days ago·discuss
In the time it takes for your query optimizer to dissect a query and "look ahead" LMDB would have already answered a million queries. You think your magical "future oracle" is zero cost? How many KLOCs is it? LMDB's hot paths fit entirely inside a CPU's L1 cache.
hyc_symas
·9 days ago·discuss
Indeed, there's no need to use the doc website. There's nothing there that isn't embedded in the LMDB source code. All of the docs are generated from doxygen comments in the source.
hyc_symas
·9 days ago·discuss
If you're downloading binaries from a plaintext documentation site, I think that's on you.
hyc_symas
·9 days ago·discuss
Nonsense. The best you will ever do, even with full application knowledge and complete control of the machine, is an LRU cache replacement algorithm. But when you do it yourself you have to juggle the fine details of which indices to prioritize, and you will never get it perfect. If you're not running a dedicated machine, as soon as any other processes run all your careful tuning goes out the window.

Since LMDB manages multiple tables as a tree of trees, no fine tuning is needed. The internal paths to every hot page automatically take priority, regardless of which index or how large each index is. So a simpleminded LRU always makes optimal use of available cache, regardless of access pattern or other load on the system.
hyc_symas
·9 days ago·discuss
Obligatory "that paper is garbage" https://www.symas.com/post/are-you-sure-you-want-to-use-mmap...
hyc_symas
·9 days ago·discuss
Irrelevant, since LMDB doesn't dirty the pages in the mmap. The mmap is read-only.
hyc_symas
·9 days ago·discuss
LMDB 1.0 no longer uses a P_DIRTY flag, it no longer has to explicitly mark pages as clean.
hyc_symas
·2 months ago·discuss
> The price is intended to decrease.

No. Monero's tail emission rate was specifically chosen to be less than the rate of global gold production. Do you claim the price of gold is intended to decrease?

A continuous emission like Monero's doesn't equate to inflation/devaluation. It allows its userbase to expand organically, without artificial scarcity pumping its price.
hyc_symas
·2 months ago·discuss
I don't know what the current versions do, it's been a while since I touched that code.

I have no reason to lie, I'm not selling anything. Bitmain is selling mining hardware, take a look at their claims. They've had 7 years to try to crack it.
hyc_symas
·2 months ago·discuss
FPGAs have no particular advantage. You could dedicate a chunk of their resources to implement a softcore CPU but it'd be several times slower than a real CPU.

The random programs change too quickly to just implement them directly on an FPGA. Reprogramming the entire chip like that takes too much time.
hyc_symas
·2 months ago·discuss
> Loved your 2019 talk on "is XMR still ASIC-proof" – is it still, in 2026, in your opinion?

Yep. Nothing about computing architecture has changed.
hyc_symas
·2 months ago·discuss
>> just killing the process will never corrupt the blockchain DB

> I would love to show you how easy this is to reproduce, even on fresh installs of Ubuntu and/or MacOS on otherwise-stable hardware (never tried Windows... easier?).

If it's so easy to reproduce, you should be able to screen record a session with two terminal windows:

1 with monerod running and syncing the blockchain

2 send a `kill -9` to the monerod

1b restart monerod

And then we should see the error message you're referring to.
hyc_symas
·2 months ago·discuss
> If you don't sync then you're not (cannot be) a fullnode / network verifyer / ringsigner.

I was talking about database sync, not blockchain sync. You don't need to use safe sync mode if you don't have to worry about machine crashes. And just killing the process will never corrupt the blockchain DB.

> This may be old behavior... I go way back

On this particular point, I go way back further than you.
hyc_symas
·2 months ago·discuss
Light mode is used mostly by the monerod, validating incoming blocks. But on a machine with sufficient free RAM, monerod can also be set to use Fast mode instead.

The post described both modes. The only difference is that Fast mode processes the cache to generate the full 2.1GB dataset, so subsequent programs can just reference it as needed. Light mode uses only the 256MB cache and generates the required dataset values individually, on each access. That saves RAM but costs more CPU time.
hyc_symas
·2 months ago·discuss
The limit we set at the beginning was "no one can design a custom device for RandomX with more than a 2:1 efficiency advantage over general purpose CPUs". That is and will forever remain true.

In reality, no one has been able to build any device for RandomX that isn't actually a CPU. The closest thing to a "mining ASIC" is just a bunch of RISC-V cores.
hyc_symas
·2 months ago·discuss
Since the programs are randomly generated, there's no guarantee that any particular program always uses some number of any particular instruction. There's only a probability, X/256 instructions will be somesuch operation.