Many native extensions just work with TruffleRuby, I'd estimate even the majority of the popular native extensions just work.
Performance of native extensions is not always good, especially if they do a lot of back and forth between Ruby and native.
> Cool, but what about replacing the regexp with straightforward parsing code written manually?
If you take a look at the linked snippets of C code, I think it's clear it's all but straightforward. The regexps OTOH are really short and expressive.
Ruby has no access to SIMD. And writing SIMD is basically writing inline assembly, so it's really tedious and messy.
> relying on the compiler doing autovectorization to make it fast.
I can relate to that, but Regexps are a much smaller domain, and there it's clear SIMD is always a win, so if the regexp engine uses SIMD it's very unlikely it would ever stop using it (unless something faster comes up).
When did you try?
Fixnum/Bignum are gone since many years (before 1.0RC1 which was April 2018, it's all Integer now).
So I guess many years ago, and back then TruffleRuby was basically implementing Ruby features, had fairly limited compatibility and likely could not run Rails, very different from today.
There were compatibility issues with BigDecimal, but TruffleRuby now uses the C extension, hence it should be exactly the same behavior as CRuby.
> Numerical to string formatting is a bit different too.
AFAIK that was fixed years ago if you mean float formatting.
> Regexes behave differently if they worked at all.
TruffleRuby always used Joni, which is literally a translation of CRuby's Regexp engine to Java (by the JRuby team), so that is very surprising and I have a really hard time to believe it.
At least "Regexes behave differently if they worked at all" seems harsh and highly inaccurate to me.
There likely were a couple Regexp issues but the generalization seems wildly exaggerated.
Yes, some things are slower. Most of them I'd say are unexpected performance bugs. I'd think most are easy to fix once investigated, but some can be hard to fix (recent example, `File.read` is quite fast on CRuby).
Some things are expected to be slower, for instance constantly redefining (monkey-patching) methods or constants is slower on TruffleRuby, but that's typically because the program is broken and so it'd be slow on CRuby as well.
Compatibility is certainly an issue for massive codebases, OTOH I think over time TruffleRuby is getting closer and closer to CRuby behavior to the point it would be fairly rare to find a compatibility issue.
Do you have personal experience trying to run such a codebase on TruffleRuby?
Startup is likely to be worse, because it typically runs a lot of different code for not long and the JIT might not have enough time to optimize much of that. OTOH the JIT needs to learn what the program is doing, i.e., profile it in a sense, and that has a cost on interpreter speed.
Still these numbers are worse than expected, especially [object validation], so if there is a way to reproduce it'd be great if you can open an issue about it.
That second sentence seems a fair assessment.
TruffleRuby does support many native (C/C++) extensions so that's rarely an issue.
But indeed in such a large codebase it's likely to depend unexpectedly on CRuby-specific behavior. And while that can be fixed it takes some effort either in TruffleRuby (to match CRuby) or in the app (e.g. to avoid relying on `RubyVM`).
Compatibility is one, it's hard to be 100% compatible with CRuby, and large codebases tend to sometimes depend unintentionally on weird behavior or even bugs in CRuby.
Keeping up compatibility (while keeping things efficient) is a lot of work, TruffleRuby tries to reduce that by reusing as much as possible existing code, including reusing C extensions shipped with CRuby.
Actually TruffleRuby optimizes metaprogramming more than any other Ruby implementation.
Rails is not the issue, big codebases is the issue: more code = more chances to use something which is not so fast on TruffleRuby yet = more chance to become a bottleneck and degrade the overall performance.
It's the same benchmark in the blog post, `railsbench`. So TruffleRuby already speeds up some Rails apps like that one but not every Rails app/program.
(TruffleRuby 3.27x, YJIT 1.33x on railsbench)
Interesting, I did not know that just calling the original method, even when the refinement method is not used occurs such a cost on CRuby. Seems worth reporting if not already done. No such thing on TruffleRuby though, the only peak performance cost would be for megamorphic calls (rare, even more so in combination with refinements).
Here is a bit of a critical read of this blog post, from the point of view of alternative Ruby implementations:
> instead of having to ship an entire language runtime to production
Except they ship the CRuby language runtime in production.
Ruby is not a language that can run without a runtime.
> Not only did we not need Java VM-level interoperability
So they almost discard the entire idea because they don't need a specific additional feature?
> choosing either alternative Ruby implementation would have made for a difficult migration path.
So what is it?
> Stripe relies heavily on gems with native extensions
Yes that's a problem on JRuby (when there is no java extension in that gem), but TruffleRuby supports native extensions, as very clearly stated in many places.
> as you can imagine, a multi-million line Ruby codebase over time starts to depend on Ruby-the-implementation, not just Ruby-the-language.
Except all serious Ruby implementations know they need to be compatible with whatever CRuby does, not just an incomplete ISO specification of the language.
In fact alternative Ruby implementations match CRuby behavior as much as possible for compatibility, even when it seems weird or makes little sense (they report it in this case but have to match behavior anyway).
I've also found that the Computer Language Benchmarks Game benchmarks tend to be less and less representative of real workloads, due to what the benchmarks do and to the philosophy of "write it to make it fast, not idiomatic, and parallelize things that would typically not be parallelized".
Just taking a look at the benchmark code makes it clear it's very far from idiomatic Ruby code.
Interesting results, thanks for sharing, that matches my expectations.
OptCarrot is probably one of the rare cases where Ruby 3.0+JIT is 3x Ruby 2.0.
On most benchmarks, the gains are much smaller.
I'll try to run these benchmarks on TruffleRuby (and maybe JRuby) too, would be an interesting comparison.
BTW, is there any reason you used 2.1.0 and not 2.0.0? Maybe some issue with 2.0.0?
I replied to the tweet.
At least for TruffleRuby it's to improve testing, no hidden intent.
And the PR is by someone not at Oracle, and his interest is to run some Rails app on TruffleRuby, and for that he'd like to ensure all gems work fine on TruffleRuby, which seems a very legitimate thing to do, isn't it?
That's rather vague. But yes, no matter which JIT you always need some extra memory to run the JIT, and it creates a more optimized version while also needing the unoptimized version of the code, so it needs more memory.
To give one example, DB adapters work just fine.