Go vs. Swift vs. C++ microbenchmark(lionet.livejournal.com)
lionet.livejournal.com
Go vs. Swift vs. C++ microbenchmark
http://lionet.livejournal.com/137511.html
41 comments
(Re)allocating arrays/vectors seems to be pretty much the whole point, otherwise both Swift and C++ could not only preallocate (reserve), but they could use iterators and elide all allocation instead, and then you'd get something along the lines of this:
> rustc -O test.rs -o test_rs
> time ./test_rs
9999010000
0.54 real 0.53 user 0.00 sys
> time ./test_cpp
9999010000
3.17 real 2.29 user 0.86 sysI totally agree with you on this. The code for all three languages is clearly not optimal.
I wonder how much of this performance difference is C++ being very good at optimizing your code rather than the runtime being more performant. I don't have any numbers, but I feel like that's not a negligible factor.
I wonder how much of this performance difference is C++ being very good at optimizing your code rather than the runtime being more performant. I don't have any numbers, but I feel like that's not a negligible factor.
> I totally agree with you on this. The code for all three languages is clearly not optimal.
That's not really my point. My point's we have no idea what TFA is trying to do or see[0], so you may be able to say "this is stupid and nonsensical" because it is[1], but you can't say "I've got a better version" then provide something with a completely different behaviour.
[0] most likely they just threw some shit at the wall, ended up with relatively long runtimes and called that a benchmark
[1] or maybe not, maybe it's a reduction of actual workload in the system doing computation based on some sort of dynamic number of items so you can't preallocate the intermediate storage and the point's to compare reallocation overhead[2]
[2] not very likely though, considering you never need the array in the first place as you're just summing each item with the previous one, then summing every 100 item of that… anyway
That's not really my point. My point's we have no idea what TFA is trying to do or see[0], so you may be able to say "this is stupid and nonsensical" because it is[1], but you can't say "I've got a better version" then provide something with a completely different behaviour.
[0] most likely they just threw some shit at the wall, ended up with relatively long runtimes and called that a benchmark
[1] or maybe not, maybe it's a reduction of actual workload in the system doing computation based on some sort of dynamic number of items so you can't preallocate the intermediate storage and the point's to compare reallocation overhead[2]
[2] not very likely though, considering you never need the array in the first place as you're just summing each item with the previous one, then summing every 100 item of that… anyway
Could you link to the code for `test.rs`?
fn main() {
let mut sum = 0i64;
for _ in 0..200 {
let items = (0..1000000).zip(1..1000000).map(|(a, b)| a + b);
sum = items.enumerate().filter(|&(i, _)| i % 100 == 0).fold(0, |acc, (_, v)| acc + v);
}
println!("{}", sum);
}
If the C++ version didn't use vector.reserve(1000000), what justifies the reason for the Golang version to use make(,,1000000)?
The point is to sell Swift. In the C++ example, he didn't cast each integer to int64, but he manufactured a reason to do it in Go and Swift. Perhaps this means that Swift's compiler optimizes that particular case almost as good as Go's. It doesn't matter much because it's a single threaded useless benchmark. It's the kind of thing that can change with a new Go or Swift release. As you know better than I, Go slowed down a small amount when the recent GC optimizations were done. Is that a reason to use Go less? No. The GC is quite amazing (and there is HTTP2 support).
In general, why do I care if there are slight variations in the speed of a for loop? The bottleneck in any app is the database or network access. Maintenance and readability matter to me a lot more than minuscule variations in speed.
As someone who programs in Go and Swift, I can tell you that Swift has some things Go doesn't. For one, it has generics. Even still, I would likely never use it over Go on any code that runs on a server. This is due to readability and the Go standard library. I would still choose Swift over Go for iOS development. This is due to how the iOS stack is premised on object inheritance.
In general, why do I care if there are slight variations in the speed of a for loop? The bottleneck in any app is the database or network access. Maintenance and readability matter to me a lot more than minuscule variations in speed.
As someone who programs in Go and Swift, I can tell you that Swift has some things Go doesn't. For one, it has generics. Even still, I would likely never use it over Go on any code that runs on a server. This is due to readability and the Go standard library. I would still choose Swift over Go for iOS development. This is due to how the iOS stack is premised on object inheritance.
> The point is to sell Swift. In the C++ example, he didn't cast each integer to int64, but he manufactured a reason to do it in Go and Swift.
Wow, the crazy is strong with this one. The reason for the cast is that C++ follows C in implicitly converting between integer sizes but neither Swift nor Go do, remove the casts and the code doesn't compile:
Wow, the crazy is strong with this one. The reason for the cast is that C++ follows C in implicitly converting between integer sizes but neither Swift nor Go do, remove the casts and the code doesn't compile:
> swiftc -O test.swift -o test_swift
test.swift:8:18: error: cannot convert value of type 'Int' to expected argument type 'Int64'
x.append(i);
> go build test.go
# command-line-arguments
./test.go:12: cannot use i (type int) as type int64 in append
> time ./test-go-type
9999010000
./test-go-type 10.88s user 0.86s system 153% cpu 7.660 total
> time ./test-go-cast
9999010000
./test-go-cast 11.04s user 0.87s system 154% cpu 7.699 totalIgnoring the inappropriate ad hominem...
Why not type the loop variable as int64 in go? Why not just use an int, which would default to int64 on a 64 bit system? Why not just use sum := 0?
Generally benchmarks don't say much unless one of the languages perform extremely poorly. This benchmark doesn't say anything of value.
Why not type the loop variable as int64 in go? Why not just use an int, which would default to int64 on a 64 bit system? Why not just use sum := 0?
Generally benchmarks don't say much unless one of the languages perform extremely poorly. This benchmark doesn't say anything of value.
> Ignoring the inappropriate ad hominem...
Seems appropriate to me, you're trying to spin a shit "benchmark" into some sort of conspiracy theory against your pet language and for your arch-nemesis or something.
> Why not type the loop variable as int64 in go?
Have you actually read my comment? 1. why not indeed, maybe because it's kind of a pain in the ass to do? 2. and that's got no relevance anyway, it changes pretty much nothing to the runtime
> Why not just use an int, which would default to int64 on a 64 bit system?
Because leaving that stuff to random half-specified defaults is a terrible idea when you have exact-sized integers you can be certain of.
> Why not just use sum := 0?
Because if integral numbers don't default to 64 bits or more or arbitrary-size integers, you get signed overflows.
Seems appropriate to me, you're trying to spin a shit "benchmark" into some sort of conspiracy theory against your pet language and for your arch-nemesis or something.
> Why not type the loop variable as int64 in go?
Have you actually read my comment? 1. why not indeed, maybe because it's kind of a pain in the ass to do? 2. and that's got no relevance anyway, it changes pretty much nothing to the runtime
> Why not just use an int, which would default to int64 on a 64 bit system?
Because leaving that stuff to random half-specified defaults is a terrible idea when you have exact-sized integers you can be certain of.
> Why not just use sum := 0?
Because if integral numbers don't default to 64 bits or more or arbitrary-size integers, you get signed overflows.
this case added to https://github.com/golang/go/issues/14325
No, that breaks the fair comparison. Other language examples do not pre-allocate all the vectors.
It is already broken ;)
In c++ indexing vector does no range checks while in Go indexing slice checks validity of index.
And vector in gcc growths by factor of 2 while slice in Go grows by 25% after it's bigger than 1024 elements. So there is a trade-of between memory usage and number of allocations.
And... I could go on, but the point is that microbenchmarks are in general not that useful :)
In c++ indexing vector does no range checks while in Go indexing slice checks validity of index.
And vector in gcc growths by factor of 2 while slice in Go grows by 25% after it's bigger than 1024 elements. So there is a trade-of between memory usage and number of allocations.
And... I could go on, but the point is that microbenchmarks are in general not that useful :)
The benchmark should use std::vector::at() in that case. at() performs bounds checks.
http://en.cppreference.com/w/cpp/container/vector/at
http://en.cppreference.com/w/cpp/container/vector/at
As one would expect, the reallocations way dominate the costs: here's the original program
> time ./test_cpp
9999010000
./test_cpp 2.29s user 0.86s system 99% cpu 3.170 total
> time ./test_cpp_at
9999010000
./test_cpp_at 2.31s user 0.85s system 99% cpu 3.173 total
> time ./test_cpp_at
9999010000
./test_cpp_at 1.27s user 0.30s system 98% cpu 1.587 total
> time ./test_rust
9999010000
./test_rust 0.53s user 0.01s system 97% cpu 0.556 total
> time ./test_cpp
9999010000
./test_cpp 0.00s user 0.00s system 45% cpu 0.009 total
Oh microbenchmarks.
All of this is going to be optimized away by a sufficiently smart compiler or a compiler you tell you want to spend lots of time optimizing.
In general: don't write microbenchmarks where the results can be easily statically evaluated by the compiler, unless you are testing whether the compiler can statically evaluate it (this is a reasonable thing to want to know of course, but it's not a test of the language).
There are also languages which will pretty much not bother to do a lot of the the actual work, like haskell.
(Note: In this case, the the C++ compiler being used actually knows all the recurrences involved, but decides it's too expensive to statically calculate the final result of sum. GCC would do the same.)
All of this is going to be optimized away by a sufficiently smart compiler or a compiler you tell you want to spend lots of time optimizing.
In general: don't write microbenchmarks where the results can be easily statically evaluated by the compiler, unless you are testing whether the compiler can statically evaluate it (this is a reasonable thing to want to know of course, but it's not a test of the language).
There are also languages which will pretty much not bother to do a lot of the the actual work, like haskell.
(Note: In this case, the the C++ compiler being used actually knows all the recurrences involved, but decides it's too expensive to statically calculate the final result of sum. GCC would do the same.)
Doesn't the std::vector calls mess things up? Now the compiler has to track those as well, they involve the allocator which calls into the OS.
So for the compiler to be able to statically compute the result it would need to know that the std::vector stuff doesn't have side effects.
Of course, the compiler could have special knowledge of std::vector, like for example in general it has about memcpy.
So for the compiler to be able to statically compute the result it would need to know that the std::vector stuff doesn't have side effects.
Of course, the compiler could have special knowledge of std::vector, like for example in general it has about memcpy.
"So for the compiler to be able to statically compute the result it would need to know that the std::vector stuff doesn't have side effects.
"
It can know this the same way it knows about memcpy.
Most of them have library call info for things that have standardized guarantees.
Even if it didn't, it knows what malloc does, it knows this variable never escapes, and ...
It can know this the same way it knows about memcpy.
Most of them have library call info for things that have standardized guarantees.
Even if it didn't, it knows what malloc does, it knows this variable never escapes, and ...
Not only that, usually all the ones applied via gut feeling instead of a profiler are meaninglessness.
Additionally, depending on the application domain, it might not even matter.
Additionally, depending on the application domain, it might not even matter.
As far as micro-benchmarks go, this one is particularly egregious. It is likely completely dominated by the array reallocation. So a runtime that - say - chooses to preallocate an array at size 1000000, or to go to 1000000 at first resize would win at this benchmark. More realistically, a runtime that conserves memory by reallocating arrays with a 1.5x factor vs a 2x factor will appear slower.
It's exactly because you don't want to rely on the runtime making the right call for your use-case that you preallocate arrays where it makes a difference (i.e. where you're appending a million items).
And 99% of the time you're doing less than 100 items, and so the array reallocation behavior just doesn't matter.
It's exactly because you don't want to rely on the runtime making the right call for your use-case that you preallocate arrays where it makes a difference (i.e. where you're appending a million items).
And 99% of the time you're doing less than 100 items, and so the array reallocation behavior just doesn't matter.
Concurrency model will be a big focus for Swift 4.
(Swift is currently v2.2, the focus of v3 is to stabilize the ABI.)
(Swift is currently v2.2, the focus of v3 is to stabilize the ABI.)
So what's the conclusion of this besides some useless numbers?
> "But it adds some checks and GC, so Swift also exhibits a serious slowdown, though well within 2x of C++."
Swift does not employ GC, it uses ARC. While that may be a form of GC, it doesn't suffer from the same "slowdown" that GC would in this context because ARC is doing exactly what your base case (C++) does at run-time (ARC is a compile-time tool).
I think if you're going to be doing benchmarks, you should have basic knowledge on how each system works first.
Swift does not employ GC, it uses ARC. While that may be a form of GC, it doesn't suffer from the same "slowdown" that GC would in this context because ARC is doing exactly what your base case (C++) does at run-time (ARC is a compile-time tool).
I think if you're going to be doing benchmarks, you should have basic knowledge on how each system works first.
I really think this is mostly just testing how well the list/vector/array class is handling being asked to expand itself by one element a million times.
What's the point, seriously?
FYI, unary operators and C-style for loops have been depreciated in Swift.
mangahero(2)
Where is Rust :)
In academia.
This is totally unfair. We're looking at using Rust for production workloads and cross system development.
And Haskell, in the comments.
I'd love to see Erlang here. Anyway Swift performance is impressive.
By replacing the Go slice declarations (var x[]int64) with (x := make([]int64, 0, 1000000)) you'll get an algorithm that in my iMac runs much faster:
- your version: 4.93s user 0.28s system 165% cpu 3.151 total
- my version: 1.93s user 0.04s system 168% cpu 1.165 total
EDIT: I'm sure this happens with all the examples, and that's exactly my point. What is this exactly microbenchmarking? Bad programs?