FRex · HackerTrans

vor 8 Jahrendiscuss

Are you aware of the pitfall I've mentioned here?: https://news.ycombinator.com/item?id=17323068

Just making the possible passwords be 10 chars and contain chars out of a set of 62 (lowercase + uppercase + digits) gives me 0.05 sensitivity at 0.00000001 (0.000001%) FP rate.

Edit:

(This is reply to w8rbt below, here since HN has decided I cannot post anymore for next who knows how many hours.)

I'm aware of this and wouldn't treat a bloom filter as the final decision maker if there is a hit but many people seem to act like it is intended to be one and bring up its FP rate as low enough, i.e.:

1. This comment and several of its replies: https://news.ycombinator.com/item?id=17322692

2. This comment telling me false positives don't matter: https://news.ycombinator.com/item?id=17323438

3. Both of the linked bloom filters don't mention that if there is a hit you should check with https://haveibeenpwned.com/ API.

4. (Of course, this is HN after all!) my comments pointing this paradox/pitfall/whatever out being at negative point scores. I've literally lowered my score for saying "hey, 1 in 1000 FP isn't actually super accurate and almost sure hit is a good hit". And there are one or two green named users who seem to be criticizing this bloom filter with freshly made accounts, exactly because of this BS.

FRex

vor 8 Jahrendiscuss

Due to the fact how little 500 million passwords is out of the entire set of all possible passwords (even with single digit length) the 1 in 1000 false positive rate is extremely high actually.

See my other comment: https://news.ycombinator.com/item?id=17323068

FRex

vor 8 Jahrendiscuss

Does 1 in 1000 false positive mean that 1 in every 1000 passwords not on the list returns a true? If so, that's not good because good passwords vastly outnumber the bad ones so even a small false positive rate makes a positive very very unlikely to be a true one.

For example: let's assume all passwords (including all of the ones on haveibeenpwned list) are exactly 8 chars long and composed of 50 characters.

This gives a lot of headstart to this filter since the way possible passwords outnumber ones on the list is extremely more crushing in reality due to many possible lengths, unbounded length and there being 95 printable ASCII chars to use (even alnums themselves is 62 chars already).

  passlen = 8
  passchars = 50
  badpasses = 501636842
  allpasses = passchars ** passlen
  falsepos = 0.001
  print(badpasses / (badpasses + falsepos * allpasses))

The above script prints 0.012679079642336052, which means (in these extremely forgiving scenario from above where possible passwords are limited to next to nothing compared to real life) only just over one in a hundred positives is a true positive.

This apparently comes up with FP rates for cancer and AV and such (most people dont' have cancer, most exes aren't viruses, etc.).

I might be wrong, feel free to point that out. I'm not a statistician or a doctor, I just had a statistics class as part of my CS degree. :)

Edit: I've found the English terms and explanations:

https://en.wikipedia.org/wiki/Sensitivity_and_specificity

https://en.wikipedia.org/wiki/False_positive_paradox

FRex

vor 8 Jahrendiscuss

These are not any rules. This haveibeenpwned list was made of password lists that were already leaked in hacks before and its author criticizes the 'now add a number', 'now add a symbol', 'now add a capital' rules that result in 'mypassword1_A'.

FRex

vor 8 Jahrendiscuss

1. It just does return Session(), that's easy enough to find out[1].

2. It doesn't really matter and maybe it's so they are kept closer since they are modified, Session merges your call provided and its own headers (yours take precedence) and it still handles the cookies if you provide own headers. Session also has the benefit of connection pooling so it's quicker to do more than one request with it[2] (normal get, post, etc. in requests module go through request function in the end which actually constructs a Session for that single request).

3. What's wrong there? It's just a default argument. Strings aren't mutable so it avoids this pitfall[4]. Is the " quote a problem here? It's a matter of taste/style. PEP8 is silent on it[3] and just say to pick a convention and there seems to be one here. Some people (me too) also use single quotes for non-human readable strings and double quotes for human readable strings.

4. If you mean here[5] then there is a len just above it to catch the 'expected' error/missing element, just the .text part is unchecked. As for the general lack of checks - I don't put them into my GreaseMonkey or random Python one off scraping code either. Site layout is invariant of a certain version of a scraper script so if some field is missing or something like that then the website layout must have changed and the entire script probably needs to be reworked (or the field is not always present there in the first place so the script is also useless in that particular scraping case) and might as well crash (or if its used by someone they can catch the exception). Either way (crash or catch) when something you expected to surely be there is missing the results are not coming or might be wrong. That code as it is now anticipates that there might not be such an element but if there is it must have the expected field. If the site has been observed to always work like that (certain element might be missing but surely has that field when its there) then script just works like that and guards against the first expected possibility (missing element) but not the second (missing field) since if how site is laid out, how data is stored in elements, etc. changed significantly, then the script also needs changes or risks producing bad or incomplete output (arguably worse as a default than a loud failure would be, it also depends on what you're doing and what the scrape is for).

I'd assume most users and programmers would rather get an error than have script return an empty list (despite there being content up there) just because the layout changed. The only other solution (other than return a wrong result by design and hide the errors or log them somewhere where no one cares to read anyway) would be to catch such exceptions somewhere high and either pack them into a new exception that is thrown with more information (what URL, what element content was exactly, entire response text, etc.) but that's probably too much care/work for such a one off script OR throw your entirely own ones from some low place, but it's vanity then because Python exceptions point really strongly to where they were thrown and in what call stack so it's just as clear what was broken without the need to add lots of checks yourself and throw a "element X is missing field Y that should always be there" message.

[1] - https://github.com/requests/requests/blob/master/requests/se...

[2] - http://docs.python-requests.org/en/master/user/advanced/#ses...

[3] - https://www.python.org/dev/peps/pep-0008/#string-quotes

[4] - http://www.effbot.org/zone/default-values.htm

[5] - https://github.com/tducret/amazon-scraper-python/blob/master...

FRex

vor 8 Jahrendiscuss

It's spelled Ad Nauseam and you shouldn't throw a commonly used Latin term around without specifying you're actually talking about an extension[0].

[0] - https://adnauseam.io/

FRex

vor 8 Jahrendiscuss

The segmenting backfires hilariously between countries.

I've often seen shows go "It was Mr. X all along!", everyone looked at the person who said that in shock and awe and realization, there was a sound cue, screen faded to black and then the show continued with them going to kick Mr. X's teeth in, but already in a much less dramatic tone.

And then the actual break happens in another spot, where it's much less appropriate and breaks the flow completely instead of making me eagerly await the end of the break to see Mr. X get his teeth kicked in.

This is actually a trope: http://tvtropes.org/pmwiki/pmwiki.php/Main/CommercialBreakCl...

FRex

vor 8 Jahrendiscuss

It's quite normal for there to be two sets of rules. Or maybe it's about the image of not being a "weaboo shit store" (as many people consider anything in anime style).

Even YouTube that's famous for being trigger happy with demonetization and bans has stuff like "Robin Thicke - Blurred Lines (Unrated Version)" which has 3 topless girls walk around in it without covering themselves at all.

FRex

vor 8 Jahrendiscuss

Valve can state 2 + 2 = 5 tomorrow, that doesn't make it true.

They already have geoblocks in place for several reasons ranging from their own whim, to copyright to legal reasons. I can't buy certain animes[0] and I couldn't buy Skyrim for few years (my country's physical box retailer got exclusive deal), you couldn't get Hotline Miami 2 in Australia (their censor banned it), you are forced to use your own currency (or the next best thing like dollar or euro, but there is no free choice) when buying games.

They also banned Hatred back in the day for violating their ToS, until they said it didn't actually violate it (because there was a shitstorm over it). And vague ToS that anyone can break or not break are a staple in online businesses by now.

They also have no problem selling Witcher or Mass Effect games that have way more sexual content that is way more realistic and detailed. In just Witcher 1 you have like 30 sex cards, several topless women, topless bloodied vampire girls, prostitutes, very crude references to sex, straight up softcore porn scenes with full animation, but like 10 naked anime girls in Hunie Pop is too far and a big legal problem for them?

They also had no problem breaking customer protection regulations, sniffing your DNS cache and iterating what processes you have running to find cheats, fighting in Australian court over lack of return policies, taking 0 responsibility for anything they sell for years (Skyrim paid mods, Greenlight trash, asset flips, scams and stolen games, etc.) but now hosting a couple games with anime boobs is a big no-no that opens them up to big legal problems.

All this content was also explicitly stated to not be any problem until now up to reassuring developers themselves about their specific games, do Valve and Steam operate their multi-billion global businesses in a "shoot first, ask if it's legal later" way?

All this is is just deciding to not bother with "weaboo shit" (as many people consider anything in anime style) on Steam Direct anymore for whatever reason (money, image?) and knowing they can do that without backlash because it's niche enough. There is no legal (unless countries now specifically ban static naked images of anime girls but allow 3D softcore mocap scenes of realistic models) or technical reason they couldn't make it work.

[0] - https://imgur.com/a/2Fbm9Bl

FRex

vor 8 Jahrendiscuss

The Light Phone 2 features are very poor[0].

Any dumb mobile phone from mid 2000s has those that are guaranteed in [1], they also don't run a modified Android but a bespoke locked down OS, cost less (Light Phone 2 is available for $300 as preorder on IndieGoGo and MSRP seems to be $400), have a physical keyboard, some games, colorful display, email, etc.

There are even some not so bad smart phones that cost less than $200 or $300.

I'd also expect things like calculator, GPS, unit and currency converters, weather, dictionary, directions, maps, etc. to definitely be in since they don't fight for attention and are pure utility tools that one only uses when absolutely needed, not to procrastinate.

I also own an old 8 GB black Creative Zen Style 300 media player[2] from like 2010 or 2011 that is like 7x3x1 cm, cost me like 180-190 PLN (22% or 23% of that is VAT) or so at the time and it decimates Light Phone 2 with regards to almost all non-phone non-online features.

I'd consider myself as audience for a featureful utilitarian dumb phone but not for something like Light Phone.

I wish there was an ARM flip phone (I love these as long as the hinge is good and the plastic thick), with physical keyboard (I love them when they're good), dual SIM, 1 or 2 GB of RAM, few GB of storage, SD card, low res software rendered displays (main one and one on the back when flip phone is closed) that you could code yourself against almost bare metal in pure C, calling some built in APIs when you want to make a phone call, use data transfer, access storage, etc. to truly own and personalize every aspect of it.

[0] - https://www.indiegogo.com/projects/light-phone-2-design#/

[1] - https://c1.iggcdn.com/indiegogo-media-prod-cld/image/upload/...

[2] - https://imgur.com/a/QdMdHxN

FRex

vor 8 Jahrendiscuss

There was a case on YouTube where a (sort of) satirical YouTuber Reactor (a fake over the top reaction channel) used a cut up footage of GradeAUnderA hating reaction channels as an ad on GradeAUnderA's own videos to direct people to Reactor channel as part of a scheme to bait GradeAUnderA (a 2-3 million subscribed channel) into attacking his small channel (1-2 thousand subs post these antics, much more later after the scheme came to light) to "expose" him for being a bully, starting fights (despite saying how he hates them and how they ruin YouTube and attacking people who are having them) and so on.

What a quagmire, I know.

Good video summarizing it: https://www.youtube.com/watch?v=H9_6UCkxu_U

FRex

vor 8 Jahrendiscuss

There's also using ARM, Calyos stuff[0]

Some old Macs were also 'potentially' silent if you were to today remove the HDD and the optical drive, which were the only mechanical parts[1].

[0] - https://www.youtube.com/watch?v=9LauL5JxYis

[1] - https://en.wikipedia.org/wiki/Power_Mac_G4_Cube

FRex

vor 8 Jahrendiscuss

That's a really nice line of thought. I'm no linux wizard but I've used Fedora for a few years for my laptop so if you want to ever ask about something basic we can exchange contact info.

FRex

vor 8 Jahrendiscuss

I had that "creepy" moment but it was more silly than creepy to me.

I was googling a lot about a certain piece of software (config, how to set this and that up with other things to work together, etc.) and that resulted in me getting ads for it from there on.

I mean yes, the targeting 'worked' - it's a software that interested me at the time, I even own a copy already. Targeted ad money well spent..?

The surgery was successful but the patient died.

FRex

vor 8 Jahrendiscuss

This sadly sounds like a rip off of the venerable Bielefeld Conspiracy[0], just reapplied to an entire country.

I have once entertained starting a conspiracy theory saying that unix shell pipes were originally created to allow piping fortune to cowsay.

[0] - https://en.wikipedia.org/wiki/Bielefeld_Conspiracy

FRex

vor 8 Jahrendiscuss

It's even harder to take them seriously after they said this about flat Mars to Musk: https://twitter.com/FlatEarthOrg/status/935644892721762305

FRex

vor 8 Jahrendiscuss

Why not? Is Java + Swing very heavy or something? Is NetBeans in Swing or something else? I've used NetBeans for C, C++, PHP and HTML5 on my old Core 2 Duo with 2 Gigs of RAM on Fedora during my uni years and it was fine by me. It lagged some but on that machine everything lagged and it was a full on IDE (not just a code editor) with some really nice features.

FRex

vor 8 Jahrendiscuss

This is false equivalence. The electron app is ostensibly working for your benefit, just in a very inefficient way, not mining a currency using your resources to ship off to someone.

You could argue that making users pay for savings in development by their RAM, Disk, CPU, GPU and electricity costs is also bad (and even bad for the planet) but that's a less shady business strategy for saving cost than covertly mining. I have argued that and similar points with varying success at [0]. Then again someone took it upon themselves to downvote this[2] 30 minute comment of mine already and offered no counterpoint so maybe bloatware is what people want and they will go out of their way to justify it.

Back to this topic: to stop mining requires stopping being shady and turn it off, to stop burning resources with electron requires huge staff changes, technology changes, mentality changes, etc. and goes against what Joel Spolsky said about rewrite from scratch[1] (but I do feel the irony every time I see Atom or something claim they have now way more performant code by rewriting a part of the program in C++).

It's also very unfair IMO to compare pure hearted electron users (whatever you think of people who use it and the tech itself) to someone who stealthily installs malware (which borders on a crime and is way more shady and immoral).

Coincidentally I found a very neat app for music conversion/extraction from videos today, it's light, starts instantly, the GUI never lags even for thousands of files, it's total size is 40 megs and I looked into it and it's just ffmpeg dll, some other dll for other media stuff and the main GUI is in Delphi (and to add insult to injury the author is a Pole, which is ironic because I am also a Pole and I was taught Delphi in high school too - as my first language - and use Lazarus from time to time to play around with Pascal and native GUIs again).

[0] - https://news.ycombinator.com/item?id=15948290

[1] - https://www.joelonsoftware.com/2000/04/06/things-you-should-...

[2] - https://news.ycombinator.com/item?id=17055872

FRex

vor 8 Jahrendiscuss

That might backfire on an HDD and it's a very convoluted scenario (loading from disk and decompressing when hogging all of the CPU).

Many people even with a quad core i5/i7 might have a small SSD just for Windows and important stuff and any large game goes onto the HDD, the fact you'd do such a trade off and make the game size swell only makes the effect worse and user more likely to use HDD.

Linux (the kernel) is compressed by default (that's why the filename is vmlinuz, vm for virtual memory support, z for compression) and it doesn't impact the startup enough to have many/most distros take it out.

Simple compression might not bring lots of savings but it'll at least help a bit. And with BMP instead of PNG a game would just blow up in size to crazy proportions. There also is some (still patented?) lossy tech to decompress on the GPU, not the CPU: https://en.wikipedia.org/wiki/S3_Texture_Compression

Ooodle from RAD game tools (kind of a veteran gaming middleware company with some prominent game devs employed at it and prices being 'contact us') also has some really fast compressors and decompressors (but I've never used them and I don't want to ask for a testing SDK if I don't consider buying their product).

You can also trade time on your end for optimization that is then literally free to the end user, e.g. use pngopti.

Tokyo Dark (a VN-ish game from Japan) was notorious (for like 30 reasons but this one is very annoying) and loaded entire several hours long 500+ meg game off of the disk at once at start up for no reason other than it being simple to do or done by their Construct 2 set up. Just some compression or pngopti would help a lot for that, I remember cutting like 5 or 10 percent of game size using just pngopti.

John Carmack said that to make Rage's MegaTexture system assets work (and they still ended up being huge) he used some 200 GB server to optimize it for hours.

Crash Bandicoot also used some smart packers that took ages to run at the time on multi thousand dollar workstations (90s).

All in all: I mean to say that compression and size optimization might still be worthwhile, is cheap and doesn't have to imply a big cost to the end user, more often than not the cost is very asymmetric and the compressor pays way more than the decompressor (e.g. 7zip ultra takes hours to pack many gigs of files and uses 17 GB of RAM but the unpacking of such an archive takes just 10-30 minutes and is actually limited by my HDD speed).

FRex

vor 8 Jahrendiscuss

The website version of the game was 150 KB.

You'd be hard pressed to find an engine or runtime (except electron as some people are saying it actually is..) to get a game like that (literally moving boxes and text) up to that size.

Even if he used static images and ttf fonts the size is way off. Pngs are a couple to couple dozen kilos a piece. Fonts are a few megs at most each. The single biggest font 'file' I know of/used for real (except for experiments people might do with the file formats) is Noto Sans CJK ttc file and it's not a single font but a collection (and it covers all of CJK[0] which is an insane range).

Entire Minecraft is under 300 megs and that includes the launcher, the language packs, and the entire JRE that is 140 megs in itself (!).

On gamejolt there is a (very nice) small low poly game called The Very Organized Thief, it was made in Unity3D and is just 13 megs in a zip (EDIT: and 35 unpacked).

I couldn't find a low poly game in Unreal Engine 4 nor one that is under 100-200 megs (EDIT: when packed) so maybe Unreal Engine 4 has that high static cost but I'm not sure right now.

In any case: 2048 taking over 100 megs is actually crazy, especially since it's a game so simple you can rewrite it in almost any engine overnight. He/she could have done at least that much.

[0] - https://www.google.com/get/noto/help/cjk/