cullenking

20 日前議論

I just implemented bot and crawler detection as well as ASN based blocking for our website, because I’ve seen a massive rise in scraping coming from VPNs and other networks that mix legit and illegitimate traffic to our service. My theory is that small companies are scraping the shit out of everything and selling results to llm creators. It’s going to be interesting to see this expand into residential internet providers through holes like this… wild new world!

cullenking

2 か月前議論

The people I hired in my last round, with over 600 slop and fake applicants, had honest and informal cover letters that stood out. I’m sure I passed up on real decent people as a result, but there’s no perfect way to avoid this right now.

It helps that we have something closer to a lifestyle business, where I can ask for a brief paragraph about your relationship to the outdoors and cycling, but that just means I had 500 slop cover letters gushing about cycling. The three that made it through were short concise honest and linked to real world activities they did.

Good luck, it’s a hard problem , and very very adversarial. You have true scam level applications from North Korea and India, and you have unqualified people trying to appear qualified. Sprinkled in are unqualified people who would be a good hire because of raw capability, and qualified people who are looking to do bare minimum.

cullenking

2 か月前議論

Anecdata to add to the pile...I pulled three 1u epyc gen2 servers from my production rack 1.5 years ago and replaced them with lower power alternatives for a production storage cluster. I didn't need the extra CPUs for app server stuff so they sat in my house for a while. Fast forward 1.5 years and it was making sense to upgrade some app servers to new gen stuff, get a bump in frequency and core count...when i went to spec some new servers, my normal $15k - $20k build was $55k.

Instead, I hittup ebay, got six used gen3 processors, found a "good deal" on a couple tb of new ram (still insanely expensive), and came out with the same overall horsepower for a total of $20k instead of $110k.

I know this is about consumer desktop, but seeing the comments about upgrading old hardware caused me to chime in. This is happening in the production/enterprise level in some segments.

cullenking

4 か月前議論

We started building out a set of spam/fraud/bot management tooling. If you have any decent infrastructure in place already, this is a pretty manageable task with a mismash of techniques. ASN based blocking (ip lookup databases can be self hosted and contain ASN) for the obvious ones like alibaba etc, subnet blocking for the less obvious (see pattern, block subnet, alleviates but doesn't solve problems).

If you have a logging stack, you can easily find crawler/bot patterns, then flag candidate IP subnets for blocking.

It's definitely whackamole though. We are experimenting with blocking based on risk databases, which run between $2k and $10k a year depending on provider. These map IP ranges to booleans like is_vpn, is_tor, etc, and also contain ASN information. Slightly suspicious crawling behavior or keyword flagging combined with a hit in that DB, and you have a high confidence block.

All this stuff is now easy to homeroll with claude. Before it would have been a major PITA.

cullenking

5 か月前議論

Doesn’t meet all our product requirements unfortunately. We used returned hexes in certain queries, and we also hacked in directionality of line using least significant 12 bits of the hex (didn’t need that level of hex precision), and we are doing direction oriented matching and counting. For simpler use cases it’s definitely a better option. thanks for reminding me and other people reading my comment!

cullenking

5 か月前議論

Not any differently than another indexed text field

cullenking

5 か月前議論

We do something similar for some limited geospatial search using elastic search. We make a set of h3 indexes for each of the hundreds of millions of gps recordings on our service, and store them in elastic search. Geospatial queries become full text search queries, where a point is on the line if the set of h3 indexes contains the point. You can do queries on how many cells overlap, which lets you match geospatial tracks on the same paths, and with ES coverage queries, you can tune how much overlap you want.

Instead of using integers IDs for the hexes, we created an encoded version of the ID that has the property that removing a character gets you the containing parent of the cell. This means we can do basic containment queries by querying with a low resolution hex (short string) as a prefix query. If a gps track goes through this larger parent cell, the track will have hexes with the same prefix. You don’t get perfect control of distances because hexes have varying diameters (or rather the approximation, since they aren’t circles they are hexes), but in practice and at scale for a product that doesn’t require high precision, it’s very effective.

I think at the end of this year we’ll have about 6tb of these hex sets in a four node 8 process ES cluster. Performance is pretty good. Also acts as our full text search. Half the time we want a geo search we also want keyword / filtering / etc on the metadata of these trips.

Pretty fun system to build, and the concept works with a wide variety of data stores. Felt like a total hack job but it has stood the test of time.

Thanks uber, h3 is a great library!

cullenking

5 か月前議論

I do this all the time in a dumb but effective way. Add logging statements to code paths that drop timing info. Another dumb but effective way, instead of using a step through debugger, is drop "here, value is {val}". Telling claude to do this is trivial, it's quick, and it can read its own output and self-solve the problem all with just the code itself.

IMHO git bisect is slower, especially depending on the reload/hot-reload/compile/whatever process your actual app is using.

cullenking

5 か月前議論

The above works on wayland, had to make the changes specifically when I moved over to wayland and hyprland

cullenking

5 か月前議論

Flameshot is the best! I've been using it for 10+ years. I have it wired up to some hot keys in my window manager, and have it dump to s3 so I can paste around links to screenshots everywhere for work.

https://github.com/kingcu/screendrop

cullenking

8 か月前議論

So like texting and driving, but in the air? Flying is hard, I don’t think an automated text based system would be safer than what we have now.

cullenking

9 か月前議論

it all depends on your philosophy on dependencies. if you maintain a small set of core dependencies that are there for good reasons and are actively maintained, then rails upgrades are pretty easy. if you have a Gemfile that has a bunch of third party gems that you bring in for small problems here and there, you have to occasionally pay down that debt on version upgrades. we have an 18 year old rails codebase currently on 7.1 that hasn't proven to be a big pain for upgrades. the hardest upgrade we did was because of a core dependency that had been dead for 5 years broke with a new version of rails. but that was a story of letting technical debt ride for too long and having to pay it back.

this is a common problem in any complex codebase that has a culture of using third party dependencies to solve small problems. you see this conversation all the time with modern frontend development and the resulting dependency tree you get with npm etc....

cullenking

10 か月前議論

Minidsp flex ht or htx paired with a buckeye 6 channel amp. As cheap as you can get premium sound quality. Not cheap but you get the software control you actually want via the minidsp

cullenking

10 か月前議論

Preschool is just daycare with structure, so it costs more. Optional, privately owned. Nice to do 2-3 days a week for young kids to give them more social and learning opportunites. But it’s not public school, it’s usually just a small locally owned business.

cullenking

10 か月前議論

We've been running a production ceph cluster for 11 years now, with only one full scheduled downtime for a major upgrade in all those years, across three different hardware generations. I wouldn't call it easy, but I also wouldn't call it hard. I used to run it with SSDs for radosgw indexes as well as a fast pool for some VMs, and harddrives for bulk object storage. Since i was only running 5 nodes with 10 drives each, I was tired of occasional iop issues under heavy recovery so on the last upgrade I just migrated to 100% nvme drives. To mitigate the price I just bought used enterprise micron drives off ebay whenever I saw a good deal popup. Haven't had any performance issues since then no matter what we've tossed at it. I'd recommend it, though I don't have experience with the other options. On paper I think it's still the best option. Stay away from CephFS though, performance is truly atrocious and you'll footgun yourself for any use in production.

cullenking

8 年前議論

Turns out that fat/lazy also applies to Linux. Meaning I used to also try to optimize the heck out of my linux install. Went back to Linux from OSX last year, and was perfectly fine adjusting my workflow to suit my new machine, rather than trying to make my machine suit my ideal workflow.

What this means is that I installed a base ubuntu system with awesomewm and just used it mostly stock. I already had awesomewm experience from 5+ years prior, but for the first month of moving back to Linux I was also perfectly happy chugging along with gnome.

What I am saying is give it a try. Only time I miss OSX now is when I need to sign a PDF.

cullenking

16 年前議論

No, it's not open to government monitoring, unless of course they have the ability to break "unbreakable" encryption. And, if they do manage to magically brute force a single key, they only get a single message decrypted. Each message uses a different key, so compromising a single message doesn't compromise the whole conversation.

For a school project I (sorta) implemented this on Android 1.5 two years ago. I stopped before finishing the key rotation stuff, so ended up with just a encrypted text messaging client. Another company just recently announced an OTR text message program for android, you should look it up! Textsecure by Whisper Systems.

Clickable link: http://www.whispersys.com/

コメント