jmt_

no profile record

comments

4 years agodiscuss

Hey Simon, thanks for creating Django with Adrian. I was deeply interested in programming from a young age but learning Django in my teens sparked a passion for web development that has yet to feign so many years later! Appreciate all your contributions to this space.

4 years agodiscuss

Please tell me you're legally allowed to talk more about Megaupload and the work you did - sounds like an absolutely amazing blog post, would love to hear as much as you're able to discuss.

Also, I have a project in production at work where a device needs to grab its public IP address. My code has a list of sites that provide that info and I have ip4.me as a fallback in that list, so thank you for building it!

4 years agodiscuss

It's kind of amazing anyone chooses to go into healthcare having to work like this. It's the absolute last field I would ever want to go into, even as an engineer who wouldn't need to actually practice medicine. Seems like you need to practically give up your life to save countless others. Your wife, and those like her, are truly performing an innately critical job at an absurd cost to themselves - God bless.

4 years agodiscuss

Interesting idea leveraging a cheap Android. I wonder how difficult it would be to modify an instance of a regular headless browser in order to convince a website you're using an Android browser. Not sure if Androids just come with mobile Chrome these days or if OEM/carrier-developed type stock browsers still get shipped.

Also totally right on the IP reputation point. I saw a post on HN in the last few months of someone describing how they used a cheap mobile data plan + USB LTE modem to proxy their web scraping. I believe you get effectively treated as a residential IP (depends on the complexity of the system - if they're simply blacklisting datacenter IPs then this should work) with the additional benefit of being able to change the IP assigned to the modem easily.

4 years agodiscuss

Yeah, Chrome is the worst choice for this use-case - see my last comment on this thread for more on that. Can you speak a bit more on what you'd like to use a headless anti-detect browser for over regular headless browsers? Is it to leverage their built-in fingerprinting control, effectively avoiding anti-bot measure with little effort, or management of multiple "profiles", etc? My system effectively comes down to using webkit, and storing credentials (encrypted w/ symmetric key) as well as whatever information is needed by Playwright to reconstruct the session. Simply using webkit + DB effectively achieves a headless anti-detect browser, but you're right that webkit alone isn't always a one-and-done solution.

4 years agodiscuss

> I think using anti-fingerprinting is itself a fingerprint. I imagine it would be easier to hide in the noise of regular browsers.

That's what I thought originally too. The problem is the "leaky-ness" of Chrome and Firefox - they expose a large amount of information that can be easily used to train various ML classifiers. Chrome's DevTool Protocol is most commonly used when headless access to Chrome is desired and is inherently "leaky", by design as a protocol for debugging. Don't even try to use any flavor of headless Chrome, even with stealth plugins. Firefox isn't much better.

Webkit doesn't seem to expose as much information, and having a much lesser percentage of usage, I think there's simply less information to feed into a classifier to learn to detect it reliably. There's a few sites that offer fingerprint testing such as:

- https://amiunique.org/fp

- https://webscraping.pro/wp-content/uploads/2021/02/testresul...

Try writing a script that goes to a page like this and have it take a screenshot, using Chrome, Firefox, and then Webkit to see the difference yourself. I use the Python port of Playwright personally. In the project I mentioned in my last comment, all I had to do was change the browser Playwright was using to webkit - i.e "browser = p.webkit.launch()" where "p" is a sync_playwright context manager instance. I tried Chrome and Firefox with many, many, attempts at stealth modifications and none worked. Removing my "stealth code" for the other browsers and changing it to webkit was all that was needed. Blew me away that it was that simple honestly. I've used this trick on other websites and have noticed webkit just gets processed differently by captchas/anti-bot, etc. Selenium should also offer support for a WebKit driver if you prefer it over Playwright.

4 years agodiscuss

How would you actually use an anti-detect browser programmatically? Would you need to write a custom Selenium driver for it or equivalent for Playwright? Even if the browser is built off something like Chrome, you'd still need a way to interact with the anti-detect related features.

A good trick I discovered is using webkit thru Playwright to bypass fingerprinting and related anti-bot measures. Firefox/Chrome simply leaks too much information, even with various "stealth" modifications. e.g: have been able to reliably scrape a well known companies site that implemented a "state of the art, AI-powered, behavioral analysis, etc" anti-bot product. Using Chrome/Firefox + stealth measures in Playwright did not work - simply switching to Webkit with no further modifications did the trick.

Not exactly what you're asking, but my point is, that with a little time and effort, I've usually been able to find fairly simple holes in most anti-bot measures -- it probably wouldn't be terribly hard (especially since you're versed in scraping) to build-out something similar to what you're looking to achieve without having to pay for sketchy anti-detect browsers.

4 years agodiscuss

Don't remember where I saw this but: "The act of purchasing physical books and the act of reading them are two entirely different hobbies and practices".

Additionally, if someone downloads your book for free, and barely reads it, as you suggest, have they _really_ done much more than picking up the book in person, scanning the table of contents, reading some pages in particular, then setting the book back down? What you're describing sounds like the digital version of what I used to do at physical bookstores; it seems a less compelling argument against what the z-lib founders were doing, not more.

4 years agodiscuss

I've built all my web backends in either Flask or Django - what are the selling points/advantages of using Rust vs. Python in this context? Certainly not arguing against it but am very curious and unfamiliar with Rust. If anyone has moved from Python to Rust for this kind of work, can you speak on your experiences with doing so?

4 years agodiscuss

Trash language is a bit harsh. I'm not sure I would try to put an R project into production or build a huge project with it but, at the very least, R/R Studio was the best scientific calculator I've ever used. Was particularly great during college

4 years agodiscuss

I just want to be able to choose which I want on a day-by-day basis. Some days I am much more productive in the office because I need to, say, communicate with a bunch of people or there's some sort of back and forth that needs to occur. Or I just need to get out of the house for a bit. Other days, I just want to stay home and spend the time focusing on something in particular without being bothered. But I need both to have both to do my job effectively. However, I work at a small company and am the primary software engineer which changes the dynamic a bit.

4 years agodiscuss

I can see why you're saying that, but the use of commutative diagrams communicate the structure between the functions and objects of interest - it is this perspective which is core idea to category theory. So I'd argue that the diagrams are a result of communicating a category-theoretic model rather than the end-result itself, and therefore have much deeper meaning than just "boxes being connected by arrows".

4 years agodiscuss

I see people do that all the time - inadvertently click an ad because it's one of the first few results that pop up. Not only that, the number of ads shown before the real result has increased too! Just the other day, my boss did a Google search for a common product and was shown at least FIVE ads before the first real result and had to scroll to see that result. I remember the days when you would see one or two ads and the real result as the first thing you saw after a search, not seeing only ads until you scroll down.

4 years agodiscuss

In the exact same boat. Strongly prefer writing notes for many reasons but no easy search really sucks. I currently use Notion mainly as a personal wiki and to-do tracker. Each day gets a new page where I jot down notes and tasks in a loosely structured format. I have found Notion's search to be very useful when trying to remember how or why I did something months ago. However, I still keep a physical, unruled, bound sketchbook for when I want to draw out class relationships, diagrams, etc - i.e anything that is much easier to describe on paper than with text alone. I usually don't need to actually "use" these diagrams - they're more a tool for working out and untangling an issue that eventually gets translated into code. Even with "paper-like" iPad screen protectors, I still strongly prefer writing on paper. So I have not found a way I like to combine Notion and physical writing yet.

But this post has me thinking about separation of concerns. Maybe I should stick to using Notion for anything that I want to search later and use the notebook for the aforementioned drafting and as a work-specific journal. While I haven't tried it much, I believe there's value in "brain-dumping" your day in a simple fashion similar to a diary. A therapist may advise you to deeply introspect on your overall state for that day, which I do think is a valuable tool, but may cause some inertia as a hard requirement for a "work-specific diary". Writing daily events in a simple fashion alongside some optionally additional, also simple, notes could help sort of "flush your mental buffer" of grand-scheme unimportant day-to-day information and help remember more important information. Sort of in the vein of Sherlock Holmes concept of his "mental attic" in the sense that you want to take care of removing/prevent clutter and instead keep track of what's useful to you.

4 years agodiscuss

I feel very similar but have struggled to set aside enough time to find a better replacement. For work I often build one-off scripts, web scrapers/automaters, data tools, and backend web apps/APIs. While I don't disagree with your comments about the ecosystem, I find myself very dependent on it to do the aforementioned work (playwright + beautifulsoup, peewee/native sqlite3 lib, numpy + scikit, Flask/Django) and is probably the main reason I've continued using it. Does anyone have recommendations for some directions I could research? Go and/or Rust seem to be clear contenders but I'm not sure the ecosystem has equivalents or at least mature-enough equivalents for the libraries I use. Very open to learning about other languages too but simply am out of the loop. Something with a great type system and some reasonable flexibility would be amazing (eg I like that I can mix classes and functions in modules easily in Python compared to say old-school Java where everything is a class). I'm also not looking for a language that's primarily functional at this time, too much to learn right now on top of a new language, but it's on my long term to-do list.

4 years agodiscuss

Every time I look at an update to this library my jaw drops to the floor. Once I find some spare time, I've been meaning to dive into the source code to get a feel for how you're pulling of the marvels that are rich and textualize. I've been "terminal-oriented" for at least a decade now and these frameworks feel like something out of a future I could have never imagined. Seriously impressive work Will, closely following your projects.

4 years agodiscuss

Lifelong US citizen here - I've barely heard anyone mention it in person my whole life. If it wasn't for reading about it in discussions like this, I would probably never even think about it. iMessage, social media DMs, and regular SMS seem to be the most common in my experience. Signal/Telegram are pretty much only used here by tech people and/or people discussing not-so-wholesome things and I'm always surprised to see people on HN talk about moving their whole family to one of these apps or using them more casually in general.

4 years agodiscuss

I would have never recognized this syntax as modern day C++ without being told. The language has evolved so much in the last 30+ years that I don't know how anyone is able to keep up unless they've been doing it for decades. As a junior dev, C++ scares the hell out of me - I'd rather wrestle with C, footguns and all, than wrestle with the foot-semiautomatic-rifles C++ provides.

4 years agodiscuss

PHP 7 & 8 introduce lots of improvements. I used PHP 5 when I got into web dev and remember moving to Python as soon as I could. At work I've had to use PHP 7 and have been very pleasantly surprised at how solid it is. Now that I have more experience than my PHP 5 days, it's become apparent that PHP is very clearly suited for web work and the built-in functions offer many conveniences for such work that simply aren't available in other languages (granted other languages weren't built for web like PHP was). I've been surprised by my speed in PHP and I chalk most of that up to many helpful standard library functions. Even though PHP has a lot of old/compat functions still present I'm always surprised by how little that actually ends up effecting my productivity. So it's not my choice for greenfield projects but I'm not nearly as adverse to it now after working with modern day PHP for a bit.

4 years agodiscuss

I've never heard it used like that but actually makes a lot of sense if thinking of an asterisk as the glob wildcard. Plus it sounds much better when spoken aloud in the context compared to asterisk or star