harshreality

1 saat öncediscuss

Anubis's default 1-week token lifetime may not be enough to dissuade enough scraper networks to make a difference, particularly with the default weight->difficulty level hierarchy, but that's for individual site admins to determine.

We can all argue based on how we envision "ideal" scraper networks being run and whether the web-PoW concept would stand up to that. However, what matters at present is that anubis helps many sites cope with misbehaving bot scrapers written by the script kiddies you mention, who don't care if the internet burns as long as they finish their scrape 1 hour faster. If anubis motivates them to devote a few brain cells to make their scrapers smarter, they may also fix the scrapers to not take down the sites they're scraping.

harshreality

5 saat öncediscuss

You don't have to wait for the flood. You can add rules to add weight based on system load, and add more difficulty levels beyond the ones in the default config.

I've only just started using it, and the existing config DSL doesn't let you do this, but it's just a patch away (in principle): instead of using system load as a signal to increase or decrease weights, use a combination of finalized weight and system load (or a load-equivalent or net-traffic-equivalent signal from some openmetrics agent) to select between difficulty buckets.

So, as an example, difficulty could have an arbitrary scale independent of load, and regardless of that scale, it could increase by +1 at J% cpu, +2 at K% cpu, +3 at L% CPU, etc. It doesn't look like a challenge will re-appear currently if anubis determines that a client should get a difficulty N challenge, but they only have a token for an N-1 or N-2 challenge. That would be easy enough to change in principle, but there might have to be exceptions... wouldn't want to break XHR, or regular PUT/POST requests for a client that already has a token for a lower difficulty than is currently required.

harshreality

8 saat öncediscuss

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.

It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.

The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.

For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.

The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.

[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.

harshreality

dündiscuss

MacOS Chrome? The code blocks look fine to me in linux {firefox, chromium, brave}. I think they're all using Qt.

harshreality

evvelsi gündiscuss

We're killing a lot more people than that; if we'd just tax the rich at 80% and send all that money abroad, we'd save millions more. Failure to do that is mass murder, the same as decreasing foreign aid funding; that is your thesis, right, that it's mass murder?

Doing less to save people in other countries that have no legal demand on our treasury is not "being responsible for [their] deaths." It's tragic, and it may even be a bad policy decision, but there's no responsibility (in the "duty to prevent harm" sense) or evil there.

harshreality

6 gün öncediscuss

I'm not here to defend publishers or the insane current copyright terms. However, in the traditional model of publishing, publishers subsidize production of new books from unproven authors--through book advances, copyediting services, and printing and distribution costs--via the money they make on the few successful and very few ultra-successful books.

If you take away copyright, you reduce the revenue of publishers, which reduces the number of unproven authors they can take chances on. (They're not very good at picking winners from the pool of new author manuscripts, not nearly as good as "agents" like to think they are, but that doesn't matter; they still wouldn't be able to take as many chances.)

harshreality

6 gün öncediscuss

If knowledge is to be free, then any corporate/commercial interest that locks up modified knowledge (code) to run their own services should have that locked-up knowledge freed from their commercial silo as well.

harshreality

12 gün öncediscuss

A generation of kids have grown up with just such assigned adults, who overwhelmingly did not apply sufficient oversight to the kids' use of social media.

In large part that's because, if they'd done so, the kids would've been socially isolated from their peers, at least the most normal ones with the most normal parents, which are the kinds of friends most other normal parents want their kids to have.

It's a collective action problem, except instead of "I can be better off if I ignore what I know is best for society", it's "if I ignore what I know is best for my kids psychologically, they will still have friends, and social media brainrot is a lesser evil than socially isolating my kid from all the normal kids at school."

And also, giving kids social media interaction devices is a convenient form of babysitting. It reduces up-front effort of parenting.

harshreality

13 gün öncediscuss

Why pre-cache? For speed... what is it, 30-50ms at most? If the authoritative server's TTL is <60minutes, do you force it to 3600? Do you audit all the connections that occur for every website you visit, collect all the domains hosting assets, and pre-cache those as well, or is the main site's domain the only critical one because that affects perceived latency the most?

harshreality

17 gün öncediscuss

A law like this just passed in New York State.

If this fails it'll be because the tech industry expresses disapproval too loudly to ignore.

The legislators don't care about the underlying criticism. Almost no legislators have ever used a 3d printer or written any software, beyond maybe simple assigned programs if they had a required intro-to-programming course. Few are "tech" people. The rest don't understand this technology, or any technology really, beyond it being a black box for specific purposes. They see 3d printing and plastic guns and think something must be done, because the 3d printing black boxes are producing dangerous weapons.

harshreality

18 gün öncediscuss

From the github repo readme:

> This repo was written almost entirely using AI with human oversight. If you see anything that needs fixed or would like to contribute, please email ... or reach out on Discord

Why not just provide an email address that's delivered directly to the agents you have developing Oak?

I didn't delve into the benchmark repo to understand what your loop is measuring. Why would an agent (without fine tuning or oak-specific context) be faster with oak than it is with git or jj?

harshreality

21 gün öncediscuss

(2017)

How much of this is still a problem with modern software/font stacks and harfbuzz?

harshreality

22 gün öncediscuss

Commonmark isn't serious because it doesn't include several major quality of life improvements like [^1] for footnote/reference syntax. Pandoc should be the serious "common" markdown standard.

harshreality

22 gün öncediscuss

How are you doing meaning-aware pagination if you're zipping together markdown columns?

A few minutes of asking an AI and writing a pair of simple typst docs with a few sections of #lorem() paragraphs resulted in what I think your ad-hoc scripting (in what language? add and your custom scripting to your collaborators' publishing stack) efforts do: a typst meta-document that pairs two simpler typst documents, section by section, paragraph by paragraph, each sub-document with its own column width and font size, paginated on paragraph breaks. You might balk at typst's embedded language, but it does avoid added dependencies and external scripting, and the language looks fairly simple a nice and functional.

It might require a somewhat different design to accommodate a marginal-note kind of thing with irregular vertical spacing not paragraph-aligned, but I suspect that's quite possible too, probably with some markup to label notes and align them vertically with corresponding labels in the primary text.

harshreality

24 gün öncediscuss

Platforms can't know what the false positive and false negative rates are simply from cases where a user is thought to be a minor and proves otherwise. Even if the law were written better, for example to say that 95% accuracy is acceptable, platforms would still have to verify id from at least a random sampling of thought-to-be-adult users, which doesn't help you if you're one of the ones randomly selected.

We need some kind of verification system that gives no extra information about users to the platforms, but I don't know if there's a true ZK way; it might require government involvement. I think that's fine. Govts could certainly run an age-verification system, give a signed yes or no token back to the user, with some permanently applied jitter per person so that platforms can't use cookies from returning users to figure out their birthday. As long as the government program has strict oversight to ensure it's not saving information about who's visiting what sites, it seems fine, or at least vastly better than entrusting photos of IDs to private 3rd parties.

harshreality

geçen aydiscuss

System cron can't implement your example, so I guess what you say is technically true... EINVAL is more obvious and intuitive than <systemd calendar incantation>. Cron just can't do what you want.

You keep saying systemd calendar format is "at least as complicated", but it seems to me the converse is true. Simple cases are simpler to represent in systemd because fields are optional in various circumstances; complex cases, where they can be implemented in system cron at all, look equivalently complex. Removing the weekday constraint, rounding to minutes, and removing the star for year since it's optional:

    31 * 01,03 01/2 *
    01/2-01,03 *:31

What's the point of your gripes? Would you really suggest to someone, who has a systemd-based distro, to spin up systemd-cron or cronie and keep writing crontabs? Do you have some specific complaint about systemd calendar syntax that you think could be improved?

harshreality

geçen aydiscuss

I take it that your position is: cron works for me, so why migrate to another system that uses its own "weirdo" syntax. Mine is: why not migrate since they're both "weirdo" syntax, and systemd's is at least based on ISO8601 and has integration with other systemd features.

Nobody is all that happy with cron's limitations. That's why there are so many variants, with varying features. When a variant has more than 5 fields, the syntax becomes ambiguous... not to a parser, but to whoever's reading and writing the lines, unless they already know the exact cron variant in use. Google, the one large cloud provider that seems to have stuck with basic cron syntax without quartz-scheduler extras, even invented their own pseudo-natural-language scheduling syntax to fill in for features cron lacks, or for people who don't like cron syntax.

Systemd's is definitely novel, but as I keep repeating, it's ISO8601 + cron. An alternative would be some extension to ISO8601 interval syntax, but that looks clunky and has probably never been used outside of niche applications for data interchange, probably inside xml files; it's used in https://docs.cibseven.org/manual/2.0/reference/bpmn20/events... (syntax at https://en.wikipedia.org/wiki/ISO_8601#Durations ). I think systemd's syntax is better than that.

The direct benefits, even if cron syntax suffices: Calendar timers are integrated with systemd, with all its process-management capabilities[1]. Unifying to systemd timers also avoids duplication of functionality ("is this scheduled process managed by systemd or cron?"). Cron doesn't let you `systemctl list-timers --all`. You'd have to grep through cron logs, possibly (and ironically) using journalctl, or have those logs already in a management dashboard, and that still doesn't let you see the next time the timer would fire. Cron also doesn't let you beta-test cron lines like you can with `systemd-analyze calendar --iterations 10 "Fri *~8..14/1"`

There's even a systemd-cron translator/generator that's in the official debian/ubuntu repos, and in Arch's AUR of course, which avoids the need for system cron but allows you to retain existing crontabs by importing them as systemd timers. They won't be quite as well integrated because it uses defaults for task names and other parameters, but the crontabs get run... without cron.

[1] https://unix.stackexchange.com/a/281203

harshreality

geçen aydiscuss

I think you're talking about syntactic simplicity. Systemd requires three field separators (space, :, -) because that's how ISO8601 dates are written. That means not all fields will be space separated, which may be more challenging to visually parse. It's not going to look as clean. Is that what you mean?

Everything else is simpler. You don't have to count fields to figure out whether you're dealing with a day vs a month, or an hour vs minute, no matter how the fields are aligned or how long or complex they are. You can scan left and right and look for space and dash and colon to figure out where you are. The formats for time and date are distinctive and ubiquitous. You don't have to guess what you're looking at when you just see one of them in isolation.

Very little follows from cron's format. You have to learn which fields are which for your particular implementation: does it include seconds? You have to learn the bizarre interaction between day and weekday, to do something non-trivial with them. You have to carefully keep track of which field you're in because you can't determine that from neighboring syntax. The visual clutter, if you want to call it that, of systemd's iso8601-based syntax only applies if you're doing complicated rules. In the vast majority of cases it would simply be an ISO8601 timestamp with various values replaced with *'s, and at most one /.

That first impression you had is what I thought too, before diving into it for my previous replies. But no, cron logic is that day and weekday are ORed. It lets you do clever things like run a script on the 1st and 15th of every month, AND on mondays, all in one line. I don't think I've ever seen such a thing in the wild. I'd go so far as to say it's stupid, because it's likely to be misinterpreted if anyone did use it that way. All the other fields are ANDed. I would say that systemd's AND logic is better... and more consistent, which is your own acceptance criterion.

harshreality

geçen aydiscuss

I am curious whether there's a more capable system cron that supports that kind of thing. Quartz job scheduler (java), AWS, and CF obviously wouldn't qualify. I think this is only possible if you're using a heavyweight job scheduler like those. Or... systemd.

It ultimately doesn't matter "for real", because almost nobody without some horrific legacy system to integrate with would need to use anything more than lists, ranges, and increments. There's a reason nobody's added these features to system crons. Clever trigger times for events that don't really need to be triggered at clever times... sure, but lacking that capability wouldn't change anything, the trigger would just be a more boring one.

Events like "last Friday of the month" or "nearest weekday to the 1st of the month" or even "Friday the 13th" are more for business logic, not system crontabs.

harshreality

geçen aydiscuss

Obviously you can do additional time & date checking in a shell wrapping the script or binary cron ultimately runs, just as you could do the time & date checking in the script or binary itself. That's far more complex.

Systemd enables cleaner, simpler syntax for common cases. Instead of "59 23 * * *", simply "23:59". Instead of "0 0 * * sun[day]", simply "sun[day]". 1st of the month and don't care exactly what time? Instead of "0 0 01 * *", simply "*-*-01".

Systemd started with the principle that they wanted to accept ISO8601 timestamps, then extended that with lists, increments, ranges. They developed that into a superset of basic cron capabilities, while maintaining similar syntax as best they could for increments and lists; they diverged for ranges and nth-from-last because those conflicted with the - date separator. They repurposed the little-used ~ cron randomization operator in the process. (systemd uses a separate RandomizedDelaySec line for that, which is also arguably superior because it randomizes per trigger, not on initial load of the timer)

The alternative is how AWS does it. They have separate cron() and at() syntax. at() takes timestamps. cron() takes cron+quartz syntax, mangled to remove seconds. They also have rate(value units), because apparently cron syntax is too complicated for most people. I'm sure in theory they did it to support "every 7 minutes" cases. I bet if you surveyed actual rate() AWS schedules, 99% would be cases that evenly divide a larger time unit, and could therefore be implemented with cron().

Cron syntax is more of a mess than I thought. While basic system crons don't support any advanced features, AWS and Cloudflare have adopted LW# capabilities, derived from quartz job scheduler syntax (Cloudflare references quartz explicitly). Quartz has extra fields for seconds and year (year optional, so it must go last, which breaks d m Y sequencing) so its syntax has 6 or 7 fields vs the standard 5. CF uses 5 fields, no seconds or years. AWS uses 6 fields, years but not seconds. Whenever you interact with cron-derived systems, you have to remember not just whether they support quartz-style syntax, but whether they support seconds and/or years. Is that really simpler?

The one feature I found interesting, "W", may not work the way I anticipated according to quartz's documentation. I thought "nearest weekday to the 1st of the month" might be useful because it maps to some billing cycles, but quartz (at least through 2.3) couldn't do that. The 2.3 docs say that 1W, if the 1st falls on a Saturday, triggers on the 3rd because it won't jump backwards over a month boundary. That was my only interesting use case for "W". Systemd does everything else, though it requires more verbosity in some cases, it's simpler in many others. 2.4/2.5 are from the last couple of years, and their docs are restructured and don't mention that, so maybe it was finally fixed? I'm not about to install java crud to test.

I didn't find the *-*~01 syntax that strange (last day of every month). The "fri *-*~07/1" syntax for "last friday of every month" is what I found magical, but it makes sense now that I've thought about it more.

What would you prefer for 3rd friday of the month? Cron (5-field, quartz syntax)

    0 0 ? * fri#3  (special syntax is needed because cron uses logical-or to combine day and weekday fields)

or systemd

    fri *-*-15..21 (works because systemd day and weekday are joined with logical-and)

[cf] https://developers.cloudflare.com/workers/configuration/cron...

[aws] https://docs.aws.amazon.com/scheduler/latest/UserGuide/sched...

harshreality

Submissions

Young journalists expose Russian-linked vessels off the Dutch and German coast

Database failure lessons at Amazon in 1997 (2004) [pdf]

comments