HackerTrans
TopNewTrendsCommentsPastAskShowJobs

fortytwo79

no profile record

Submissions

Show HN: Privacy-focused news feed that groups stories, using NLP

drewesnews.com
5 points·by fortytwo79·5 ปีที่แล้ว·6 comments

comments

fortytwo79
·2 ปีที่แล้ว·discuss
And could you imagine the bokeh on a portrait?!
fortytwo79
·4 ปีที่แล้ว·discuss
Isn't this already the case with marijuana? Legal in one state, if you travel with it to another state, but make a stop in an illegal state, you can go to jail?

I don't know much about laws on that topic, but it seems to be a similar case to me.
fortytwo79
·4 ปีที่แล้ว·discuss
Okay, now someone needs to do the same study with innocuous end game scenarios. (I know this study links to one other paper that makes this case, but I can't access it, and the abstract has a biased tone)

If you're going to explore worst-case, so you can think through preparedness, then you should also consider the trivial case to make sure we aren't over-responsive either.
fortytwo79
·4 ปีที่แล้ว·discuss
The article also states this:

"With just over 19 weeks into the year, this averages out to about 10 such attacks a week," when referring to the Buffalo shooting.

Saying "10 such attacks" implies the attacks are all similar in characteristics beyond just the number of people involved. The article talks about mental health, it talks about a pre-planned desire to kill. But it doesn't talk about gang violence. These are very different sources of intent. If you want to use the gun violence archive data to make a point, it should be about gun violence in totality - not cherry picking their numbers to make the case that the US is full of hate-filled crazies who are randomly shooting places up 10x per week.
fortytwo79
·4 ปีที่แล้ว·discuss
But Buffett's point is that each unit of investment he pursues has the potential to produce incrementally more value. You're saying the aggregate technology underlying bitcoin has value - which it does. But you don't get twice the tech value by doubling your amount of coin owned. Conversely, if you double your farm holdings, you have double the ability to produce food.

Also, Buffett's quote here isn't some significantly new perspective. Hasn't this been the primary argument against bitcoin since the beginning?
fortytwo79
·4 ปีที่แล้ว·discuss
And yet, you shut down the possibility of religion without any degree of exploration.

I don't need you to be religious. I understand annoyance with the fact that the culture you're in is pushing something you don't agree with (I experience that as well on other topics). What I take issue with is the attitude that religion deserves condescension and vitriol, simply because you don't subscribe to it, or because you find it annoying. I see no recognition that faith and logic aren't mutually exclusive.

Don't have religion - that's fine. But don't pretend to understand (and then trivialize) any religious tenet, if you're unwilling to fairly explore it.

> Science is exploration. Religion is not.

You're conflating religiousness with dogma. Dogma, by definition, is belief without exploration. Religion, however, invites endless exploration.
fortytwo79
·4 ปีที่แล้ว·discuss
This kind of condescension towards religion is so close-minded, and it's incredibly tiring. I think you'd be surprised just how much the faithful appreciate and embrace science, and how much philosophical thinking about reality and existence we actually do. Adult religiousness is not the same as the cartoon version that children are introduced to, that so many atheists use to attack. The lazy, unintellectual thing to do is to base all your understanding of reality only on what science can describe, and to never explore beyond what can be proven as a "fact." Your statement about explaining away facts paints the faithful as simple-minded rubes, which is a cheap shot and is absent any understanding of how the religious actually view the world.
fortytwo79
·4 ปีที่แล้ว·discuss
I appreciated the "retro encabulator"-esque reference in the industrial sample.
fortytwo79
·4 ปีที่แล้ว·discuss
Agreed. Speaking as someone who has hired hundreds of software engineers, I can tell you that having a degree at all tells me a lot about your work ethic. Sure, I'd prefer a comp sci background, but your degree is nearly as good. It's an indicator of how you think.

You know what other degree I see a LOT of in SWE? Music. I've hired countless music majors.
fortytwo79
·4 ปีที่แล้ว·discuss
People have always been that way. Before Rogan showed up, before the internet was even invented, there were "nutters" that had deeply misinformed perspectives which they followed, sometimes, to death. I'm sure someone will argue scale is the differentiator, but I'd argue that the proportions of people who ignore facts vs consider facts, is probably unchanged. (Although admittedly, I have no statistics)

Also I think a big part of the problem is the aggressive labeling of content as "misinformation" or "fake news." To me, misinformation implies propaganda issued and promoted by an enemy entity. But today, it is a term that is used to mean anything that has a fact (whether verifiably correct or incorrect) that implies a conclusion that is generally unacceptable.

For example, if the generally acceptable premise is: "everyone who is able should get a vaccination," then publicly talking to someone harmed by a vaccination (even if it's true) would be considered misinformation, because it potentially concludes something opposing the acceptable premise.

If we can't openly share ideas, good, bad, informed, misinformed, then the 99% (fake number) of us who aren't "nutters" that follow bad advice to extreme conclusions, will be denied the volume of data, perspectives, and opinions we need to make a truly informed decision.
fortytwo79
·5 ปีที่แล้ว·discuss
I don't think that comment was implying those measures are to blame for rising deaths. I think it was to argue that they haven't had the impact they were expected to have.
fortytwo79
·5 ปีที่แล้ว·discuss
Yeah, I need to fix that. You're seeing a performance bug. There's a cache in front of the API, and sometimes if you hit the API outside of the cache window, the lambda running in the background is a bit slow. Subsequent loads will hit the cache.

Although, I'm surprised you didn't see the page load UI. There should be a pulsating circle to indicate it's loading.

Thanks for the feedback!
fortytwo79
·5 ปีที่แล้ว·discuss
You bet!

There's three major phases to the document matching: Corpus Management, Frequency Modeling, and Matching & Clustering.

1. Corpus Management

- First we pull the raw articles

- Then we build a bag-of-words document, based on our overall dictionary, so we end up with a bunch of numbers to represent all the words and their frequencies for each document.

- We store all the stories for that day as bag-of-word, in a Matrix Market format.

- The day's .mm file is updated every few minutes into an S3 bucket, along with the dictionary. This way I can easily compose a corpus of documents for a single day, multiple days, or all time

- I use the Gensim library for Python to do most of the above.

2. Frequency Modeling

- As stories come in throughout the day, I periodically refresh a TF-IDF model for the entire corpus of stories.

- TF-IDF just allows us to see which terms happens frequently in a story, but relatively infrequently across all stories. So a word like "Hacker" would be relevant if it appears frequently, but "the" would not since it occurs across all documents.

- TF-IDF modeling also uses the Gensim library.

- TF-IDF model is stored in another S3 bucket, to be pulled on update by the Matching & Clustering job.

3. Matching & Clustering

- First we fit a limited corpus (usually the last 10k or so stories) to the TF-IDF model, and keep that as a sparse matrix.

- This gives us the ability to quickly determine the per-word importance of a document, and to represent that as a vector.

- Next we do a simple cosine similarity of the 10k documents against themselves. This tells us how much of an angle there is between each document vector against each other document vector. In other words, it's a measure of similarity.

- We limit all of this to 10k documents, because it would be computationally prohibitive to compare ALL stories to ALL OTHER stories. Since most news stories are published relatively close to one another, we only have to compare recent stories. 10k seems to produce good results. We can further shard this data set by news category if we need to (i.e., compare only sports stories or political stories).

- Next we use SciPy, and create a Ward linkage matrix.

- Next, also using SciPy, we use fcluster to do agglomerative clustering. These last two steps produce a cluster tree that puts similar stories together.

- Finally, we slice apart the clustered data set at a certain similarity height. Kind of like cutting through a head of broccoli above the stalk. The clusters we're left with are the most similar articles of that batch of 10k.

After all that is done, we just store associations between articles that were determined to be similar to one another. Since we're constantly running this process, the 10k story window keeps sliding forward, so you're able to store similarities of stories that are similar to far older stories, provided there is another story in-between that both are similar to. For example, if story 12,000 is similar to story 9,000, and story 9,000 is similar to story 10, you'll end up storing that story 10 is similar to story 12,000, even though you didn't directly compare them.
fortytwo79
·5 ปีที่แล้ว·discuss
I built Drewes.NEWS as a way to learn NLP and serverless architecture, and to find bias in news by reading the same story from multiple outlets. Now it's evolved into a useful, privacy-focused tool.

It's privacy-focused in that there are no cookies, no usage of Google or Facebook components (like Google Analytics or Ads). No data tracking on users whatsoever.

There are bugs I'm aware of, but am looking for feedback on if this format and function is useful.

For those interested in the NLP side of this or the serverless side, I'd be happy to answer questions about how it was put together. The short version is, I pull down RSS feeds from 33 news sites (approximately 1M stories in the database so far), store them, create a term frequency model, cluster the most recent 10k stories based on TF similarity vectors, then store story similarities.

In the future I'd like to add paging, searching, and more filtering. I'm also thinking about having a URL for each story, that would show all the similar articles. That way, if you don't want to link directly to a particular news source, you can link to the drewesnews aggregate URL, and the reader can pick whichever source he/she wants to read.

Any feedback would be much appreciated!
fortytwo79
·5 ปีที่แล้ว·discuss
Ha! I literally LOL'ed this. Well played.
fortytwo79
·5 ปีที่แล้ว·discuss
I think the full lyric here is actually "pandemic ain't real, they just planned it."

The meaning and intent of the lyrics are open to interpretation, but I think the latter part of the lyric implies political commentary rather than medical commentary.

Also, I think we need a tighter definition of "medical misinformation." Is it to promote anything that is medically harmful? Well, then why can musicians sing endlessly about drugs and unsafe sex? What about videos on non-fda regulated vitamins and minerals, or weight loss solutions? Surely most doctors wouldn't advise taking recreational drugs, or jumping on to the latest weight loss magic pill.

And yes, YouTube is a private enterprise. But we can still, as a society, openly dialog on what we think is wrong with their practices.
fortytwo79
·5 ปีที่แล้ว·discuss
Watched it here: https://mobile.twitter.com/Saint_BTC/status/1451559663753859...

Mostly just the rappers waving their hands around and showing their guns.
fortytwo79
·5 ปีที่แล้ว·discuss
I wish people would pay me to write headlines like this.

How about this idea for an article:

"People who find baseball stadium chairs uncomfortable risk developing diabetes.