HackerTrans
TopNewTrendsCommentsPastAskShowJobs

turc1656

no profile record

comments

turc1656
·il y a 6 ans·discuss
I think you may be confusing bravado/ego/bombast with actual desire and by extension, risk.
turc1656
·il y a 8 ans·discuss
Ohhhhh, yeah that changes things. I didn't realize the pagelinks database you mentioned was not generated by you based on the master Wiki dump that they keep available for download. In that case, yes this seems unsolvable without 1) petitioning them to update the code on their side to adjust for this and exclude these erroneous links or 2) deciding to create your own pagelink database based on the master wiki download and updating this database periodically (I forget how often Wiki updates the master downloads). While 1) is clearly the preferable option, it is unlikely to occur, unless maybe they are willing to add some columns to provide more details about the links and what sections, tags, classes, etc. they are a part of. That might be more palatable for Wiki to provide as it is just a code change on their end to supply more information rather than something possibly affecting many users by excluding the links altogether.
turc1656
·il y a 8 ans·discuss
I'm not sure how the backend is structured, but it seems that you must parse the individual pages at some point or another. I took a quick look at the Wikipedia HTML for a few pages and I would suggest stripping out anything within (or nested inside) of classes like "mw-cite-backlink", "reference-text", "citation book", "citation journal", etc. Also, you can probably strip out anything inside of a <cite> HTML tag.

I'm sure there are more classes and tags, but that hopefully should give you a solid place to start.

EDIT - You can also strip out or ignore anything inside of the ordered list for references - <ol class="references">...</ol>

Also, some pages aren't documented in the same way, so something like this page - https://en.wikipedia.org/wiki/X_Window_System - doesn't have any classes or easy way to parse it for the References section even though the Notes section was set up in a more organized way. However, you could take note that the <span> tag contains class="mw-headline" id="References" and the text value is also References and then ignore everything until the next <span> begins.
turc1656
·il y a 8 ans·discuss
Not sure if you deliberately designed it this way, but I noticed when spot checking some results that it includes the bibliography section links as connections. This seems like it may not be desirable. Example, I did a search that went from the Crusades to Buzz Aldrin and I noticed that Routledge was the first hop from the Crusades. It strikes me as odd that Routledge (a publishing company) would be mentioned on the Wiki article for the Crusades. So I went to look and noticed the link it took was the citation for a book published by this company. I wouldn't really count that as a legitimate hop since it's a citation, not a content link.

EDIT - I noticed this also applies to the Notes section.
turc1656
·il y a 8 ans·discuss
Very cool. And a lot less degrees of freedom between my tests than I expected - so far everything has a legitimate 3 degrees when I expected much more. Higgs Boson to Taylor Swift in 3 steps, amazing. Not only that, but 91 ways to get there in just 3 steps.

This basically quantifies what my wife and I jokingly refer to "rabbit-holing" online. She'll ask me what I'm reading and it will be something totally unrelated to what I said I was coming to look up. And she's always like "how did that happen?" And I never have a good answer. But now I do (if it involves Wikipedia)!