Palantir acquires Kimono(kimonolabs.com)
kimonolabs.com
Palantir acquires Kimono
https://www.kimonolabs.com/
140 comments
Sort of plays right into the whole argument being made over on another thread (https://news.ycombinator.com/item?id=11105198) that companies shouldn't rely too heavily on proprietary tools to solve their data challenges.
Palantir is the poster child for black box software company that keeps trying to sell you more expensive licenses for their mystery algorithm. I know they've had some success in the government sector but I've seen them make a real mess of things in the private sector. At my last company their system just became a giant paracite where data went in and stuff came out the other end but it was entirely unclear what happened in between. The more we looked into the mystery middle ground the uglier it got.
Palantir is the poster child for black box software company that keeps trying to sell you more expensive licenses for their mystery algorithm. I know they've had some success in the government sector but I've seen them make a real mess of things in the private sector. At my last company their system just became a giant paracite where data went in and stuff came out the other end but it was entirely unclear what happened in between. The more we looked into the mystery middle ground the uglier it got.
There is no mystery algorithm. It's just cheap college labor working obscene hours to pump out "data analysis" scripts. The sad thing is that it actually works. The places that hire palantir are so mired in bureaucracy and process that whatever palantir builds seems to be better. Of course, just like SAP and Accenture you're now kinda locked in with support contracts.
Modern tools like spark and mesos in theory let you do what palantir does and then some but it assumes your internal IT team is semi-competent which more often than not is not a correct assumption.
Modern tools like spark and mesos in theory let you do what palantir does and then some but it assumes your internal IT team is semi-competent which more often than not is not a correct assumption.
Is it cheap college labour? They pay pretty well.
I thought they pay in the 100-110k range for new grads, which is not great by any standard (given how 'selective' they are)
For interns maybe. But average salary across the board is not high at all.
Maybe it tops out early, but I'm given to understand junior salaries are also very competitive. (Maybe this is less true by comparison in the States -- it's fairly true here in the UK.)
Dear Kimono Users
We at CloudScrape.com are happy to welcome you onboard our platform - you get 20 hours of scraping runtime for free and the ability to navigate much more complex websites, opening up tons more opportunities then you've had previously. And we're here to stay :)
Kind regards, Henrik Hofmeister, CTO, Co-Founder @ CloudScrape.com
http://cloudscrape.com
We at CloudScrape.com are happy to welcome you onboard our platform - you get 20 hours of scraping runtime for free and the ability to navigate much more complex websites, opening up tons more opportunities then you've had previously. And we're here to stay :)
Kind regards, Henrik Hofmeister, CTO, Co-Founder @ CloudScrape.com
http://cloudscrape.com
"Used by" smells of being a vanity metric - probably signed up users, rather than monthly active.
My guess, they weren't commercially successful, and aquihire was the best way out.
My guess, they weren't commercially successful, and aquihire was the best way out.
Yup. I signed up with an account to play with it once. Never used it after the initial week. I wonder how many others of that 125k were just drive by test drives.
Well, if you want the real answer:
Why? Money.
Why? Money.
You guys need replacement? We will be launching the web version of Data Scraping Studio on same day(Feb, 29 2015)
http://www.datascraping.co
http://www.datascraping.co
> we couldn’t be happier to announce
The tone of the announcement is in bad taste given the bad news for their customers.
The tone of the announcement is in bad taste given the bad news for their customers.
This is really bad news for us, and the tone of the announcement grated. Two weeks notice is very bad form, I'm having to come back from vacation to fix this.
When we evaluated options a year ago, kimono was the best fit. We'd happily have paid, and did ask. I don't understand what happened... Did they run out of funding?
When we evaluated options a year ago, kimono was the best fit. We'd happily have paid, and did ask. I don't understand what happened... Did they run out of funding?
Here is a statement that can give you some insight
"Kimono Labs explained that in the two years since launching, it wasn’t able to have the impact that it wanted, and that the acquisition by Palantir will give the team “unmatched support, resources, and the ability to work on things we could not tackle alone as a small startup.”"
from http://venturebeat.com/2016/02/15/palantir-acquires-kimono-l...
ParseHub is much better at getting websites from shitty, complicated sites. Kimono was great for something simple, but I found ParseHub to be a more powerful and flexible web scraper.
Calling all (almost) Kimono Labs developers to migrate to Webhose.io
http://blog.webhose.io/2016/02/16/calling-all-almost-kimono-...
If you have been burned by free services like this shutting down, give https://scrapehero.com a try - we love our customers and they love us back. We dont plan to abandon them with a 2 week notice.
Here are some of our thoughts https://learn.scrapehero.com/demise-of-kimono-labs-and-the-f...
We've had enterprises that depended on Kimono come to us; which isn't surprising anymore :) Hope others will give Grepsr a try as well! - https://www.grepsr.com/
Kind of bummed on the two weeks notice, but I'm sure there's some pressure and tradeoff the team had to make, which I guess softens the blow a bit (for me at least).
The desktop download softens the blow somewhat...
...except it's not possible to run APIs on schedule - which is exactly what I used it for.
Gutted.
[deleted]
The worst part about Palantir is their ability to masquerade as this hip startup, as though working for them is as innocent as working for Imgur or Twitter or whatever. They recruit the hell out of MIT students, and it sucks seeing my friends interview for this incredibly shitty company that has openly said it won't go public since that would make running their business "very difficult" (hm, because you're doing reprehensible work perhaps?).
I wish people picked their jobs to match their ethics instead of the other way around. It makes me sad when I hear about my friends going to intern for this place.
I wish people picked their jobs to match their ethics instead of the other way around. It makes me sad when I hear about my friends going to intern for this place.
I know some engineers who work or worked there. Aside from the normal engineers (tools, backend, etc.), they are essentially high-priced tech consultants sent to companies and orgs that have no internal software talent at all, to help them analyze their (atrociously stored/managed) data. None of them work on anything shady. They essentially build web apps and backend apps to analyze and visualize data. Nothing vodoo military about it, although the military is a customer.
The pay actually isn't actually particularly high for SV standards. I think Facebook outpays them, for example - and Facebook is less selective than Palantir. But their reputation is that they're full of top engineers and you'll learn a ton there, and that's why the people I know work/worked there.
The downsides I've heard about are no work/life balance and fratty culture. Those are real downside, not the FUD about ethics.
The pay actually isn't actually particularly high for SV standards. I think Facebook outpays them, for example - and Facebook is less selective than Palantir. But their reputation is that they're full of top engineers and you'll learn a ton there, and that's why the people I know work/worked there.
The downsides I've heard about are no work/life balance and fratty culture. Those are real downside, not the FUD about ethics.
> it won't go public since that would make running their business "very difficult" (hm, because you're doing reprehensible work perhaps?)
Well, more generally, anyone who has a nonstandard set of ethical beliefs, and is basing their company around that set of beliefs, can't really go public without their belief-set being thrown under the bus of their shareholders' belief-set.
For example, I've often considered starting a company in the game industry that actually hires experienced adults and treats them well, rather than hiring fresh college students and burning them out. I imagine, though, that if my hypothetical company became publicly-traded, I'd be forced to relinquish this policy in the name of short-sighted market competitiveness.
Well, more generally, anyone who has a nonstandard set of ethical beliefs, and is basing their company around that set of beliefs, can't really go public without their belief-set being thrown under the bus of their shareholders' belief-set.
For example, I've often considered starting a company in the game industry that actually hires experienced adults and treats them well, rather than hiring fresh college students and burning them out. I imagine, though, that if my hypothetical company became publicly-traded, I'd be forced to relinquish this policy in the name of short-sighted market competitiveness.
You could go the Google and Facebook route of having voting-class and non-voting class shares. Sergey and Larry, and Zuck still retain control of their companies.
Until they die, the company goes bankrupt, or it's subject to such a compelling buyout offer that they can't responsibly decline it.
Capitalism wins, eventually.
Capitalism wins, eventually.
Interning for Palantir is a pretty big resume boost for college students, and they have one of the highest paying internships in the valley. There's some serious benefits to going to Palantir.
If you have the ability to intern at Palantir you have the ability to intern at any number of more responsible companies. Palantir has a bigger booth at the career fair, but I'd bet working at Apple or Uber is just as much of a resume boost at much less of a societal cost. (source: am a college student)
Interning at Apple or Uber has much more stiff competition though. The barrier to entry for Palantir is remarkably low considering their pay, it's not surprising that college students apply for Palantir. (Source: am college senior)
[deleted]
Will they hire you if you smoke pot? I hear college students sometimes do that.
Thiel has an entire cannabis investment club.
Just rail on the hypocrisy if confronted.
Just rail on the hypocrisy if confronted.
Apple and Uber are not as much of a resume boost (in Silicon Valley, to a SV recruiter). Palantir is sort of on the level of quant finance / Dropbox 2 years ago.
Apple and Uber are hiring like crazy and so it's quite a lot easier to get in there as an intern than at Palantir (which is more selective).
Apple and Uber are hiring like crazy and so it's quite a lot easier to get in there as an intern than at Palantir (which is more selective).
That's surprising, as they're known for being incredibly shady and have a pretty bad rep. They might pay a lot, but it's because it's less desirable to work for the Evil Empire.
Maybe they are useful for students to signal their lack of ethics to employers? “Look, they once worked at Palantir, so they’ll have no problems doing whatever shady stuff we tell them to!”
It's a big resume boost yes. The pay is competitive but not especially higher than any other top companies/startups, though.
Sorry I can't stop myself:
Interning as a prison ward in Auschwitz is a pretty big resume boost for anyone who served in a prison system, and Nazis have one of the highest paying internships abroad. There's some serious benefits of becoming a prison ward in Auschwitz.
That's how I see this parasite company.
Interning as a prison ward in Auschwitz is a pretty big resume boost for anyone who served in a prison system, and Nazis have one of the highest paying internships abroad. There's some serious benefits of becoming a prison ward in Auschwitz.
That's how I see this parasite company.
Curious to know what Palantir are doing that you have a such an issue with - surely they aren't committing genocide?
Directly not, but to say the company is adding any benefits to our society would be a long stretch.
That's a rather large downgrade to the previous accusations. From evil company to a company with no benefit to mankind. I think you can say that about a lot of peoples jobs - what's your point?
What makes them a shitty company? Why is their work reprehensible?
They have one of the worst work/balances in the valley, fratty/kool aid culture, claim to be "making a difference" but really what they actually work on would be a major downside for most normal people. For example, they make targeting software and are very likely responsible for assassinations by the US and when we disappear people. They're also working on big brother tech if that's your thing.
What is targeting software? How are they responsible for assassinations and disappearing people? What is this big brother tech?
First of all:
https://en.m.wikipedia.org/wiki/Palantir_Technologies#Contro...
Also:
> When intruders from the hacker group Anonymous gained access to thousands of emails stored on the servers of the security firm HB Gary Federal, the emails revealed that Palantir had worked with HB Gary Federal to develop proposals for attacking WikiLeaks’ infrastructure, blackmailing its supporters and identifying donors.
http://www.forbes.com/sites/andygreenberg/2011/02/09/did-sec...
There was also the sex scandal with their founder and a Stanford student: http://www.nytimes.com/2015/02/15/magazine/the-stanford-unde...
This thread on HN also has some discussion about their government and military contracts:
https://news.ycombinator.com/item?id=11036156
https://en.m.wikipedia.org/wiki/Palantir_Technologies#Contro...
Also:
> When intruders from the hacker group Anonymous gained access to thousands of emails stored on the servers of the security firm HB Gary Federal, the emails revealed that Palantir had worked with HB Gary Federal to develop proposals for attacking WikiLeaks’ infrastructure, blackmailing its supporters and identifying donors.
http://www.forbes.com/sites/andygreenberg/2011/02/09/did-sec...
There was also the sex scandal with their founder and a Stanford student: http://www.nytimes.com/2015/02/15/magazine/the-stanford-unde...
This thread on HN also has some discussion about their government and military contracts:
https://news.ycombinator.com/item?id=11036156
The sex accusations ended up being baseless, and Stanford reversed their ban on Lonsdale entering the campus (and Stanford has a lower evidentiary standard than the courts)
Also, your quote doesn't appear in that article at all any more.
Also, your quote doesn't appear in that article at all any more.
Sorry here's another link:
http://www.forbes.com/sites/andygreenberg/2013/06/07/startup...
Second to last paragraph.
As for the sex accusations, who knows what really happened. Though the article doesn't exactly paint him- edit- either of them- in a flattering light. It does show poor judgement for the dude to get involved with a student he is mentoring. Then again, he also stumped on the campaign trail for Rand Paul of all people and also pals around with Mitt Romney.
http://www.forbes.com/sites/andygreenberg/2013/06/07/startup...
Second to last paragraph.
As for the sex accusations, who knows what really happened. Though the article doesn't exactly paint him- edit- either of them- in a flattering light. It does show poor judgement for the dude to get involved with a student he is mentoring. Then again, he also stumped on the campaign trail for Rand Paul of all people and also pals around with Mitt Romney.
Look, my point is, it's not valid to bring up the sex accusations, because they were proved to be false.
Their YouTube channel has 100's of videos that show what they do. The one that blew my mind: GovCon7: Introduction to Palantir: https://youtu.be/f86VKjFSMJE
"Targetting software" (as in, software that actually points a weapon at a target) most likely isn't something that Palantir is involved in.
However, data mining for target leads (e.g. through cell phone, financial, social network + other data) would be much more up their alley.
However, data mining for target leads (e.g. through cell phone, financial, social network + other data) would be much more up their alley.
On one hand, Palantir have (at best) questionable connections and I wouldn't be a rush to work for them. On the other, with the level of pay/benefits they provide, I'm not surprised they can find people without similar qualms (or people who overlook or aren't aware of them).
I won't share specifics (as they aren't mine to share), but I have a friend working for Palantir overseas. Once essential benefits (e.g. accommodation) are factored in, his salary is north of six figures (GBP). For someone only a few years past university, I imagine that sort of offer is difficult to refuse.
I won't share specifics (as they aren't mine to share), but I have a friend working for Palantir overseas. Once essential benefits (e.g. accommodation) are factored in, his salary is north of six figures (GBP). For someone only a few years past university, I imagine that sort of offer is difficult to refuse.
It's cool to get a 6 fig GBP salary but tbh, assuming what I've heard about the working hours there is true, I could still just do a combo of freelancing and contracting and make way, way more cash.
[deleted]
Care to identify any of your concerns?
My wife is a student and looking for internship she will work for a company that helps people write about how they ran over than grandma if she gets a legal. Morality and ethics have a price always.
Relevant Scraping/Crawling Lead Mining Tools:
http://grabby.io
http://fullcontact.com
http://emailhunter.co
http://clearbit.com
http://toofr.com
http://import.io
http://kimonolabs.com
http://apifier.com (favs)
http://elink.club
http://www.eliteproxyswitcher.com/ - ;)
http://grabby.io
http://fullcontact.com
http://emailhunter.co
http://clearbit.com
http://toofr.com
http://import.io
http://kimonolabs.com
http://apifier.com (favs)
http://elink.club
http://www.eliteproxyswitcher.com/ - ;)
Three more:
https://github.com/scrapinghub/portia/ (our open source equivalent of Kimonolabs)
http://scrapy.org (allows to scrape more complex sites; also OSS)
http://scrapinghub.com (for a cloud-based platform and a smart proxy rotator)
(Disclaimer: working there.)
https://github.com/scrapinghub/portia/ (our open source equivalent of Kimonolabs)
http://scrapy.org (allows to scrape more complex sites; also OSS)
http://scrapinghub.com (for a cloud-based platform and a smart proxy rotator)
(Disclaimer: working there.)
scrapinghub is pricy, isn't it?
Pricey as in we've a free plan? :-)
More seriously though, it's basically a bargain. You get:
- Access to the largest team of Scrapy, Portia, and Frontera experts around at your fingertip.
- AWS functionality for actually less than what it would cost you to set things up yourself (think VBR IP over ATM).
- Smart proxy rotating tech at your fingertip with tens of thousands of IP addresses spread across a slew of different IP networks.
- Built-in integrations with a growing list of third parties.
The list could go on and on... It's definitely not pricey. :-)
More seriously though, it's basically a bargain. You get:
- Access to the largest team of Scrapy, Portia, and Frontera experts around at your fingertip.
- AWS functionality for actually less than what it would cost you to set things up yourself (think VBR IP over ATM).
- Smart proxy rotating tech at your fingertip with tens of thousands of IP addresses spread across a slew of different IP networks.
- Built-in integrations with a growing list of third parties.
The list could go on and on... It's definitely not pricey. :-)
https://webrobots.io - we give free Chrome extension to run scraper robots on your own computer. For paid customers we host robust cloud of scraping instances with all advanced features one can expect. To be fair - you need to understand javascript to write robots on our platform, we are not point-and-click doodling tool. Upside of writing robots in code - nearly unlimited possibilities with dynamic content sites, in-flight data cleaning, etc.
Was really hoping someone would share a comment like this, thank you. I feel like I just got completely screwed by Kimono...
damn. I was gonna say this means it's time to actually launch my version of this I built 3 of 4 years ago, but it looks like apifier is almost the exact same thing.....
i have found import.io to be pretty good
Have you tried pulling out anything from LinkedIn there?
Full disclosure: I'm one of the co-founders of ParseHub.
For folks looking for an alternative, check out http://www.parsehub.com
ParseHub is especially good at dealing with dynamic websites and multidimensional data (arbitrary relationships instead of just rows and columns).
Our team would be happy to help you migrate from Kimono. We're bootstrapped and profitable, so we don't have pressure to sell when an offer comes along. We also pledged to release the back-end code under a liberal open source license if we were ever in a position where we could no longer provide service.
For folks looking for an alternative, check out http://www.parsehub.com
ParseHub is especially good at dealing with dynamic websites and multidimensional data (arbitrary relationships instead of just rows and columns).
Our team would be happy to help you migrate from Kimono. We're bootstrapped and profitable, so we don't have pressure to sell when an offer comes along. We also pledged to release the back-end code under a liberal open source license if we were ever in a position where we could no longer provide service.
Parsehub is easily the best hosted craping service on the market. We investigated everybody from kimono to import.io and if you want it done right they are the only option. Furthermore the guarantee of an open source version should they ever decide to close is extremely reassuring.
Awesome. Thanks for sharing your service. My current project has been faltering because of my ignorance of these types of tools. Yours looks like it's going to make a huge difference for me!
This is a good list of alternatives:
https://www.quora.com/Which-are-some-of-the-best-web-data-sc...
https://www.quora.com/Which-are-some-of-the-best-web-data-sc...
As someone who used Kimono a few times before the closure...this announcement does not surprise me at all.
The target demographic was developers who were too lazy to write simple scrapers for a given predefined use case. However, scraping most modern websites is not simple, and requires a lot of code to ensure the correct HTML elements are scraped and the data is transformed correctly...which defeats the purpose of using Kimono.
Kimono joining Palantir was likely an acquihire, especially since the core service was permissively free.
The target demographic was developers who were too lazy to write simple scrapers for a given predefined use case. However, scraping most modern websites is not simple, and requires a lot of code to ensure the correct HTML elements are scraped and the data is transformed correctly...which defeats the purpose of using Kimono.
Kimono joining Palantir was likely an acquihire, especially since the core service was permissively free.
"too lazy" seems like an unfair characterization. If a service lets you do what you want faster, why not use it? I'm not familiar with Kimono, but if you can just rip it out and replace it with your own code later, why not?
It wasn't really faster. The OP actually touched on that point, but let me repeat it without ambiguity:
An actual parser for the complex use-cases requires some custom code and isn't covered by what Kimono does.
What Kimono does can trivially be achieved by writing 15 lines of python (after importing httplib2 + BeautifulSoup). Which is even reusable.
The only real use-case they had was the visual interface, so that "non-programmers can do it. No code required, etc. etc." Heh. As if that ever did, or ever could work.
An actual parser for the complex use-cases requires some custom code and isn't covered by what Kimono does.
What Kimono does can trivially be achieved by writing 15 lines of python (after importing httplib2 + BeautifulSoup). Which is even reusable.
The only real use-case they had was the visual interface, so that "non-programmers can do it. No code required, etc. etc." Heh. As if that ever did, or ever could work.
> "non-programmers can do it. No code required, etc. etc."
translating for the Palantir's business - "a DHS/FBI/CIA analyst can dot it. No code required" Easy scraping websites for terms like "subversion", "Arab", "airplane" (cue starting scenes from Harold & Kumar 2) ...
translating for the Palantir's business - "a DHS/FBI/CIA analyst can dot it. No code required" Easy scraping websites for terms like "subversion", "Arab", "airplane" (cue starting scenes from Harold & Kumar 2) ...
We, as programmers, are called in to solve the "complex use-cases." So we tend to divide the world into "things that need a novel program written for them" and "trivial problems."
You know what else has a visual interface? Excel. It works great! For "trivial problems." It gets stretched all the time into places where programming would have been faster, but for the "trivial problems", just being able to have a domain-expert non-programmer use Excel to implement the solution directly, is a lot faster and simpler than calling in a programmer and attempting to fully specify the problem to them.
You know what else has a visual interface? Excel. It works great! For "trivial problems." It gets stretched all the time into places where programming would have been faster, but for the "trivial problems", just being able to have a domain-expert non-programmer use Excel to implement the solution directly, is a lot faster and simpler than calling in a programmer and attempting to fully specify the problem to them.
I'll be the first to agree that Excel "actually works" and "solves a real problem". But if you had actually used Kimono, you'd know that it was one of those tools that was just finicky enough to be unusable by "regular people". And just weak enough to not be productive for programmers.
Laziness is a secret form of technical debt.
For example, I hit an issue when using Kimono where it refused to accept a given regular expression as valid, even after trying testing many variations which passed a regex checker. If I had just started with Python/BeautifulSoup, I could have completed the entire scraper in that time.
For example, I hit an issue when using Kimono where it refused to accept a given regular expression as valid, even after trying testing many variations which passed a regex checker. If I had just started with Python/BeautifulSoup, I could have completed the entire scraper in that time.
While as I said, I don't know anything about Kimono, I think that statement requires qualification. If the "lazy" solution allows me to iterate faster on solving the problem, then even if I need to replace it later when my requirements are more complex but better defined, it's potentially a net win (even if you have to reimplement some of the same pieces yourself later).
See x1024's comment above. (best case scenario is that it's not significantly faster)
Interesting you say this, since it seems like DiffBot is doing something similar and received positive media recently: http://techcrunch.com/2016/02/11/diffbot-aims-to-build-the-i...
DiffBot appears to be different (pun intended). Even when implementing your own scraper, you still have to understand the inherent structure/organization of the webpage. DiffBot appears to do that automatically, which could be a timesaver if it works reliably without significant data fidelity issues.
I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.
I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.
Hi Max,
All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3...
If you wanted just a list of submissions and score, you could use our custom API toolkit(similar to Kimono): http://www.diffbot.com/products/custom/
All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3...
If you wanted just a list of submissions and score, you could use our custom API toolkit(similar to Kimono): http://www.diffbot.com/products/custom/
Just throwing this out there in case you happen to know....I've always wanted this ability to scrape stock discussion forums. Just one example (there are probably 3 to 10 forums I'd like to scrape eventually) is:
http://www.stockhouse.com/companies/bullboard/t.y/yellow-med...
Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?
http://www.stockhouse.com/companies/bullboard/t.y/yellow-med...
Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?
Yes, that's something we definitely support – you'll want to take a look at our crawler: http://www.diffbot.com/products/crawlbot/
If you'd like me to set you up with a demo, feel free to email me: [email protected]
If you'd like me to set you up with a demo, feel free to email me: [email protected]
if you run HN front page through import.io's magic, it works pretty well:
https://magic.import.io/?site=https:%2F%2Fnews.ycombinator.c...
Another one for http://ourincrediblejourney.tumblr.com/
Man, this is disappointing. No way to trust any cloud offering, more sooner than later the service is discontinued.
I agree. I actually referred people to Kimono. I'm so sorry I did. This is unacceptable behavior in this day and age. I mean seriously? Servers are cheap. Keep the thing up for existing users. Don't add new features. Don't invest any more time in the thing. But don't pull the rug from under people's feet. That's just bad form. If this is who Palantir acquires or acquihires, then they have some really bad judgement too. It reflects poorly on Palantir to not care about commitment and responsibility.
It reminds me of a PC World magazine that I had. It took me over a year and a half to get around to reading it. It had an article about the 50 best websites or something like that. By the time I got around to reading it, at least 1/3 of the sites were discontinued.
I was just thinking the same. I rely on Kimono for a few of my sites, so I'm looking around at what I could use instead. The first I thought of is http://import.io/, http://cloudscrape.com, or https://phantomjscloud.com/site/index.html, but I fear the same might happen to whichever I pick.
Does anyone have any suggestions on how to run something similar on my own server?
Does anyone have any suggestions on how to run something similar on my own server?
You can use portia[1]. You can try out a hosted version at scrapinghub[2] or you can host your own. It's a web based spider creator like kimonolabs but the spiders it creates are just scrapy spiders.
If you need to do something that the UI can't handle it's not a problem, you can extend it with a little python.
[1] https://github.com/scrapinghub/portia [2] https://scrapinghub.com
[1] https://github.com/scrapinghub/portia [2] https://scrapinghub.com
Download the Kimono desktop tool for Mac and unzip it. It is nothing but a nodejs based web application.
Perhaps a viable alternative to using cloud services would be roll-your-own OSS packages
[deleted]
"access to the world’s most important data problems"
I'm not sure how to put this, but I've used Palantir's products in a couple different warzones. I can't imagine they're not pointing those tools at everyday Americans. In fact I'm certain of it (just read FedBizOpps postings from the DHS and DoD). I'm sure most of the people working there have no idea, but I'm also sure there are some truly bad people working for that company. Man, we really need to regulate this a lot better.
I'm not sure how to put this, but I've used Palantir's products in a couple different warzones. I can't imagine they're not pointing those tools at everyday Americans. In fact I'm certain of it (just read FedBizOpps postings from the DHS and DoD). I'm sure most of the people working there have no idea, but I'm also sure there are some truly bad people working for that company. Man, we really need to regulate this a lot better.
> I can't imagine they're not pointing those tools at everyday Americans.
http://www.dhs.gov/state-and-major-urban-area-fusion-centers
Powered by Palantir, basically it is real Big Brother brain network where all the information like lic. plate readers, video cam feeds, cell phone data, Internet data and everything else is "fused" together. One of the immediate resulting products of it is generation of Suspicious Activity Reports. Hoover (and Beria) would die of envy.
http://www.dhs.gov/state-and-major-urban-area-fusion-centers
Powered by Palantir, basically it is real Big Brother brain network where all the information like lic. plate readers, video cam feeds, cell phone data, Internet data and everything else is "fused" together. One of the immediate resulting products of it is generation of Suspicious Activity Reports. Hoover (and Beria) would die of envy.
Yeah, I'm not sure how they can drop such an unqualified statement like that. Who says Palantir has the world's most important data problems?
Could someone give an example?
Why is access to these most important problems limited to Palantir, or otherwise not accessible to most people (as implied by this shutdown notice)?
Could someone give an example?
Why is access to these most important problems limited to Palantir, or otherwise not accessible to most people (as implied by this shutdown notice)?
Disappointing to see just 2 weeks notice for their existing users.
* shameless plug * For longer than most such tools, we've been offering what our customers call the "easiest scraper of 'em all": https://feedity.com Try it out and see if it works for your needs.
* shameless plug * For longer than most such tools, we've been offering what our customers call the "easiest scraper of 'em all": https://feedity.com Try it out and see if it works for your needs.
From the FAQ:
https://www.kimonolabs.com/faq#faq-303
> Will Palantir (or anyone else) have access to my personal data?
No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016.
Uh...but what about the scraped data? I don't think I've used Kimono since it was first announced on HN as a side project...but I'm assuming that you got to save scraped data semi-privately with Kimono? Is that considered "user data"? Kind of a vague term, and one I usually associate with data of the personal information variety. I'm sure someone at Palantir can find some interesting meta-insights from analyzing the data collected by the kind of people (data scientists, etc) who would use Kimono heavily.
https://www.kimonolabs.com/faq#faq-303
> Will Palantir (or anyone else) have access to my personal data?
No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016.
Uh...but what about the scraped data? I don't think I've used Kimono since it was first announced on HN as a side project...but I'm assuming that you got to save scraped data semi-privately with Kimono? Is that considered "user data"? Kind of a vague term, and one I usually associate with data of the personal information variety. I'm sure someone at Palantir can find some interesting meta-insights from analyzing the data collected by the kind of people (data scientists, etc) who would use Kimono heavily.
Palantir is one of those organizations that makes me throw up a little in the back of my throat whenever I hear about them. They are sort of like the Nickelback of companies.
I'm no fan of Nickleback's but I don't think they deserve to be compared to Palantir. If I was a Kimono dev I would certainly be leaving now.
Is it because what they do (helping governments snoop on social media) or are they actually a bad place to work?
You can host scrapers you need to migrate on morph.io ( https://morph.io ) a free platform for hosting and running scrapers, storing your data and providing access to it in a range of formats (JSON, CSV, SQLite). It’s free, open source, supports a bunch of languages and is run by an Australian non-profit, where I happen to work ;-).
Here's a tutorial all about running a scraper on morph.io https://www.openaustraliafoundation.org.au/2015/10/13/ruby-w...
Here's a tutorial all about running a scraper on morph.io https://www.openaustraliafoundation.org.au/2015/10/13/ruby-w...
It's gross they're giving their customers 2 weeks to "GTFO".
Disappointing. Was my goto service as it was a joy to use and always seemed to work. Wish they would follow the Parse lead and release something open source. Desktop doesn't really cut it for API use...
I loved Kimono! It had the best interface and most flexibility for this type of service. A bittersweet congratulations to the team.
Evaluated Kimono for a tool I was building, decided to roll my own with Scrapy instead. So happy right now that I did...
I'm always disappointed when I read of Palantir acquihires.
Technically talented employees receive a pittance and are sold into the Palantir machine, a morally unscrupulous high-tech consulting shop that takes in talented engineers and churns out one-off software black boxes designed to lock in customers and extract a long-term tax.
Technically talented employees receive a pittance and are sold into the Palantir machine, a morally unscrupulous high-tech consulting shop that takes in talented engineers and churns out one-off software black boxes designed to lock in customers and extract a long-term tax.
I had a hard time using this product. I was trying to use another developer's APIs but it didn't seem possible. The one positive thing about the announcement, from the FAQ:
Will Palantir (or anyone else) have access to my personal data? No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016.
https://www.kimonolabs.com/faq
Will Palantir (or anyone else) have access to my personal data? No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016.
https://www.kimonolabs.com/faq
Their "desktop tool" is complete rubbish. I have to manually click on each source to crawl it and then manually click a few buttons to save the response.
Thanks for nothing guys!
Thanks for nothing guys!
ParseHub is much better at getting websites from shitty, complicated sites. Kimono was great for something simple, but ParseHub is a more powerful and flexible web scraper. https://www.parsehub.com/
Sad to see Kimono close down. I enjoyed using it for a small project, and was waiting to see a paid version with private APIs, but it was always "coming soon"...
I don't quite get why startups are not trying to generate revenue from day one.
I don't quite get why startups are not trying to generate revenue from day one.
These people should be totally ashamed for the tool they have given. It is just nodejs wrapped inside a desktop app. They could have very well made it open sourced and offered to community.
Kimono had a beautiful chrome extension which allowed me to easily extract things quickly without needing to write any selectors / scraping. Much love to this team; best of luck!
Anyone offering a GUI solution that can export to an RSS feed like Kimono did? I know RSS is old but I have no clue how to display the other formats in wordpress. Thanks!
[deleted]
As usual with acquisitions, they have stripped any information about what their product actually is (was?) from their site.
ohhhhh man that stinks. I love Kimono and have been using for 6+ months.
Hopefully it works out okay for the Kimono team.
There is not a download function on the desktop page, what is the direct download link? https://www.kimonolabs.com/desktop
There is not a download function on the desktop page, what is the direct download link? https://www.kimonolabs.com/desktop
few other options using NodeJs - http://blog.webkid.io/nodejs-scraping-libraries/
This is just an acquihire.
Looks like we cannot schedule an API in the desktop version
We can conclude from this that they weren't doing very well, right? I mean, you don't accept an acquihire if you're doing great.
Its a two edged sword, sometimes you're doing great - but the fact they're shutting down means they're not doing great but have a team of experts in parsing that will benefit Palantir since they're likely into analysing surface and deep web data and just data in general :)
basically its like hiring a bunch of regex gods because you have regex problems on large scale
basically its like hiring a bunch of regex gods because you have regex problems on large scale
[deleted]
I wonder what it's going to look like for import.io as well, their business model is identical to kimono it seems.
edit: why the downvotes seriously.
edit: why the downvotes seriously.
Here's an upvote, because I was thinking exactly the same as someone who has used most of these "scraping" tools.
To all those badmouthing Palantir. I think they are just a hogwash. They are just pretending to be something that Fox Maulder would investigate I dont think they are. They are just helping the government spend more money on problems that are not worth solving and create new problems for citizens.
The place is likely to be very shitty to work for but by suggesting that they might be doing something that is James Bond equivalent in software world might make it sound too cool for some developers.
The place is likely to be very shitty to work for but by suggesting that they might be doing something that is James Bond equivalent in software world might make it sound too cool for some developers.
As a team, we’re very proud of the product we built, now used by over 125k developers, data scientists and businesses" ... from FAQ: "service will shut down Feb 29, 2016"
In fact, they are so proud, they're giving all but ~2 weeks notice to said 125K developers, data scientists, and businesses to migrate off the platform? That is an abysmally small amount of time and if I was a paying customer, i'd be furious.
On the other hand, congrats to Kimono team!