Our service, 80legs, will let you easily do this. We let you specify seed links, how deep you want to crawl, and control many other aspects of the crawl. By default, we control the hard bits, like redirects and spider traps, but if you want to override our default functionality you can easily insert your own code to do it.
Our default functionality will let you identify mp3 files by regex or keyword, but if you need something more sophisticated you can override that too. I'm pretty sure, based on what you've said, that you could simply put in a few parameters and start running some jobs within a few minutes of getting started with 80legs that will do exactly what you want. If not, adding custom code to 80legs is pretty simple too.
Just send us your contact info on our website (http://www.80legs.com) and mention HN and I'll make sure you get a beta invite. BTW - we're still in private beta and the service is still free for right now.
We actually have a surprising number of customers come to us at 80legs wanting to crawl google search results. I don't think most of them are trying to reverse engineer google or anything like that. Most probably just want a fast way to find relevant topics to crawl.
They are disappointed when they learn we obey robots.txt, so we have them manually do searches to pull out seed lists for their 80legs crawls. It's a pain, but there's not really a way around it within the rules.
Great question. We've actually done a lot of work on this to ensure that there isn't a problem with running the code on various people's machines.
First, Plura actually runs the processPage() function in the restricted java sandbox so there is no way to actually see any data on the user's computer or do anything bad to their computer. Also, the code goes through a short verification process before it is deployed.
For the results, we do have a reasonably sophisticated validation process as well. For someone to change results from one node, they would have to do quite a bit of work.
Our service actually allows you to push your code into the system rather than trying to pull back all of the page contents. So, you end up running your semantic analysis, image analysis, or whatever you want to do on our grid. Very specifically, you implement a processPage() function of the following form:
byte[] processPage ( String url, byte[] pageContents, Object userData); (EDIT: remove code tag that didn't work...)
We run your function on the contents of the pages/images/objects you want to analyze and give you back your results from the millions or billions of pages you want to analyze.
The results from the processPage() function are completely free form. You serialize your results into a byte array and that's what you get back (except you get it back for all of your urls).
Now, since the processPage() function is free form, you can just turn around and "return pageContents;" from your function. That will give you all of the page contents from your crawl. That's not an ideal case for us, but we can handle it. We might eventually charge a small bandwidth or storage cost for this type of usage, but we do not intend to do so for our normal use case.
The bigger charge to the customer if they try to pull back all of the contents will be their local bandwidth charge. They would need to pull all of these pages' contents to their own servers. That will cost them quite a lot of bandwidth assuming they don't have their own fat pipe.
In summary, $2/million-pages-crawled is our real price and is not just marketing.
Thanks, we will work on the wording. Just for clarity, we actually do custom crawls based on your needs. If you need to access one million pages, you tell us how to get to them and pay us $2 plus any time you spend processing those pages. You can do a generic crawl from http://dir.yahoo.com or you can give us a very customized seed list and just read those pages or crawl only a few levels deep from there. Your choice.
You certainly don't need to crawl two billion (2,000,000,000) pages per day. In fact, that's our total estimated capacity right now.
Thanks for the nice comments luckystrike. We had a great time at the Web 2.0 Expo and we've been overwhelmed by all the interest in our service. We're pretty hopeful that we've built something that people want. :)
Plura affiliates actually accept responsibility for getting the permission of their users. Plura encourages disclosure and has found that it is actually very well received by the users once it is explained. It always works our better for Plura affiliates when they disclose. To that end, Plura has actually changed it's TOS with affiliates so that they directly take responsibility for getting user acceptance.
Most Plura apps/websites give users optin/optout capabilities. Rather than anything ill-intentioned, the actual model is really that Plura gives application developers a means of offering their application at a discount (or free) to users that don't mind trading their excess computer resources for the app. For those that don't want Plura+free, the application developer can give them other options (pay, ads, whatever).
Once the users really understand it, they are almost always happy that the developer has a new means of monetization so that the developer will continue to improve the software they are using.
BTW, this all runs in a secure java sandbox where nothing can actually see the users data, disk, what programs are running, or anything else about the computer. Plura has gone to great lengths to try to sanitize the entire process and be good guys.
Hi guys. We were actually about to do an "Ask HN: Review our startup" post, but I guess someone beat us to it.
So, please review our startup. :)
We are launching the beta today to a handful of users and will be letting in more and more users over time.
One other note: We don't just offer crawling. Our model is actually to allow you to analyze the web content that you discover. Using your own custom code that you push into 80legs, you can do sophisticated text processing, image processing, look inside PDFs, etc.
Does this technology work using both images simultaneously? Or could lots of images be pre-processed individually into something that allowed a cheaper comparison?
Just to try to explain the question better, here's an example. Let's say I have 10 images and I want to find the most similar people among any pair of images. Do I need to run every pair of images (45 full comparisons) or can I pre-process the 10 images into something such that the 45 comparisons can be done in a less expensive way?
My angel group uses Angelsoft to organize the deals we receive from startups. I believe we pay Angelsoft for the privilege and get to use their tools to discuss and evaluate the deals that come our way. It is useful.
There is also tons of crap listed from other cities and groups. I guess these guys pay Angelsoft to pitch, but that part of the site is a complete waste. There is so much noise in those deals that I never look at them.
Very cool algorithm demo. How well would this compare to a very extreme JPEG compression?
Also, I'm not an image compression expert, but I wonder if you could use this technique as a pre-pass before a more standard image compression algorithm to improve the overall results. In other words, maybe subtract the image created using the polygons from the original image and then compress the result using JPEG (or something else more suitable?).
Wow, I just saw this. I was an early lender on Prosper to the tune of $15k. I "invested" it over a couple of months in about 100 different loans. I was considering it a test before investing considerably more.
Well, the test didn't go well and I've been steadily withdrawing my money over the past couple of years. At the end of the day, I expect (pending the outcome of this new development) that I'll lose about $3k total.
No, it does not have to be a game. We're just focusing our initial affiliate marketing on games because of the higher engagement.
The type of sites you mention will work well too. Send us a note through the Contact Us page on the website and we can help you figure out how much you could earn.
The apps that we're running right now control it pretty well using a sneaky trick. We can't get much computer information from java in an unsigned app, so we measure how long each sub-part of a work unit has taken and sleep an appropriate amount of time. For example, if we are trying to use 25% and a sub-work takes 100ms, we sleep for 300ms. This effectively gives us 25% usage of the available CPU. It will be no higher than that anyway - it might be lower if something else is using substantial CPU time.
As far as the fast versus slow nodes, we normalize everything based on the average computer that completes work each day. So a fast computer may earn 2X or more what a slow computer earns.
These are all very good questions and we certainly know that we're not the first company to do grid computing on PCs. :)
First, $1500/node-year (or even $200/node-year) comes to a very substantial amount of money for our target customers. These customers use 10000s of nodes and buy high-end computers and pay a ton for network, power, cooling, and support personnel.
My background is in HPC and I've faced and solved these issues many times. We have a multi-pronged approach to using the compute power:
1) We are definitely writing our own apps on top of Plura. It's a great source of compute power and we have lots of ways we want to use it ourselves.
2) We have a short list of very high impact embarrassingly parallel customers that are evaluating Plura. We intend to help our customers absorb the porting cost associated with Plura.
3) We are open to letting students and universities use Plura's excess capacity for free. We've had several nibbles at this already.
That being said, we realize Plura is not for everyone. It is for a subset of the class of embarrassingly parallel applications out there.
Good question. There really isn't anything that stops someone from reading and trying to interpret the byte code.
There is some protection in the fact that you never know what type of work unit is happening at a given time. The algorithms are typically each snippets of code instead of full applications, so someone would need to piece together quite a lot of information.
Of course, if someone is particularly concerned, they can run a jar obfuscator.
Oh, and most developers use 50% so far. We have a download app that affiliates can distribute too if they want to. We can set that to run at different numbers when idle and in use. For example, I have mine set to 100% when I'm not using the keyboard and mouse and 50% when I am using it.
Our default functionality will let you identify mp3 files by regex or keyword, but if you need something more sophisticated you can override that too. I'm pretty sure, based on what you've said, that you could simply put in a few parameters and start running some jobs within a few minutes of getting started with 80legs that will do exactly what you want. If not, adding custom code to 80legs is pretty simple too.
Just send us your contact info on our website (http://www.80legs.com) and mention HN and I'll make sure you get a beta invite. BTW - we're still in private beta and the service is still free for right now.