HackerTrans
TopNewTrendsCommentsPastAskShowJobs

owor

no profile record

Submissions

Why we've turned Rarchy into a Social Enterprise

rarchy.com
2 points·by owor·hace 4 años·0 comments

Show HN: Generate a free visual sitemap by crawling any website

rarchy.com
65 points·by owor·hace 5 años·27 comments

comments

owor
·hace 5 años·discuss
We're currently working on scaling up to 25,000+ pages, hopefully sometime in the next 12 months!
owor
·hace 5 años·discuss
Noted, we'll look into returning more descriptive messaging for common errors.
owor
·hace 5 años·discuss
Hey Tepix - head to https://rarchy.com/sitemaps/visual-sitemap-generator and enter the website you wish to crawl in the left-hand side input.

This will crawl the site & generate a visual sitemap without requiring you to signup for an account.
owor
·hace 5 años·discuss
Thanks for your feedback dmje.

We've had similar discussions with customers in the past - we've found the vast majority (90%) are happy to pay monthly, and the remainder are very vocally against.

We are looking to accomodate both sides, and the current plan is to offer a subscription 'pause', where all of your existing data is stored for free, and you can unpause to edit/create new projects.

It's still being worked out, but hopefully this will be a happy medium ground in the future.
owor
·hace 5 años·discuss
We use an auto-throttling algorithm for the crawler to ensure we are a 'polite' crawler!

We also send a unique user-agent with every request, so any host can easily block us if required - luckily, this has never happened so far.
owor
·hace 5 años·discuss
Yep - Screaming Frog (desktop tool) has a limit of 500 pages in the free version, which is probably enough for most smaller-sized websites.

In terms of web-based tools, VisualSitemaps.com also has a freemium option with a 50 page limit.
owor
·hace 5 años·discuss
Watch out, Google!
owor
·hace 5 años·discuss
Thanks for the feedback.

We have looked into developing the crawler for SPAs, but haven't managed to create a robust enough crawler just yet.. we'll get there one day!
owor
·hace 5 años·discuss
We do offer an XML sitemap importer if you're in a hurry, but the free generator is always from a fresh crawl :)

Thanks for letting us know about the pricing page on mobile!