HackerTrans
TopNewTrendsCommentsPastAskShowJobs

owor

no profile record

Submissions

Why we've turned Rarchy into a Social Enterprise

rarchy.com
2 points·by owor·4 lata temu·0 comments

Show HN: Generate a free visual sitemap by crawling any website

rarchy.com
65 points·by owor·5 lat temu·27 comments

comments

owor
·5 lat temu·discuss
We're currently working on scaling up to 25,000+ pages, hopefully sometime in the next 12 months!
owor
·5 lat temu·discuss
Noted, we'll look into returning more descriptive messaging for common errors.
owor
·5 lat temu·discuss
Hey Tepix - head to https://rarchy.com/sitemaps/visual-sitemap-generator and enter the website you wish to crawl in the left-hand side input.

This will crawl the site & generate a visual sitemap without requiring you to signup for an account.
owor
·5 lat temu·discuss
Thanks for your feedback dmje.

We've had similar discussions with customers in the past - we've found the vast majority (90%) are happy to pay monthly, and the remainder are very vocally against.

We are looking to accomodate both sides, and the current plan is to offer a subscription 'pause', where all of your existing data is stored for free, and you can unpause to edit/create new projects.

It's still being worked out, but hopefully this will be a happy medium ground in the future.
owor
·5 lat temu·discuss
We use an auto-throttling algorithm for the crawler to ensure we are a 'polite' crawler!

We also send a unique user-agent with every request, so any host can easily block us if required - luckily, this has never happened so far.
owor
·5 lat temu·discuss
Yep - Screaming Frog (desktop tool) has a limit of 500 pages in the free version, which is probably enough for most smaller-sized websites.

In terms of web-based tools, VisualSitemaps.com also has a freemium option with a 50 page limit.
owor
·5 lat temu·discuss
Watch out, Google!
owor
·5 lat temu·discuss
Thanks for the feedback.

We have looked into developing the crawler for SPAs, but haven't managed to create a robust enough crawler just yet.. we'll get there one day!
owor
·5 lat temu·discuss
We do offer an XML sitemap importer if you're in a hurry, but the free generator is always from a fresh crawl :)

Thanks for letting us know about the pricing page on mobile!