Banner
Views: 695,413,031
Time:
5 users online: Higsby, juzcook, mte_95, RichHobos, westslasher2 - Guests: 21 - Bots: 184Users: 36,368 (1,518 active)
Latest: AmericanLeatherJackets
Tip: Use palettes that don't burn the eyes. A blazing bright yellow and a night-black are not pleasant.Not logged in.
Question about scraping
Forum Index - Valley of Bowser - Site Questions, Feedback & Bugs - Question about scraping
Pages: « 1 »
Hi!

I'm pretty new here and haven't played a lot of ROM hacks but I would like to ask permission before I do this.

Since there is no 'the one' directory on your site containing all ROM hacks published. I was wondering whether you are allowed to scrape the site to download every ROM hack created.

I'm asking this, since I'm making a console atm and it would be fun to have a bunch of ROM hacks, you can try out, easily, without having to download them one by one.

Of course the scraper, in this case, would include a wait timer, to not overload the site with a *beep* ton of HTTP requests at the same time.

Sorry for the weird question!
:)
I'm not directly responsible for the server but I personally don't see much of a problem with it. We usually tend to be pretty lenient with scripts and bots which scrape the site as long as they don't cause too much strain on the server - even if they did, I assume Cloudflare would step in before anything too bad happens. Anyhow, a script which leisurely traverses the hack section is unlikely to cause problems given the more intensive bots we already have.

We recently started discouraging "HTML blenders" so we wouldn't have to worry about changing markup, but it won't be a problem if you only go through the section once. If you plan on making your script more persistent (so you get any newly released hacks), consider getting in touch with me or another developer for a more future-proof solution.
Thanks for the quick reply!

For now, I'm only looking to download every hack created so far, since I don't really have a server or anything to run scripts on atm.

And it's mostly only to automate the tedious beginning process of getting all hacks downloaded, since I won't mind adding 1 or 2 every few days manually.

If I get a "taste for more" or want to make my script more persistent on your site, I'll get in contact with you first
Zandro also operates an SMW Database that hosts basically every SMW romhack in existence going back years and years. The main landing page for the site does request that you get in contact with him before doing a mass download, so be sure to do that if you want to mass download things from him. Otherwise you can just pick and choose from a functionally endless supply of hacks to play.
Pages: « 1 »
Forum Index - Valley of Bowser - Site Questions, Feedback & Bugs - Question about scraping

The purpose of this site is not to distribute copyrighted material, but to honor one of our favourite games.

Copyright © 2005 - 2018 - SMW Central
Legal Information - Privacy Policy - Link To Us


Total queries: 23

Menu

Follow Us On

  • Facebook
  • Twitter
  • YouTube

Affiliates

  • Talkhaus
  • SMBX Community
  • GTx0
  • Super Luigi Bros
  • ROMhacking.net
  • MFGG
  • Gaming Reinvented