The joys of knowing Software – Hacking the Boat License Course

Boat Licenses

NOTE: June 17th 2016 – I received a cease and desist letter from boat-ed.com and they have fixed the way in which the script in this post circumvents the wait. I am not aware of any other way to bypass it.

Summer is fast approaching and me and a group of friends decided to take the water on some ridiculously fun jet skis. In the state of Washington, a boat license is required to operate any PWC (personal watercraft) which thankfully can be done online. As a procrastinator I put aside 30 minutes of my time the night before the excursion to do the online exam on www.boat-ed.com. I was in for a large shock when I discovered that you had to sit through all the course material on the website prior to attempting the exam; average time for completion 6-8 hours.

Time is precious

Time is precious. It’s constantly fleeting and the last thing I wanted to do was sit through 6-8 hours of instructions that I knew and are largely common sense. The final exam should be indicative of whether you know the material or not, therefore it is not necessary to force people to sit through the course (all though you can Google the answers to the exam…). As my roomate decided to sit through the course material blindly hitting next when the button becomes available as he watched tv shows on Netflix, I decided to investigate if my knowledge of Computer Science can save me!

Chrome Extensions FTW

tl;dr; Install this Chrome Extension, which will enable you to skip any State Boat License course on www.boat-ed.com

Having some prior experience to writing Chrome Extensions and knowing more than my fair share of web services, I was able to dig into their JavaScript source files and find how they were forcing the users to sit through the course material. I will leave future blog posts perhaps detailing how one might go about “hacking” websites to reverse engineer their web services and fiddle around with it, briefly the steps were:

  1. Reverse obfuscate their JavaScript
  2. Find the offending timer code
  3. Figure out how to diable the timer (was a server call
  4. Fake the code in a Chrome Extension

Update – Google Chrome HypeMachine Extension Alive and Well!

I’m still kicking

tl;dr; You can get the latest and greatest of the Chrome HypeMachine Download Extension on my Github Account.

I hope everyone hasn’t written me off as dead since the somewhat shutdown of MetaHypem. I’ve been trying to think of a new project and devoting to more time to my professional life (i.e. working at Amazon). I have however not forgotten about all of you and your insatiable desire for music.

I recently received a Pull-Request that detailed some changes I’ve been planning to make for quite a while! (props to landr0id). The changes outlined:

  1. changing to HypeMachine’s new private API (which I’ve detailed earlier)
  2. adding the new sexy HTML5 download attribute
  3. Downloaded songs are now named according to artist and title
  4. general performance improvements

What does this mean for you?

The extension is still alive and well. You can even now Left Click on the download arrow to kickoff the download. Goodbye confusing Save As… instructions!

For those looking for a replacement

Older work, still valid

A lot of people have been sending me e-mails due to the current state of MetaHypem. If you’re looking to find alterative ways to download the music I can recommend the following.

  1. Just follow the blog link for each song. The blogs let you download the music.
  2. I’ve written previously a Chrome Extension that injects a download button into the web page.e web page.

    Don’t forget you need to right-click and select Save-As however…

  3. I wrote an example Python script outlining how to download songs that you can use or help build other tools with.

MetaHypem has been temporarily shut down

The game resumes

Over the course of this weekend, my latest pet project MetaHypem has been shut down but by HypeMachine and not through us. HypeMachine seems to have added either our domain or instance running our server onto a black-list.

Here is a nice screen capture of what current data HypeMachine is returning my scraper. The MP3 Artist/Title was a friendly sting I guess.
hypemshutdown

I haven’t done much publicizing of the server and daily hits to the site was only ~40 visitors (most returning). However clearly there was a need/love for having access to the music (~50k downloads). All the blogs that HypeMachine scrapes allow users to download the music. The purpose of a music blog is to help promote bands and new music. Keeping the music locked onto their platform seems going against their goal.

Where to go from here

I’m currently investigating (alongside my partner) different ways to resolve this issue. I’d love feedback from people expressing perhaps which they would prefer or even have experience with!

  • Move the server to EC2. Perhaps they only block the IP at which I can just tear down the server to get a new IP.
  • Change the scraping of HypeMachine to be distributed and have other instances perform the scraping.
  • Release the source code to the project and let others run local instances.
  • Figure out other inventive ways!

Small Update + More to Come!

Happy New Year!

I’ve finally gotten back into development for MetaHypem (alongside Alex) and most of it has been spurred/inspired by the amount of songs downloaded! It figuratively blew my mind to have 30,000 downloads…

I’ve just released a small update that includes some fixes and 1 specific feature that has been asked about! We have some cool stuff planned for the site so I hope you keep checking it and send me feedback! We are using the feedback to drive what to work on next!

Changelog

  • 28/12/2012 – You can now search for a specific url (i.e. http://hypem.com/artist/Noosa )
  • 27/12/2012 – Added an Expire time for each song record (currently 1 hour). This is incase HypeMachine updates the download URL.
  • 27/12/2012 – Generic Bug Fixing for cases where no songs are returned from HypeMachine

As always, please send me feedback and if anyone wants to contribute (especially in terms of design/HTML/CSS contact us!)

Someone must be using the site!

Milestone Baby!

Update Since posting (~1 month) we generated 30 000 songs. Awesome!

I’ve only told few people (not that many read my blog) about Metahypem, largely because it is still a huge WIP and a learning experience. Nevertheless, the few that have been checked it out seem to be using it quite extensively.
I haven’t managed to grep the logs yet to see some interesting statistics, however we’ve recently passed the milestone of over 10 000 songs liberated.

That’s a pretty awesome number! That many times someone wanted to hear a particular song while not at their computer or over an internet connection and we aided them in that! I’ll follow up with some interesting tidbits from the request logs when I get a chance.
Thanks <3

Downloading Songs from Soundcloud

The next task

I’ve been fiddling around with HypeMachine for quite a while now even though my personal music taste has taken me elsewhere in search of music. Lately, I’ve been enjoying subscribing to individual artists on Soundcloud as a way of finding more niche electronica.

My experience so far with Soundcloud has been great. They even let you download some songs based on settings set by the artist! I however want the music while I’m on the go; especially on my jogs. I’ve now moved onto the task of seeing how I can liberate much of my beloved music from Soundcloud similarly to what I’ve accomplished with HypeMachine.

Investigation

Once again most of the information gleamed was from saving the raw HTML of a request to SoundCloud outside the browser (i.e. via python or perl). They look to be doing something similar to HypeMachine in removing certain HTML elements because I cannot find them when I view source through my browser.

We can see one of the track elements is the streamURL:

"streamUrl":"http://media.soundcloud.com/stream/WtquUHiysWgk?stream_token=H7TPl"

Node.JS Style

Grabbing the bufferTracks variable can easily be accomplished in Node.JS with the help of the jsdom. I’d simply need to build the DOM according to the HTML file and access the JavaScript variable! I’m curious if there is even an easier way…. Drop a comment if you know of one!

New Domain Name

More information to come

tl;dr; I’ve moved the download site to metahypem.com

Me and my friend Alex have been hard at work revising my original concept for hypemfinder.

Out with the old and in the new. I’ve purchased a new domain name, metahypem.com which will be the domain name for the application for now!

The site is redesigned in Node.JS and I will be providing some cool information regarding it in the near future. For now we have deployed the instance on a small server. Please don’t murder our traffic 🙂

Getting things to play nice

The move to Nginx

It’s been only one month on my new Linode machine and I’ve only ever known of Apache as a front end HTTP server. Apache sadly had to get the boot as I was preparing in deploying me and Alex Hortin’s new Node.JS version of the site to facilitate downloading songs from HypeMachine.

Node.JS comes with a complete event-driven web server however I needed a front-end server so that I can host multiple sites on my single instance (such as this blog!). Having a blocking HTTP front-end such as Apache proxy requests to Node? Sounds retarded.

The whole appeal of Node.JS is that it’s event-driven and non-blocking which hopefully results in really speedy responses. Luckily Nginx (pronounced EngineX) is also an event-driven web server and is a perfect match as a proxy server for a Node.JS application.

Apache is like Microsoft Word, it has a million options but you only need six. Nginx does those six things, and it does five of them 50 times faster than Apache. — Chris Lea

Continue reading “Getting things to play nice”

Python Download Script Updated

I’m so slow

I’m been so caught up in my new website project (downloading songs from HypeMachine as well), that I’ve really left the extension and python script fall behind.
In an attempt to try and rectify this, I’ve made the first step by fixing/updating the python script. I’ve already documented many of the changes that needed to occur in my previous blog post but now here is the full source for those who are curious!

You can download the script here: https://github.com/fzakaria/HypeScript

I’ve substantially cleaned up the code and removed the sqlite (database) aspect to it. This has resulted in a nice clean barebone downloader which does the one job it needs to do; download’em songs!. I’m always looking for advice, comments and feedback.