Building a scraper for recreation.gov

The start of a new project

A friend has recently asked if I can look into building a tool / site to scrape https://recreation.gov — with a end-goal of building a system to automatically reserve for a desired permit.

This piqued my interest and lets take a look at what I can do! At a high level I imagined building:
register desired site -> continously scrape -> reserve -> notify via text

Looks like a good chance to put together some interesting technologies: web-framework (django?) & twilio to send notifications

Alternatives

Before beginning any project, I take a look at the current space and see if there are any current open source alternatives or even a paid platform to leverage.

I found the following:

I could not find a paid service and the OSS options seemed very difficult for non technical people to use.

Can I haz API?

Browsing online — I was ecstatic when I came across ridb.recreation.gov which is a REST API for the recreation.gov website — unfortunately it doesn’t let you perform reservations and I couldn’t decipher yet how to link them to the reservation portion. Perhaps it might be leveraged in the future!

Time to use our favorite reverse engineering tools: wireshark & Charles — I ended up using Charles specifically because I find it easier to setup as a man-in-the-middle HTTPS proxy.

You can follow the simple guide on how to setup Charles as a HTTPS proxy here

Here is the raw request from Charles when searching locations matching whitney at https://www.recreation.gov/unifSearch.do
(unimportant parts stripped out)


POST /unifSearch.do HTTP/1.1
Host: www.recreation.gov
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:60.0) Gecko/20100101 Firefox/60.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.recreation.gov/unifSearch.do
Content-Type: application/x-www-form-urlencoded
Content-Length: 275
Connection: keep-alive
Upgrade-Insecure-Requests: 1

currentMaximumWindow=12&locationCriteria=whitney&interest=&locationPosition=&selectedLocationCriteria=&resetAllFilters=true&filtersFormSubmitted=false&glocIndex=0&googleLocations=Whitney+Place+Northwest%2C+Seattle%2C+WA%2C+USA%7C-122.39853319999997%7C47.6974492%7C%7CLOCALITY

The important part is that it is x-www-form-urlencoded with locationCriteria=whitney.

The response is HTML however we can use various tools to strip out the desired list.