Brad Lucas

Programming, Clojure and other interests

Downloading the TTD API documentation site

May 17, 2019

Recently I've been working with The Trade Desk API whose documentation is hosted on a password protected web site. The site requires a username and password to access. These you get from your TTD representative.

Now, the site is fine but very dense. In a few cases I wanted to search the entire site and having a copy of the site locally would make this possible.

How to download the site is described in this post.


The first attempt to download using wget didn't work because the site doesn't support access with Basic Authentication. This meant that passing your username and password as parameters to wget will not work.

Notice that when logging in that the site asks for the username and then prompts for the password in two steps. This means another technique will be needed.

One idea would be to build a crawler that knew how to navigate the login dialogs and then copy the site's files.

Another idea, is to split the work between the login and the file copying between the browser and wget. With this idea we login using the browser to have the site set it's cookies with the browser. Then, we borrow the cookie values and have wget use them to copy the files.


Continue reading →

Run A Local Amazon Linux 2 Instance

November 10, 2018

Amazon Linux 2 is available as virtual machine images for development on-premises. If you are using Amazon Linux 2 EC2 instances it is a compelling idea to have a local version of the same for testing and development.

But how? The follow details a path to running Amazon Linux 2 locally in a VirtualBox VM.


Before continuing it's assumed that you've worked with VirtualBox before. To continue realize that Virtual Box creates a virtual machine inside a named directory and uses an associated virtual disk image. This disk image as you can imagine is the virtual machine's hard disk.

The virtual machines (VMs) are stored in a directory defaultly named VirtualBoxVMs under your home directory. When you create a VM from scratch the associated virtual disk image file (VDI) will be in the same directory. For the plan detailed here we will want to keep our VDI file in a separate directory. The recommendation to keep the VDI separate allows for the configuration files and script to be located with the VDI. It also allows you to easily delete your VM and rebuild using a new copy of the VDI file.

Download Amazon Linux 2 VDI File

Amazon Linux 2 is a newish version of Linux available on AWS when setting up EC2 instances. To make a version useful inside of VirtualBox, Amazon has release a VDI file. This VDI file will be the virtual disk image for your local virtual machine after you do a bit of configuring.

Because you are likely to create multiple VMs or may want to re-create your local VM to replicate tests and experiments I suggest you store your original Amazon VDI in a known directory and use copies for each VM you create.


Continue reading →

Updating Your Pip Requirements File

November 9, 2018

While revisiting an old Python project you may come to realize that your requirements.txt file contains out of date versions. You could go through and research each library and get it's latest version number but may be discouraged if there too many.

The following is a technique to update the entire file without having to take too much time.

This technique is good if your goal to to update each library to it's most recent first.

Step 1 Change the == to >= for each entry

This will unpin the reference to a specific version

Old requirements.txt


Edited requirements.txt


Continue reading →

List Globally Installed Packages With npm

November 3, 2018

You may have installed npm packages globally. To find out try the following command.

$ npm list -g --depth 0

Continue reading →

Learning Ring Next Steps

May 22, 2018

Previously, I wrote about Clojure and Ring in an introductory post where a small debugging application called Echo was developed. If you haven't looked at that post and are looking for an introduction to Ring I suggest you do and then come back here.

For this post, I've created another sample application called ring-next which contains routines to demonstrate a number of concepts in Ring. These are:

  • Middleware
  • Responses
  • Parameters
  • Cookies
  • File Uploads
  • Routes

For each of these, I'll show a few code snippets and then explain what is going on. The repo for this project is linked at the end of this post.


Handlers are the functions that make up your application. They accept responses and return responses. Middleware is a set of functions that add additional functionality to handlers. They are higher-level functions in that they accept a handler as a parameter and return a value which is a new handler function that will call the original handler function. This pattern allows you to wrap your handler in a number of middleware functions to build up layers of functionality.

To give you an idea of middleware functions here are three from ring-next. There are meant to show the idea.

The first simply add a new field with a value to all requests. The second prints each request to STDOUT. When you lein run the application from a terminal window you'll see requests printed to the output. The last was built during the development of ring-next because I found that Chrome was during some requests looking for the favicon.ico file. This was cluttering up my debugging with extra requests being printed. The wrap-ignore-favicon-request function solved the problem by effectively neutering the call. By having this function wrapped before all the mothers ensured a 404 was returned before any other middleware and handlers were called.

Continue reading →

Learning Ring And Building Echo

May 18, 2018

When you come to Clojure and want to build a web app you'll discover Ring almost immediately. Even if you use another library you are likely to find Ring in use.

What is Ring?

As stated in the Ring repository, it is a library that abstracts the details of HTTP into a simple API. It does this by turning HTTP requests into Clojure maps which can be inspected and modified by a handler which returns an HTTP response. The handlers are Clojure functions that you create. You are also responsible for creating the response. Ring connects your handler with the underlying web server and is responsible for taking the requests and calling your handler with the request map.

If you've had experience with Java Servlets you'll notice a pattern here but will quickly see how much simpler this is here.


Requests typically come from web browsers and can have a number of fields. Requests also have different types (GET, POST, etc), a unique URI with a query string, and message body. Ring takes all of this information and converts into a Clojure map.

Here is an example of a request map generated by Ring for a request to http://localhost:3000.

{:ssl-client-cert nil,
 :protocol "HTTP/1.1",
 :remote-addr "0:0:0:0:0:0:0:1",
 {"cache-control" "max-age=0",
  "upgrade-insecure-requests" "1",
  "connection" "keep-alive",
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36",
  "host" "localhost:3000",
  "accept-encoding" "gzip, deflate, br",
  "accept-language" "en-US,en;q=0.9"},
 :server-port 3000,
 :content-length nil,
 :content-type nil,
 :character-encoding nil,
 :uri "/",
 :server-name "localhost",
 :query-string nil,
 :body "",
 :scheme :http,
 :request-method :get}

How did I get this map so I could show it here? From Echo, which is the application that I'll describe next.

Continue reading →

Unzip File If Necessary

May 17, 2018

Ran into a situation where an external vendor started sending csv files gzipped. The trouble was they started zipping only larger files. Since the system that ingested the files would break on a gz file I needed a routine to unzip the gz files and leave the non-zipped files alone.

    def unzip_if_necessary(self, filename):
        print "unzip_if_necessary: " + filename
        f = open(filename)

        # Read magic number (the first 2 bytes) and rewind.                                                                                                                                                                               
        magic_number =

        if magic_number == '\x1f\x8b':
            print "gzip file"
            data = ''
            # with  gzip.GzipFile(fileobj=f) as f:                                                                                                                                                                                        
            with, 'rb') as f:
                data =
            print "Writing to " + filename
            with open(filename, "w") as f:
            print "csv file"


Continue reading →

Download All Files From Url

January 12, 2018

Use wget to download a web page and all associated files.

$ wget -r -np -k

If you don't have wget and you are using a Mac use brew.

$ brew install wget

Continue reading →

Advent Of Code

December 5, 2017

After taking a month off from writing blog posts I've just found something interesting to point out. The other day I found this "Advent Calendar" idea for programmers. Each day you solve a puzzle or two to unlock the day. With twenty five days to complete it looks like an interesting activity for the month of December.

If you are curious a bunch of Clojurians are listing their solutions repos here:

My solutions are at:

Continue reading →

Gpg Signing Failed Inappropriate Ioctl For Device

November 5, 2017


gpg: signing failed: Inappropriate ioctl for device

To fix:

$ GPG_TTY=$(tty)
$ export GPG_TTY

You will notice a popup after this and you'll be able to enter your passphrase properly.

Continue reading →

Clojure Ads.txt Crawler Reporting Site

October 19, 2017

Continuing with the Ads.txt crawler has lead to the idea to store the crawler results in a database and have them available from a web site. This post introduces the first pass as such a site with the source code available in the following repository:

Ads.txt Review

As a quick review the Ads.txt standard is one where publishers can host a simple text file with the names of authorized ad networks that have permission to sell the publisher's inventory. There is a reference Python crawler for such files and I've built a crawler in Clojure as an alternative. See this link for a series of posts about the Ads.txt specification and the development of the crawler. The crawler project is here.

Crawler Library

The crawler previously developed has a command line interface but can be easily used as a library from another Clojure application. To facilitate this I've pushed a version of the crawler to It's listing is:

Luminus Framework

For a batteries included framework to build a Clojure web application I highly recommend Luminus. Start with the tutorial and get that working first. Then I'd suggest repeating the excercise with the database you are ultimately going to us. For my example I built the Guestbook with a local Postgres instance. Then, I knew I was going to deploy to Heroku so moved the app to Heroku as another excercise.

Continue reading →

Heroku Keep Alive

October 18, 2017

Running a free Heroku app which falls asleep? You can use a service to ping it periodically to keep it alive.

Here is one:


Continue reading →

Coin Market Cap Update

October 17, 2017

So, it looks like the CoinMarketCap has changed their site a bit and it breaks the script I wrote back in July. Many thanks to cool007zqw for pointing this out by adding an issue on the repo.

On investigation it looks like CoinMarketCap changed the url to the page which the script was developed for.

The new page is as follows:

This does show the fragility of building scripts to pull pages from web sites. I do see that CoinMarketCap has an API over at so that may be the way forward to build something more robust.


Continue reading →

Heroku Pushing A Non-Master Local Branch

October 16, 2017

Heroku will only deploy code pushed to the master branch of your heroku remote. Knowing this as the only method to push code to your remote is limiting if you want to deploy from a feature branch.

Luckily, Heroku does support pushing a non-master branch with simple change of syntax.

$ git push heroku feature/my-cool-feature:master

Simply tack on the :master and you can push your feature branch or any other branch for that matter.

Continue reading →

Upgrading Ethereum Geth On Linux

October 15, 2017

If you've found this post and are rushing to upgrade a previously installed version of geth on Linux before the hard fork on October 16th here are a quick set of instructions.

The instructions assume you've built your geth from source. If that is the case the following should be helpful.

Stopping Geth

Make sure to stop your current geth instance. I used the following command:

$ kill -INT $(pgrep geth)

Rebuilding From Source

Find where you cloned the go-ethereum repo. It will be a directory called "go-ethereum".

Enter the following commands:

$ cd go-ethereum
$ git fetch
$ git pull
$ make geth

Continue reading →