Deploy To Clojars gpg

October 10, 2017

I posted a version of ads-txt-crawler to Clojars the other day and ran into a bit of trouble as I was on a new machine.

The command "lein deploy clojars" returned the following error.

Could not sign /Users/brad/work/github/ads-txt-crawler/target/ads-txt-crawler-0.0.6.jar
Cannot run program "gpg": error=20, Not a directory

See `lein help gpg` for how to set up gpg.

Being on a Mac I used to install gpg

brew install gpg
brew insall gpg-agent

Some other interesting links found related gpg and releasing to Clojars

Continue reading →

Increase Your Macbook Pro's Display Resolution

October 9, 2017

It turns out your Retina MacBook Pro 15" can run beyond the resolutions offered in it's System Preferences. In researching this fact I found that there are a number of third-party utilities that can be used to go beyond your Mac's offered resolutions.

The following utility from avibrazil is a good one and I recommend it. I've been using it for a few days and it works flawlessly with my MacBook as well as an external display.

The first link is to the project itself and the second link is the installable binary dmg file.

Continue reading →

Remove Siri From Touch Bar On A New Macbook Pro

October 8, 2017

I don't use Siri and on a new MacBook I hit the Siri button far too often as it is just about the Delete key. Here is how to remove the Siri button from the Touch Bar

  • System Preferences
  • Keyboard
  • Customize Control Strip

Notice your Control Strip will be wiggling Hold the Siri button and notice the Done button change to a Trash Can Drage and drop the Siri button to the Trash

Click Done

Continue reading →

Auto Increment Key In Postgresql

October 7, 2017

In MySql and MariaDB you can create a unique identity value for rows with the AUTO_INCREMENT attribute. PostgreSQL has the same idea with the SERIAL attribute.

 name VARCHAR(254) UNIQUE,

The example also shows how to have a column which stores the time when the record was created.

Continue reading →

Install PostgreSQL On A Mac Using Brew

October 6, 2017

To install Postgres on a Mac you can use brew.

$ brew update
$ brew install postgres

To create a database for yourself and login.

$ createdb `whoami`
$ psql

To drop a database.


Continue reading →

Clojure Ads Txt Crawler Saving To A SQLite Database

October 5, 2017

SQLite is a small self contained SQL database engine that reads and writes to a normal file. It is a good first pass for adding database storage to your application. It's easy to use and with a single file to deal with offers no real issues. Being a SQL database though means you won't be far off if you decide to up your database requirements to Postgres or some other database system.

My recent project of building a Clojure Ads.txt file crawler is a good target for SQLite support for a couple reasons. One is that the project is simple and works just fine writing to result files but having a database would be good in the long term especially as the project grows. Second is that the original Python Ads.txt file crawler defaulted to writing to a SQLite database. I figure the Clojure project should at the very least be able to do the same.

The results of adding SQLite are in the repository if you are curious.


  1. Model the data and create a database table schema
  2. Add the required libraries for SQLite to our project
  3. Support optionally writing to the database
  4. Support the writing to the database

Model the data

Since we'll be storing the output from the crawler we'll need to create a table that will support this data. We can start with the sql over in the project and tweak it a bit.


       SITE_DOMAIN                  TEXT    NOT NULL,
       EXCHANGE_DOMAIN              TEXT    NOT NULL,
       SELLER_ACCOUNT_ID            TEXT    NOT NULL,
       ACCOUNT_TYPE                 TEXT    NOT NULL,
       TAG_ID                       TEXT    NOT NULL,
       ENTRY_COMMENT                TEXT    NOT NULL,
       UPDATED                      DATE    DEFAULT (datetime('now','localtime')),


Continue reading →

Git Stash Pop Conflict

October 4, 2017

It is true that creating branches in Git are easy and cheap so you don't really need to git stash something. But, sometimes you don't want to save something to a branch because you really want it to be a temporary thing so you stash it.

Sometimes though you'll stash something then merge your feature branch and when you git pop conflict the stash back you'll have a conflict.

You'll see "unmerged paths" which is your clue to this situation.

What to do?

I've found that one path to take is to reset the file to unstage it, fix the conflict then add it back.

$ git reset HEAD [file(s)]

This does, if you notice, leave the stash still around.

When you are ready to get rid of it you can clear it.

$ git stash clear

Continue reading →

Ads Txt Top 100 Domain Results

October 3, 2017

The Clojure Ads.txt crawler is working well. Using the top 100 domains list from the crawler creates the following results:

Domains with Ads.txt files

Domains without Ads.txt files


The complete results file is available for inspection at the following url.

If you want to see all the generated files see this repo

Continue reading →

Optics Game

October 2, 2017

The other day at lunch, I heard this interesting phrase, "It's an optics game". It was spoken by a friend I was having lunch with. He was detailing his current work environment which is less than positive. The story went like a lot of stories I hear from people in the same company as my friend, in that things are not positive. The company appears to be failing, a lot of people have left, no raises, etc.

My friend's boss was recently working on a series of marketing messages and powerpoint presentations. What made this odd to my friend was that my friend is in BizOps and his manager is in charge of the entire department. Why does BizOps need to market itself was my friends' first thought. Sensing this oddness, my friend asked the boss why he was doing the presentations and got back the answer, "It's an optics game".

Continue reading →

Updated Clojure Ads Txt Crawler

October 1, 2017

Work continues on the Ads.txt crawler with a focus on error handling and reporting. A new version (0.0.2) is not available and it works well against the 'Top 100 domains' file.

One particular issue was found with http-kit. It seems to have difficulty with sites that are SNI-enabled. When such a site is encountered you need a SNI-aware client.

In my top 100 list the url causes the following error:

Error: Received fatal alert: handshake_failure for

This can be overcome by configuring http-kit.

(defn sni-configure
  [^SSLEngine ssl-engine ^URI uri]
  (let [^SSLParameters ssl-params (.getSSLParameters ssl-engine)]
    (.setServerNames ssl-params [(SNIHostName. (.getHost uri))])
    (.setSSLParameters ssl-engine ssl-params)))

(def client (http/make-client {:ssl-configurer sni-configure}))

To use this you pass the client in as one of the options like:

@(http/get url {:client client})

See for a specific example. Note that I've left my old routine in for now to show what was in use in the previous release. Also, not that I've decided to use the SNI configuration only when there is an error.

Continue reading →

A Clojure Ads Txt Crawler

September 30, 2017

I recently discovered the Ads.txt specification and the IAB Tech Python crawler. Being more interested in Clojure I decided last weekend to write a crawler for Ads.txt files in Clojure. The first pass is available at the following repo on the release/0.0.1 branch.

Currently as of this writing, the 0.0.1 version supports passing a target domain list to have the sites crawled, their content parsed and output written in comma delimited format to STDOUT. The Python crawler from IAB saves it's output to a SQLite database. I decided that would be a feature for a later release. With my current version it is simple enough to pipe the output to another program or file for subsequent processing.

If you find this post at a later date than the release/0.0.1 branch feel free to investigate any progress I've made.



This file is concerned with reading the target-domains file passed in via the -t option. The goal is to read the file and remove blank and commented out lines then to collect the remaining lines and reduce each url or domain to it's core domain name. Having each domain reduced to just it's component domain name will make processing them simpler.

See the read-domain-file function for an example of how the sequence of lines read in the file are filtered then cleaned. The cleaning consists of lower casing each line, trimming whitespace and then removing the http and precending www values if present.

(defn read-domain-file [fname]
  ;; read file and return list of non-commented lines
  ;; - remove commented lines
  ;; - trim leading and trailing whitespace
  ;; - remove http[s]:// prefixes
  ;; - remove www. prefixes
  ;; - lower case
  (with-open [r ( fname)]
     (->> (line-seq r)
          (filter ignore-line)
          (map #(-> %

Continue reading →

Clojure And STDERR

September 29, 2017

How to print to STDERR in Clojure came up the other day.

It turns out there are two vars *out* and *err* that are bound to STDOUT and STDERR. There vars are instances of `' and you can simply write to them.

For example, here is a string being printed to STDOUT.

user> (.println *out* "hello")

The same can be done for STDERR. Just send a string out through *err*.

 (.println *out* "hello")

In addition you can also send something to *err* by temporarily binding the value of *out* to the current value of *err*. This allows you use your normal println function and have it's output go to a different place.

  (binding [*out* *err*]
    (println msg))

The repo linked below has a simple Clojure project you can build and run to show both techniques.

Continue reading →

New Mac Python

September 28, 2017

Some new Mac notes concerning Python. Out of the box the Mac has Python and when I ran pip list I saw a bunch of stuff that I didn't recognize. When I tried to remove things they looked baked in because they were installed as root.

Turns out you can ignore all that. Just install a new Python using brew.

$ brew install python

This gives you a new Python called Python2. That's not that helpful because who wants to remember to always use Python2?

One avenue which you'll be going down at some point is to use a virtual environment for all your projects. For that install virtualenv.

$ pip2 install virtualenv

Not the pip2. This is the version of pip installed along with your Python2. Odd, yes and not remember-able just like Python2 but that's probably the last time you need to remember.

For each project you have create a virtualenv using the following from your project's root.

$ virtualenv env
$ source ./env/bin/activate

Continue reading →

Ads Txt Files

September 27, 2017

Recently, my major focus has been AdTech. With this I came across the Ads.txt project. This project is a simple initiative proposed as a way to help publishers ensure inventory is sold only through authorized dealers and partners.


Publishers create and add a file called ads.txt at the root of their sites. This text file contains a list of names and information about authorized ad networks, SSPs and exchanges that have permission to sell the publisher's inventory.


Buyers can when purchasing inventory through and exchange can go to the publisher's domain and check that the exchange is authorized to sell inventory from the publisher by reading the publisher's ads.txt file.

Reference Crawler

The folks over at the IAB Tech Lab have released a reference implementation in Python of a simple crawler for Ads.txt files. You pass it a list of domains and it reads the ads.txt from each site and adds the contents of each to a local SQLite database.

Here is the repo.

Continue reading →

Fixing Your Master Branch

September 26, 2017

You commited a change to master to it is now ahead of orign/master by 1 commit. You realize that you want this commit on a branch so you create a new branch branch.

Now your branch and master are the same but you want master to go back to being in sync with origin/master

$ git checkout -B master origin/master

Continue reading →