Increase Your Macbook Pro's Display Resolution

October 9, 2017

It turns out your Retina MacBook Pro 15" can run beyond the resolutions offered in it's System Preferences. In researching this fact I found that there are a number of third-party utilities that can be used to go beyond your Mac's offered resolutions.

The following utility from avibrazil is a good one and I recommend it. I've been using it for a few days and it works flawlessly with my MacBook as well as an external display.

The first link is to the project itself and the second link is the installable binary dmg file.


Continue reading →

Remove Siri From Touch Bar On A New Macbook Pro

October 8, 2017

I don't use Siri and on a new MacBook I hit the Siri button far too often as it is just about the Delete key. Here is how to remove the Siri button from the Touch Bar

  • System Preferences
  • Keyboard
  • Customize Control Strip

Notice your Control Strip will be wiggling Hold the Siri button and notice the Done button change to a Trash Can Drage and drop the Siri button to the Trash

Click Done


Continue reading →

Auto Increment Key In Postgresql

October 7, 2017

In MySql and MariaDB you can create a unique identity value for rows with the AUTO_INCREMENT attribute. PostgreSQL has the same idea with the SERIAL attribute.

CREATE TABLE domains
(id SERIAL PRIMARY KEY,
 name VARCHAR(254) UNIQUE,
 timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW()
 );
 

The example also shows how to have a column which stores the time when the record was created.



Continue reading →

Install PostgreSQL On A Mac Using Brew

October 6, 2017

To install Postgres on a Mac you can use brew.

$ brew update
$ brew install postgres

To create a database for yourself and login.

$ createdb `whoami`
$ psql

To drop a database.

$ dropdb DATABASE_NAME

Continue reading →

Clojure Ads Txt Crawler Saving To A SQLite Database

October 5, 2017

SQLite is a small self contained SQL database engine that reads and writes to a normal file. It is a good first pass for adding database storage to your application. It's easy to use and with a single file to deal with offers no real issues. Being a SQL database though means you won't be far off if you decide to up your database requirements to Postgres or some other database system.

My recent project of building a Clojure Ads.txt file crawler is a good target for SQLite support for a couple reasons. One is that the project is simple and works just fine writing to result files but having a database would be good in the long term especially as the project grows. Second is that the original Python Ads.txt file crawler defaulted to writing to a SQLite database. I figure the Clojure project should at the very least be able to do the same.

The results of adding SQLite are in the repository if you are curious.

Strategy

  1. Model the data and create a database table schema
  2. Add the required libraries for SQLite to our project
  3. Support optionally writing to the database
  4. Support the writing to the database

Model the data

Since we'll be storing the output from the crawler we'll need to create a table that will support this data. We can start with the sql over in the https://github.com/InteractiveAdvertisingBureau/adstxtcrawler project and tweak it a bit.

BEGIN TRANSACTION;
DROP TABLE IF EXISTS adstxt;

CREATE TABLE adstxt(
       SITE_DOMAIN                  TEXT    NOT NULL,
       EXCHANGE_DOMAIN              TEXT    NOT NULL,
       SELLER_ACCOUNT_ID            TEXT    NOT NULL,
       ACCOUNT_TYPE                 TEXT    NOT NULL,
       TAG_ID                       TEXT    NOT NULL,
       ENTRY_COMMENT                TEXT    NOT NULL,
       UPDATED                      DATE    DEFAULT (datetime('now','localtime')),
    PRIMARY KEY (SITE_DOMAIN,EXCHANGE_DOMAIN,SELLER_ACCOUNT_ID,ACCOUNT_TYPE,TAG_ID)
);

END TRANSACTION;

Continue reading →

Git Stash Pop Conflict

October 4, 2017

It is true that creating branches in Git are easy and cheap so you don't really need to git stash something. But, sometimes you don't want to save something to a branch because you really want it to be a temporary thing so you stash it.

Sometimes though you'll stash something then merge your feature branch and when you git pop conflict the stash back you'll have a conflict.

You'll see "unmerged paths" which is your clue to this situation.

What to do?

I've found that one path to take is to reset the file to unstage it, fix the conflict then add it back.

$ git reset HEAD [file(s)]

This does, if you notice, leave the stash still around.

When you are ready to get rid of it you can clear it.

$ git stash clear

Continue reading →

Ads Txt Top 100 Domain Results

October 3, 2017

The Clojure Ads.txt crawler is working well. Using the top 100 domains list from PPC.land the crawler creates the following results:

Domains with Ads.txt files

accuweather.com
allrecipes.com
ameblo.jp
babycenter.com
babygaga.com
bhaskar.com
businessinsider.com
cbsnews.com
cnet.com
daily.co.jp
dailymail.co.uk
dingit.tv
diply.com
drugs.com
elmundo.es
elpais.com
espn.com
forbes.com
foxnews.com
genius.com
globo.com
healthline.com
howstuffworks.com
latimes.com
littlethings.com
nypost.com
nytimes.com
ranker.com
softonic.com
speedtest.net
theguardian.com
therichest.com
tmz.com
topix.com
uol.com.br
usatoday.com
vice.com
washingtonpost.com
weather.com
webmd.com
wikihow.com

Domains without Ads.txt files

163.com
abril.com.br
avito.ru
azlyrics.com
bbc.com
blastingnews.com
boredpanda.com
cnn.com
cookpad.com
dailymotion.com
dictionary.com
ebay.co.uk
ebay.com
ebay.de
expedia.com
express.co.uk
goal.com
goo.ne.jp
ibtimes.com
ifeng.com
independent.co.uk
indiatimes.com
jagran.com
kakaku.com
line.me
linkedin.com
livedoor.jp
livestrong.com
mawdoo3.com
mirror.co.uk
msn.com
naver.jp
ndtv.com
nfl.com
onet.pl
outlook.com
people.com
reddit.com
reuters.com
reverso.net
sina.com.cn
spanishdict.com
spiegel.de
spotify.com
tabelog.com
telegraph.co.uk
thefreedictionary.com
thesaurus.com
thesun.co.uk
tistory.com
tribunnews.com
tripadvisor.com
walmart.com
wikia.com
wordpress.com
wordreference.com
wp.pl
xfinity.com
yahoo.com

Results

The complete results file is available for inspection at the following url.

If you want to see all the generated files see this repo https://github.com/bradlucas/top-100-domains-ads-txt


Continue reading →

Optics Game

October 2, 2017

The other day at lunch, I heard this interesting phrase, "It's an optics game". It was spoken by a friend I was having lunch with. He was detailing his current work environment which is less than positive. The story went like a lot of stories I hear from people in the same company as my friend, in that things are not positive. The company appears to be failing, a lot of people have left, no raises, etc.

My friend's boss was recently working on a series of marketing messages and powerpoint presentations. What made this odd to my friend was that my friend is in BizOps and his manager is in charge of the entire department. Why does BizOps need to market itself was my friends' first thought. Sensing this oddness, my friend asked the boss why he was doing the presentations and got back the answer, "It's an optics game".


Continue reading →

Updated Clojure Ads Txt Crawler

October 1, 2017

Work continues on the Ads.txt crawler with a focus on error handling and reporting. A new version (0.0.2) is not available and it works well against the 'Top 100 domains' file.

One particular issue was found with http-kit. It seems to have difficulty with sites that are SNI-enabled. When such a site is encountered you need a SNI-aware client.

In my top 100 list the url http://elpais.com/ads.txt causes the following error:

Error: javax.net.ssl.SSLException: Received fatal alert: handshake_failure for https://elpais.com/

This can be overcome by configuring http-kit.

(defn sni-configure
  [^SSLEngine ssl-engine ^URI uri]
  (let [^SSLParameters ssl-params (.getSSLParameters ssl-engine)]
    (.setServerNames ssl-params [(SNIHostName. (.getHost uri))])
    (.setSSLParameters ssl-engine ssl-params)))

(def client (http/make-client {:ssl-configurer sni-configure}))

To use this you pass the client in as one of the options like:

@(http/get url {:client client})

See https://github.com/bradlucas/ads-txt-crawler/blob/release/0.0.2/src/adstxtcrawler/httpkit.clj for a specific example. Note that I've left my old routine in for now to show what was in use in the previous release. Also, not that I've decided to use the SNI configuration only when there is an error.


Continue reading →

A Clojure Ads Txt Crawler

September 30, 2017

I recently discovered the Ads.txt specification and the IAB Tech Python crawler. Being more interested in Clojure I decided last weekend to write a crawler for Ads.txt files in Clojure. The first pass is available at the following repo on the release/0.0.1 branch.

Currently as of this writing, the 0.0.1 version supports passing a target domain list to have the sites crawled, their content parsed and output written in comma delimited format to STDOUT. The Python crawler from IAB saves it's output to a SQLite database. I decided that would be a feature for a later release. With my current version it is simple enough to pipe the output to another program or file for subsequent processing.

If you find this post at a later date than the release/0.0.1 branch feel free to investigate any progress I've made.

Files

domains.clj

This file is concerned with reading the target-domains file passed in via the -t option. The goal is to read the file and remove blank and commented out lines then to collect the remaining lines and reduce each url or domain to it's core domain name. Having each domain reduced to just it's component domain name will make processing them simpler.

See the read-domain-file function for an example of how the sequence of lines read in the file are filtered then cleaned. The cleaning consists of lower casing each line, trimming whitespace and then removing the http and precending www values if present.

(defn read-domain-file [fname]
  ;; read file and return list of non-commented lines
  ;; - remove commented lines
  ;; - trim leading and trailing whitespace
  ;; - remove http[s]:// prefixes
  ;; - remove www. prefixes
  ;; - lower case
  (with-open [r (clojure.java.io/reader fname)]
    (doall
     (->> (line-seq r)
          (filter ignore-line)
          (map #(-> %
                   (clojure.string/lower-case)
                   (clojure.string/trim)
                   (hostname)
                   (strip-www)))))))

Continue reading →

Clojure And STDERR

September 29, 2017

How to print to STDERR in Clojure came up the other day.

It turns out there are two vars *out* and *err* that are bound to STDOUT and STDERR. There vars are instances of `java.io.PrintWriter' and you can simply write to them.

For example, here is a string being printed to STDOUT.

user> (.println *out* "hello")
hello
nil

The same can be done for STDERR. Just send a string out through *err*.

 (.println *out* "hello")
hello
nil

In addition you can also send something to *err* by temporarily binding the value of *out* to the current value of *err*. This allows you use your normal println function and have it's output go to a different place.

  (binding [*out* *err*]
    (println msg))

The repo linked below has a simple Clojure project you can build and run to show both techniques.


Continue reading →

New Mac Python

September 28, 2017

Some new Mac notes concerning Python. Out of the box the Mac has Python and when I ran pip list I saw a bunch of stuff that I didn't recognize. When I tried to remove things they looked baked in because they were installed as root.

Turns out you can ignore all that. Just install a new Python using brew.

$ brew install python

This gives you a new Python called Python2. That's not that helpful because who wants to remember to always use Python2?

One avenue which you'll be going down at some point is to use a virtual environment for all your projects. For that install virtualenv.

$ pip2 install virtualenv

Not the pip2. This is the version of pip installed along with your Python2. Odd, yes and not remember-able just like Python2 but that's probably the last time you need to remember.

For each project you have create a virtualenv using the following from your project's root.

$ virtualenv env
$ source ./env/bin/activate

Continue reading →

Ads Txt Files

September 27, 2017

Recently, my major focus has been AdTech. With this I came across the Ads.txt project. This project is a simple initiative proposed as a way to help publishers ensure inventory is sold only through authorized dealers and partners.

Publishers

Publishers create and add a file called ads.txt at the root of their sites. This text file contains a list of names and information about authorized ad networks, SSPs and exchanges that have permission to sell the publisher's inventory.

Buyers

Buyers can when purchasing inventory through and exchange can go to the publisher's domain and check that the exchange is authorized to sell inventory from the publisher by reading the publisher's ads.txt file.

Reference Crawler

The folks over at the IAB Tech Lab have released a reference implementation in Python of a simple crawler for Ads.txt files. You pass it a list of domains and it reads the ads.txt from each site and adds the contents of each to a local SQLite database.

Here is the repo.


Continue reading →

Fixing Your Master Branch

September 26, 2017

You commited a change to master to it is now ahead of orign/master by 1 commit. You realize that you want this commit on a branch so you create a new branch branch.

Now your branch and master are the same but you want master to go back to being in sync with origin/master

$ git checkout -B master origin/master
  • https://superuser.com/questions/273172/how-do-i-reset-master-to-origin-master

Continue reading →

Push Your Feature Branch

September 25, 2017

You've created a feature branch, feature/my-latest-feature. To push it.

$ git push -u origin feature/my-latest-feature

Continue reading →