Overview

I have a few old sites that were developed in PHP using the Zend framework. Since moving to Clojure mostly full-time I thought it would be good to do some house cleaning and move these old sites over to Clojure. While doing so I and seeing that there was a basic pattern to the process I thought it might help others if I mocked up an example and put some notes together.

Now, before I go on I'll say that my example is very simple and it covers a basic site. Highlighting the structure of the PHP/Zend site and the Clojure/Compojure site is the main idea I'd like to get across. Those with more complex sites will clearly need to do more. Hopefully, the following will help get them started.

Repo

To start visit or clone the following repo. It contains two examples of the same simple site. One in PHP and the other in Clojure. I'll walk through the differences.

https://github.com/bradlucas/php-to-clojure

The PHP site is currently (as of today) running at http://php.beaconhill.com and the Clojure version at http://clj.beaconhill.com.

PHP

Starting with the PHP/Zend site you'll see the structure looks like the following:

.
|-- application
|   |-- Bootstrap.php
|   |-- configs
|   |   `-- application.ini
|   |-- controllers
|   |   |-- AboutController.php
|   |   `-- IndexController.php
|   |-- layouts
|   |   `-- scripts
|   |       `-- layout.phtml
|   `-- views
|       `-- scripts
|           |-- about
|           |   |-- contact.phtml
|           |   `-- index.phtml
|           `-- index
|               `-- index.phtml
|-- library
|   |-- Emailer.php
|   `-- Library.php
`-- public
    `-- index.php

Continue reading →

---

I previously mentioned reviving this blog by starting to use Cryogen.

Cryogen works by compiling your posts created in Markdown into HTML using the theme and configuration settings you've setup. This is great because I can edit everything in Emacs just as I do for nearly everything else.

Here, I've been using org-mode for nearly all lists, plans and documents for a very long time. With the discovery of Markdown Mode for Emacs) I've found a new balance. With keystrokes for expanding and moving around your Markdown document that are familar to the org-mode user I've found the new mode is making creating these blog posts as comfortable as when I was creating documents in org-mode.

Link

• http://jblevins.org/projects/markdown-mode/

Continue reading →

---

I recently revived my blog by building it with the static site generator Cryogen. Working primarily in Clojure these days made using Cryogen a nice fit.

Benefits

• Static site generators let you create posts in your own editor using a simple markup language such as Markdown
• Because Cryogen is developed in Clojure you have a software package you can edit if you'd like
• All the basics are included including support for tags, rss feed files and commenting with Disqus

Setup

As stated on the Cryogen home page the creation of a new blog is simple because the authors have created a Leiningen template.

Simply enter:

$ leiningen new cryogen blog-name

Then cd into blog-name and enter lein ring server and you can view your new blog at http://localhost:3000.

Next you can start editing and creating files in blog-name/resources/templates/md/posts. The md assumes you are going to work with Markdown files.

Continue reading →

---

Story

So I took the early train today. Earlier than I usually do and as a result I was both croggy and rushing. While I was coming down the hill and about to cross through the parking lot at the train a car rushed by with what seemed like little care as to where I was.

Now I titled this post partially with the words nearly hit but to be honest that wasn't the case. What really was the case was that I think the car should have stopped or at least slowed because of where I was. Normally this seems to be the polite behavior of drivers around here and for a moment I was put back as the car rushed around the corner and I continued walking in a fashion to prove how close I was.

I think I muttered something under my breath and it might not have been all that nice. Then, because I was rushing I pushed on and as I reached the stairs I noticed the car pulling into a spot down the way along the track. Ah, I thought they are trying to catch the train. Well, that explains their behavior somewhat.

Now, back to me. Another part of the story is time and one's sense of it. When you commute you get these very fixed goals in your head revolving around the train schedule. You also learn how long it takes for you to walk to the train and as it happens you whittle that walking commute down to the shortest possible amount. So I left by looking at the clock and starting the game of getting to the train in my allotted amount of time. Today was no different than any other and I was cutting it close. Not running close but just the exact amount.

This was so true that when I got to the platform and found my spot to stand it wasn't but a minute before the train started to pull in. As this was happening I noticed a women running along the other side to reach the stairs. Ah, it's the woman from the car in the parking lot. She is cutting it very close.

Now, here is where you take something with you about getting upset at others in their interactions with you.

I get on the train and sit and you'll never quess who gets on the train as the doors are closing and sits down next to me. Yes, the woman from the car.

So, I say to her. "That was close, it seems like the train is early". She acknowledge this then looked at your phone and said "Actually, it was exactly on time".

I looked at my phone and noticed the time. She was right. I bet the clock at home has become slow. I thought I had more time and she knew that she didn't.

Continue reading →

---

Talking to Java

Clojure is hosted on the Java Virtual Machine (JVM) and as such can access Java libraries and classes quite easily. Some of Java is built in so to speak and included in Clojure. Look inside of the clojure.java.io namespace. A good example is file which returns a java.io.File object.

These and other examples are useful to understand. But, what if you have an external Java library which you'd like to use. Here is an example:

Apache Commons Lang

The commons lang library provides among other things a host of utilities for working with the Java API. Let's pick something within it that we'd like to use from within our Clojure program to demonstrate the process of working with an external Java library.

How about the class used in a recent post.

There I needed to unescape any Java literals from within a String. Specifically this function https://commons.apache.org/proper/commons-lang/javadocs/api-release/org/apache/commons/lang3/StringEscapeUtils.html#unescapeJava-java.lang.String-

What are the steps to integrate this into your Clojure project and all the function. Let's work up from the function.

Step 1

The function is a static member function of a class within a library. Look at the home page of the library https://commons.apache.org/proper/commons-lang/. Here you'll find the maven information. This is what you'll need to know to add the library to your project. Midway down the page see the following block.

Continue reading →

---

Overview

As a final post about the new Yahoo Finance quote download page and how to access the historical data I thought to offer a bit of a summary.

The four previous posts with examples for Bash, Python, Java and Clojure are available at the following urls. Please visit them if you have just arrived at this post without seeing them. At the very least, read the beginning of the first one for more details on the issue with the new Yahoo Finance download data page and the technique to call it programmatically.

• 1. Yahoo Finance Quote Download (Bash)
• 2. Yahoo Finance Quote Download (Python)
• 3. Yahoo Finance Quote Download (Java)
• 4. Yahoo Finance Quote Download (Clojure)

Bash

Creating a Bash script to solve a problem is something every developer should be able to do. The Unix Philosophy of build modular functional components and connecting them together is a very powerful technique and well worth becoming comfortable with. To be successful though you must know a number of the available Unix commands. For our script here, prior knowledge of curl, tr, cut, grep and sed was necessary.

In practice the building of a Bash script could be considered a first step. Depending on what you eventually will need you may leave the script as a proof of concept as you add on layers of production requirements. Also, if you start building a number of scripts I suggest you always try to modularize them and start building libraries to share functions. If you get in the habit of building large scripts from active cut and paste sessions you'll eventually have a difficult situation to manage.

Python

If you look at the Python example soon after the Bash script you might think things look a bit cleaner. I'll say that. You'll notice I built the Python script up as a collection of small functions. This technique allows for an iterative development cycle in the repl where each function is carefully constructed and tested individually. Note, that this technique is also heavily used in the creation of Clojure programs. You'll see that later. For now, though realize that another Python developer who fashioned themselves as an object oriented one would have create an entirely different application.

If I was to continue with the Python script I'd share the same advice just given for Bash in that carefully consider moving useful functions to libraries so they are easily used for other applications. Other than that I'd find the Python example to be a good first step and one that could be used to grow with.

Continue reading →

---

Overview

Today's post is a final version of the Yahoo Finance Quote Download application written using the Clojure programming language. If you are interested there are three previous versions of this application using Bash, Python and Java.

• Bash
• Python
• Java

For the impatient you can find the source for this post in the following git repo.

https://github.com/bradlucas/get-yahoo-quotes-clojure

Investigation

The first post in this series details the new way the Yahoo Finance pages work with their cookie and a crumb value embedded in the page data. Here I'll just state the steps and how they can be easily accomplished in Clojure.

First, just like in Java we need to consider using a library to do the http requests against the Yahoo site. A popular choice is clj-http which is a Clojure wrapper and interface over the Java HttpClient library. Considering we just used the HttpClient library I decided to use clj-http here.

So the steps we need to accomplish are as follows:

• Read the finance page for a specific symbol
• Grep the page for crumbStore and save the associated crumb value
• Keep the cookie which comes with the interaction with the page
• Build a url with the crumb and data parameters to request a data download
• Call the download url and pass along the cookie we received earlier.

Continue reading →

---

Overview

Today's post includes and overview of a Java implementation created to support the new Yahoo Finance page changes. Previous posts include implementations in Bash and Python. Please review them if curious or if you feel the following doesn't have enough of an explanation of the issue created when Yahoo changed their finance page back in May.

For the impatient you can find the program in the following git repo.

https://github.com/bradlucas/get-yahoo-quotes-java

Investigating

The issue detailed previously is that to download quote data on the new Yahoo pages you need to do two things. First, you navigate to the symbol's data page. There your browser or client will be given a cookie that is needed for the later download. Also, the page will have a crumb value associated with CrumbStore. This value needs to be parsed out of the page's data.

We'll do this in Java and use the HttpClient library to do our client connection requests. One immediate bonus is by sharing the HttpClient instance we won't have to manually grab the B cookie and use it because the client will take care of sharing on the subsequent download.

So, for our first page request we only need to focus on retrieving he crumb value.

Data in the page

We want to call the initial http://finance.yahoo.com/quote page to get the crumb. Working from top down the first function is getCrumb. This function will find the crumb by looking in the lines of the page returned from getPage.

Continue reading →

---

Overview

Picking up the thread after yesterday's post on the new Yahoo Finance page changes http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/ I thought it would be useful to repeat the exercise from yesterday and build the new download script in Python.

For the impatient you can find the script in the following git repo.

https://github.com/bradlucas/get-yahoo-quotes-python

Investigating

I suggest you visit the previous post for details on the new scheme Yahoo has for which you need to save a cookie and parse a page for a crumb value.

We'll do the same here in Python.

To summarize we'll do the following:

• Use the requests library to make url requests
• Save the 'B cookie value for later calls to request the data
• Parse the initial page for the crumb value associated with the key CrumbStore
• Assume we want all the data and create a timestamp value for today/now

Data in the page

Continue reading →

---

Overview

On May 18th 2017 the Yahoo ichart data download link http://ichart.finance.yahoo.com/table.csv stopped working. This link has for many years been a convenient access point for downloading historical quote data. It was simple to get quotes by building up this url with your symbol as a parameter.

Since this link was under the Download Data button on the https://finance.yahoo.com/ page after you've submitted a symbol the first place to look in finding a resolution was to visit this page to what Yahoo had replaced it with.

Investigating

Using Google as an example we can excercise this url https://finance.yahoo.com/quote/GOOG?p=GOOG to look for the current Download Data button. Investigate the page and find the Historical Data tab. When clicked the Download Data button appears above the default range of quote values.

Notice, the link created for downloading the data.

https://query1.finance.yahoo.com/v7/finance/download/GOOG?period1=1493836545&period2=1496514945&interval=1d&events=history&crumb=4c1fh7TK/VW

Two things are noticable. The period values are likely be the from and to date values and the mysterious crumb value. Reviewing the page you'll see the Time period date range. These dates when converted to Unix (or Epoch) time match the period values. The crumb is likely to be a unique value generaeted for our page.

Now, how can we verify this.

Data in the page

Continue reading →

---

Hosting a Git repo on your own VPS server is a simple. Though paying for GitHub to host a private repo does not cost very much you may have reasons to keep your source repositories on your own server. You might enjoying do things yourself, have reasons to keep your source on your own server or would like to simply have a place to manage your personal projects. The following will walk you through the process of creating and hosting a git repository on your own server.

Server

On your server create a directory where you will store your repository. I'd image you'll create more than one over time so name the directory something appropriate such as repos. If you are just starting off and it is only you I suggest for simplicity you create a directory called repos under your home directory.

For our example here I'm going to create a repo called project on a machine with the domain name of example-server.com. My username for this example will be user. Before continuing make sure you can ssh easily into your account on your server. In our example you would need to be able to execute the following command.

$ ssh user@example-server.com

For this you should configure your ssh so you can do so without having to enter your password.d

Creating the repository

On the server the command to create the initial empty repository is git init –bar REPONAME.git. For example:

$ cd $HOME
$ mkdir repos
$ cd repos
$ git init --bare project.git

Continue reading →

---

I've reached Level 1 on Project Euler using Clojure as the programming language.

Level 1 is reached when you've completed 25 exercises.

You get this nice badge.

I've put my solutions on Github here.

https://github.com/bradlucas/project-euler

Continue reading →

---

The problem. Find all the countries in the world without an ‘A’ in their name.

1. Find a list of the world’s countries. [http://www.listofcountriesoftheworld.com(http://www.listofcountriesoftheworld.com)
2. Cut/paste the list to a file. countries.txt
3. Note the file is a list of numbers and names. Need to remove the numbers and filter out the ones without ‘A’s.

 def remove_country_line_numbers():
    return [l.strip() for l in  open('countries.txt').readlines() \
                 if not l.strip().isdigit() and l.lower().find('a')==-1]

Usage:

In [81]: ", ".join(remove_country_line_numbers())
Out[81]: "Belgium, Belize, Benin, Brunei, Burundi, Chile, Comoros, Congo, 
Republic of the, Cote d'Ivoire, Cyprus, Czech Republic, Djibouti, Egypt,
 Fiji, Greece, Guernsey, Hong Kong, Jersey, Lesotho, Liechtenstein, Luxembourg, 
Mexico, Morocco, Niger, Niue, Peru, Philippines, Puerto Rico, Reunion, Seychelles, 
Sweden, Timor-Leste, Togo, Turkey, United Kingdom, Yemen"

Continue reading →

---

WTF is CISPA

http://images.paralegal.net.s3.amazonaws.com/cispa.jpg

Continue reading →

---

Getting back to things and have updated my personal page at http://bradlucas.com.

It is a simple page with six thumbnails using Twitter Bootstrap (http://twitter.github.com/bootstrap/). What made it nice is that it is responsive so it works well on any device.

Continue reading →

---