I have a few old sites that were developed in PHP using the Zend framework. Since moving to Clojure mostly full-time I thought it would be good to do some house cleaning and move these old sites over to Clojure. While doing so I and seeing that there was a basic pattern to the process I thought it might help others if I mocked up an example and put some notes together.
Now, before I go on I'll say that my example is very simple and it covers a basic site. Highlighting the structure of the PHP/Zend site and the Clojure/Compojure site is the main idea I'd like to get across. Those with more complex sites will clearly need to do more. Hopefully, the following will help get them started.
To start visit or clone the following repo. It contains two examples of the same simple site. One in PHP and the other in Clojure. I'll walk through the differences.
https://github.com/bradlucas/php-to-clojure
The PHP site is currently (as of today) running at http://php.beaconhill.com and the Clojure version at http://clj.beaconhill.com.
Starting with the PHP/Zend site you'll see the structure looks like the following:
.
|-- application
| |-- Bootstrap.php
| |-- configs
| | `-- application.ini
| |-- controllers
| | |-- AboutController.php
| | `-- IndexController.php
| |-- layouts
| | `-- scripts
| | `-- layout.phtml
| `-- views
| `-- scripts
| |-- about
| | |-- contact.phtml
| | `-- index.phtml
| `-- index
| `-- index.phtml
|-- library
| |-- Emailer.php
| `-- Library.php
`-- public
`-- index.php
I previously mentioned reviving this blog by starting to use Cryogen.
Cryogen works by compiling your posts created in Markdown into HTML using the theme and configuration settings you've setup. This is great because I can edit everything in Emacs just as I do for nearly everything else.
Here, I've been using org-mode for nearly all lists, plans and documents for a very long time. With the discovery of Markdown Mode for Emacs) I've found a new balance. With keystrokes for expanding and moving around your Markdown document that are familar to the org-mode user I've found the new mode is making creating these blog posts as comfortable as when I was creating documents in org-mode.
I recently revived my blog by building it with the static site generator Cryogen. Working primarily in Clojure these days made using Cryogen a nice fit.
As stated on the Cryogen home page the creation of a new blog is simple because the authors have created a Leiningen template.
Simply enter:
$ leiningen new cryogen blog-name
Then cd into blog-name
and enter lein ring server
and you can view your new blog at http://localhost:3000
.
Next you can start editing and creating files in blog-name/resources/templates/md/posts
. The md
assumes you are going to work with Markdown files.
So I took the early train today. Earlier than I usually do and as a result I was both croggy and rushing. While I was coming down the hill and about to cross through the parking lot at the train a car rushed by with what seemed like little care as to where I was.
Now I titled this post partially with the words nearly hit but to be honest that wasn't the case. What really was the case was that I think the car should have stopped or at least slowed because of where I was. Normally this seems to be the polite behavior of drivers around here and for a moment I was put back as the car rushed around the corner and I continued walking in a fashion to prove how close I was.
I think I muttered something under my breath and it might not have been all that nice. Then, because I was rushing I pushed on and as I reached the stairs I noticed the car pulling into a spot down the way along the track. Ah, I thought they are trying to catch the train. Well, that explains their behavior somewhat.
Now, back to me. Another part of the story is time and one's sense of it. When you commute you get these very fixed goals in your head revolving around the train schedule. You also learn how long it takes for you to walk to the train and as it happens you whittle that walking commute down to the shortest possible amount. So I left by looking at the clock and starting the game of getting to the train in my allotted amount of time. Today was no different than any other and I was cutting it close. Not running close but just the exact amount.
This was so true that when I got to the platform and found my spot to stand it wasn't but a minute before the train started to pull in. As this was happening I noticed a women running along the other side to reach the stairs. Ah, it's the woman from the car in the parking lot. She is cutting it very close.
Now, here is where you take something with you about getting upset at others in their interactions with you.
I get on the train and sit and you'll never quess who gets on the train as the doors are closing and sits down next to me. Yes, the woman from the car.
So, I say to her. "That was close, it seems like the train is early". She acknowledge this then looked at your phone and said "Actually, it was exactly on time".
I looked at my phone and noticed the time. She was right. I bet the clock at home has become slow. I thought I had more time and she knew that she didn't.
Clojure is hosted on the Java Virtual Machine (JVM) and as such can access Java libraries and classes quite easily. Some of Java is built in so to speak and included in Clojure. Look inside of the clojure.java.io namespace. A good example is file which returns a java.io.File object.
These and other examples are useful to understand. But, what if you have an external Java library which you'd like to use. Here is an example:
The commons lang library provides among other things a host of utilities for working with the Java API. Let's pick something within it that we'd like to use from within our Clojure program to demonstrate the process of working with an external Java library.
How about the class used in a recent post.
There I needed to unescape any Java literals from within a String. Specifically this function https://commons.apache.org/proper/commons-lang/javadocs/api-release/org/apache/commons/lang3/StringEscapeUtils.html#unescapeJava-java.lang.String-
What are the steps to integrate this into your Clojure project and all the function. Let's work up from the function.
The function is a static member function of a class within a library. Look at the home page of the library https://commons.apache.org/proper/commons-lang/. Here you'll find the maven information. This is what you'll need to know to add the library to your project. Midway down the page see the following block.
As a final post about the new Yahoo Finance quote download page and how to access the historical data I thought to offer a bit of a summary.
The four previous posts with examples for Bash, Python, Java and Clojure are available at the following urls. Please visit them if you have just arrived at this post without seeing them. At the very least, read the beginning of the first one for more details on the issue with the new Yahoo Finance download data page and the technique to call it programmatically.
Creating a Bash script to solve a problem is something every developer should be able to do. The Unix Philosophy of build modular functional components and connecting them together is a very powerful technique and well worth becoming comfortable with. To be successful though you must know a number of the available Unix commands. For our script here, prior knowledge of curl, tr, cut, grep and sed was necessary.
In practice the building of a Bash script could be considered a first step. Depending on what you eventually will need you may leave the script as a proof of concept as you add on layers of production requirements. Also, if you start building a number of scripts I suggest you always try to modularize them and start building libraries to share functions. If you get in the habit of building large scripts from active cut and paste sessions you'll eventually have a difficult situation to manage.
If you look at the Python example soon after the Bash script you might think things look a bit cleaner. I'll say that. You'll notice I built the Python script up as a collection of small functions. This technique allows for an iterative development cycle in the repl where each function is carefully constructed and tested individually. Note, that this technique is also heavily used in the creation of Clojure programs. You'll see that later. For now, though realize that another Python developer who fashioned themselves as an object oriented one would have create an entirely different application.
If I was to continue with the Python script I'd share the same advice just given for Bash in that carefully consider moving useful functions to libraries so they are easily used for other applications. Other than that I'd find the Python example to be a good first step and one that could be used to grow with.
Today's post is a final version of the Yahoo Finance Quote Download application written using the Clojure programming language. If you are interested there are three previous versions of this application using Bash, Python and Java.
For the impatient you can find the source for this post in the following git repo.
https://github.com/bradlucas/get-yahoo-quotes-clojure
The first post in this series details the new way the Yahoo Finance pages work with their cookie and a crumb
value embedded in the page data. Here I'll just state the steps and how they can be easily accomplished in Clojure.
First, just like in Java we need to consider using a library to do the http requests against the Yahoo site. A popular choice is clj-http
which is a Clojure wrapper and interface over the Java HttpClient
library. Considering we just used the HttpClient
library I decided to use clj-http
here.
So the steps we need to accomplish are as follows:
crumbStore
and save the associated crumb
valuecookie
which comes with the interaction with the pagecookie
we received earlier.Today's post includes and overview of a Java implementation created to support the new Yahoo Finance page changes. Previous posts include implementations in Bash and Python. Please review them if curious or if you feel the following doesn't have enough of an explanation of the issue created when Yahoo changed their finance page back in May.
For the impatient you can find the program in the following git repo.
https://github.com/bradlucas/get-yahoo-quotes-java
The issue detailed previously is that to download quote data on the new Yahoo pages you need to do two things. First, you navigate to the symbol's data page. There your browser or client will be given a cookie that is needed for the later download. Also, the page will have a crumb
value associated with CrumbStore
. This value needs to be parsed out of the page's data.
We'll do this in Java and use the HttpClient library to do our client connection requests. One immediate bonus is by sharing the HttpClient instance we won't have to manually grab the B
cookie and use it because the client will take care of sharing on the subsequent download.
So, for our first page request we only need to focus on retrieving he crumb
value.
We want to call the initial http://finance.yahoo.com/quote
page to get the crumb
. Working from top down the first function is getCrumb
. This function will find
the crumb by looking in the lines of the page returned from getPage
.
Picking up the thread after yesterday's post on the new Yahoo Finance page changes http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/ I thought it would be useful to repeat the exercise from yesterday and build the new download script in Python.
For the impatient you can find the script in the following git repo.
https://github.com/bradlucas/get-yahoo-quotes-python
I suggest you visit the previous post for details on the new scheme Yahoo has for which you need to save a cookie and parse a page for a crumb
value.
We'll do the same here in Python.
To summarize we'll do the following:
requests
library to make url requests'B
cookie value for later calls to request the datacrumb
value associated with the key CrumbStore
On May 18th 2017 the Yahoo ichart data download link http://ichart.finance.yahoo.com/table.csv
stopped working. This link has for many years been a convenient access point for downloading historical quote data. It was simple to get quotes by building up this url with your symbol as a parameter.
Since this link was under the Download Data
button on the https://finance.yahoo.com/ page after you've submitted a symbol the first place to look in finding a resolution was to visit this page to what Yahoo had replaced it with.
Using Google as an example we can excercise this url https://finance.yahoo.com/quote/GOOG?p=GOOG to look for the current Download Data
button. Investigate the page and find the Historical Data
tab. When clicked the Download Data
button appears above the default range of quote values.
Notice, the link created for downloading the data.
https://query1.finance.yahoo.com/v7/finance/download/GOOG?period1=1493836545&period2=1496514945&interval=1d&events=history&crumb=4c1fh7TK/VW
Two things are noticable. The period values are likely be the from
and to
date values and the mysterious crumb
value. Reviewing the page you'll see the Time period date range. These dates when converted to Unix (or Epoch) time match the period values. The crumb is likely to be a unique value generaeted for our page.
Now, how can we verify this.
Hosting a Git repo on your own VPS server is a simple. Though paying for GitHub to host a private repo does not cost very much you may have reasons to keep your source repositories on your own server. You might enjoying do things yourself, have reasons to keep your source on your own server or would like to simply have a place to manage your personal projects. The following will walk you through the process of creating and hosting a git repository on your own server.
On your server create a directory where you will store your repository. I'd image you'll create more than one over time so name the directory something appropriate such as repos
. If you are just starting off and it is only you I suggest for simplicity you create a directory called repos
under your home directory.
For our example here I'm going to create a repo called project
on a machine with the domain name of example-server.com
. My username for this example will be user
. Before continuing make sure you can ssh easily into your account on your server. In our example you would need to be able to execute the following command.
$ ssh user@example-server.com
For this you should configure your ssh so you can do so without having to enter your password.d
On the server the command to create the initial empty repository is git init –bar REPONAME.git
. For example:
$ cd $HOME
$ mkdir repos
$ cd repos
$ git init --bare project.git
I've reached Level 1 on Project Euler using Clojure as the programming language.
Level 1 is reached when you've completed 25 exercises.
You get this nice badge.
I've put my solutions on Github here.
https://github.com/bradlucas/project-euler
The problem. Find all the countries in the world without an ‘A’ in their name.
def remove_country_line_numbers():
return [l.strip() for l in open('countries.txt').readlines() \
if not l.strip().isdigit() and l.lower().find('a')==-1]
Usage:
In [81]: ", ".join(remove_country_line_numbers())
Out[81]: "Belgium, Belize, Benin, Brunei, Burundi, Chile, Comoros, Congo,
Republic of the, Cote d'Ivoire, Cyprus, Czech Republic, Djibouti, Egypt,
Fiji, Greece, Guernsey, Hong Kong, Jersey, Lesotho, Liechtenstein, Luxembourg,
Mexico, Morocco, Niger, Niue, Peru, Philippines, Puerto Rico, Reunion, Seychelles,
Sweden, Timor-Leste, Togo, Turkey, United Kingdom, Yemen"
WTF is CISPA
http://images.paralegal.net.s3.amazonaws.com/cispa.jpg
Getting back to things and have updated my personal page at http://bradlucas.com.
It is a simple page with six thumbnails using Twitter Bootstrap (http://twitter.github.com/bootstrap/). What made it nice is that it is responsive so it works well on any device.