Brad Lucas

Overview

Picking up the thread after yesterday's post on the new Yahoo Finance page changes http://blog.bradlucas.com/posts/2017-06-02-new-yahoo-finance-quote-download-url/ I thought it would be useful to repeat the exercise from yesterday and build the new download script in Python.

For the impatient you can find the script in the following git repo.

https://github.com/bradlucas/get-yahoo-quotes-python

Investigating

I suggest you visit the previous post for details on the new scheme Yahoo has for which you need to save a cookie and parse a page for a crumb value.

We'll do the same here in Python.

To summarize we'll do the following:

• Use the requests library to make url requests
• Save the 'B cookie value for later calls to request the data
• Parse the initial page for the crumb value associated with the key CrumbStore
• Assume we want all the data and create a timestamp value for today/now

Data in the page

We want to call the initial finance.yahoo.com/quote page to get the cookie as well as the crumb. Working from top down the first function is get_cookie_crumb.

def get_cookie_crumb(symbol):
    cookie, lines = get_page_data(symbol)
    crumb = split_crumb_store(find_crumb_store(lines))
    # Note: possible \u002F value
    # ,"CrumbStore":{"crumb":"FWP\u002F5EFll3U"
    # FWP\u002F5EFll3U
    crumb2 = crumb.decode('unicode-escape')
    return cookie, crumb2

The support routines are as follows. Some points to note:

• Once in a while you'll get a value of \u002F in your crumb ("CrumbStore":{"crumb":"FWP\u002F5EFll3U"). The encode('utf-8') method sorts this out.
• The line with CrumbStore looks like ,"CrumbStore":{"crumb":"9q.A4D1c.b9. You can grab the crumb with a split on :.
• The cookie you want to save is B

  def split_crumb_store(v):
      return v.split(':')[2].strip('"')
  
  

def find_crumb_store(lines): # Looking for # ,"CrumbStore":{"crumb":"9q.A4D1c.b9 for l in lines: if re.findall(r'CrumbStore', l): return l print "Did not find CrumbStore" def get_cookie_value(r): return {'B': r.cookies['B']} def get_page_data(symbol): url = "https://finance.yahoo.com/quote/%s/?p=%s" % (symbol, symbol) r = requests.get(url) cookie = get_cookie_value(r) # lines = r.text.encode('utf-8').strip().replace('}', '\n') lines = r.content.strip().replace('}', '\n') return cookie, lines.split('\n')

Download Data

The main entry point to the script is download_quotes. This routine accepts a symbol, builds the data parameters, calls get_cookie_crumb and finally does the download with get_data.

The get_data routine builds the required url using the parameters create and the cookie and crumb values retrieved earlier.

With the response returned from requests the routine writes the contents to the symbol.csv file.

def get_data(symbol, start_date, end_date, cookie, crumb):
    filename = '%s.csv' % (symbol)
    url = "https://query1.finance.yahoo.com/v7/finance/download/%s?period1=%s&period2=%s&interval=1d&events=history&crumb=%s" % (symbol, start_date, end_date, crumb)
    response = requests.get(url, cookies=cookie)
    with open (filename, 'w') as handle:
        for block in response.iter_content(1024):
            handle.write(block)


def get_now_epoch():
    # @see https://www.linuxquestions.org/questions/programming-9/python-datetime-to-epoch-4175520007/#post5244109
    return int(time.mktime(datetime.datetime.now().timetuple()))


def download_quotes(symbol):
    start_date = 0
    end_date = get_now_epoch()
    cookie, crumb = get_cookie_crumb(symbol)
    get_data(symbol, start_date, end_date, cookie, crumb)

The Final Script

My final version of the script discussed here is up on GitHub in the following repo:

https://github.com/bradlucas/get-yahoo-quotes-python

Tags: yahoo quotes trading python

« Yahoo Finance Quote Download Java New Yahoo Finance Quote Download Url »