Stock Ticker Symbols for Machine Learning and Artificial Intelligence

I received a few questions from multiple followers regarding a working list of all traded stocks. This is generally followed by an explanation that they want to apply machine-learning or artificial intelligence to predict stock prices. My initial response is always, that sounds like fun, let's start with a data-set that is a bit smaller first and ideally one that you are familiar with.

Capturing ALL traded stocks would become unwieldy rather quickly. Most stock data services have limits even on their paid plans. Using funnhub.io as an example, their most expensive plan at \$500 a month only allows for 300 API calls a minute. That is 216,000 API calls a day to retrieve data and a quick look at gurufocus.com indicates around 120,000 stocks, and I'm not certain that is all of them. At 300 API call an hour it would take half a day to get the latest information and most likely you would want to run multiple API calls on at least some of those symbols. The only way you are going to get faster data than that is to spend many thousands of dollars and if you have that kind of money you wouldn't be asking me how to get all the ticker symbols.

As stated above, let's make something a bit more reasonable. Everyone is familiar with the New York Stock Exchange (NYSE), and whether they know it by its actual name or not, the National Association of Securities Dealers Automated Quotation System (Nasdaq). So let's start with these, and the very first thing you are going to need is a list of all their associated tickers symbols.

If you are just getting started with the stock market and do not yet have a Robinhood account, you can collect a free stock from Robinhood by following this link. Robinhood.com

Now we could scrape this data and indeed if we wanted to constrain our search to the S&P 500 only we could get its list of symbols in three lines of python

import pandas as pd
table=pd.read_html('https://en.wikipedia.org/wiki/List_of_S%26P_500_companies')
df = table[0]

df.head()

Scraping the entire list would be quite cumbersome however as most websites that maintain an entire list break that list into multiple pages. Interestingly enough the website nasdaqtrader.com keeps a list of all symbols.

df = pd.read_csv('ftp://ftp.nasdaqtrader.com/SymbolDirectory/nasdaqtraded.txt', sep='|', index_col=False)

df.head()

In order to use this list we need to know some things about it. First lets understand the Listing Exchange Column. This column gives us information regarding what exchange the stock was actually listed on. We can interpret this information using the below list.

• A = NYSE MKT
• N = New York Stock Exchange (NYSE
• P = NYSE ARCA
• Z = BATS Global Markets (BATS)
• V = Investors' Exchange, LLC (IEXG)

Since we are specifically looking at stocks, we need to remove our ETF's.

no_ETFs = df['Listing Exchange']=='N'
df = df[no_ETFs]
df.head()

Next, we need to filter out all the test securities using the Test Issue Column.

no_Tests = df['Test Issue']=='N'
df = df[no_Tests]
df.head()

This list provides various differences in securities by adding a period or dollar sign and denoting this variance after that indicator. In the list above you can see that AAIC is list 3 times. The second and third have \$B and \$C respectively. This indicates that AAIC has multiple classes, normal, class B, and class C stocks. These types of annotation can also denote common stock, preferred stock, etc. For this, we really only need to look at the base ticker symbol.

removeDollar = df['Symbol'].str.contains('$', regex=False) == False
df = df[removeDollar]

removePeriod = df['Symbol'].str.contains('.', regex=False) == False
df = df[removePeriod]

df.head()

Now we need to put the ticker symbols into a list.

symbols = df['Symbol'].to_list()
print(len(symbols))
symbols[:10]

2537





['A', 'AA', 'AAIC', 'AAN', 'AAP', 'AAT', 'AB', 'ABB', 'ABBV', 'ABC']

We now have a rather large list of ticker symbols and better yet, we can parse information from finnhub.io under 10 minutes rather than taking 12 hours. Yahoo finance has a 2000 API hit per hour limit, so I am only going to grab the first 100 ticker symbols.

First we are going to write a function to retrieve the last close price of each stock in our list.

def getLastAdjustedClose(symbol):
    import datetime
    end = datetime.date.today()
    start = datetime.date.today() - datetime.timedelta(days=7)
    df = pdr.DataReader(symbol, "yahoo", start, end).reset_index()
    return(df['Adj Close'].tail(1))

Next, I am going to loop through our entire list and get the last adjusted close price

import pandas_datareader as pdr   
import datetime

symbol_dict = dict()
for symbol in symbols[:100]:
    adjclose = getLastAdjustedClose(symbol)
    symbol_dict[symbol] = adjclose

pd.DataFrame.from_dict(symbol_dict).round()
    

As always, open an interactive version of this notebook at binder.org

And don't forget to claim your free stock