Import from Nasdaq Data Link

Nasdaq Data Link (formerly Quandl) is a popular source for a wide variety of data, especially financial and economic datasets. In 2018, Quandl was acquired by Nasdaq and rebranded as Nasdaq Data Link. The platform provides an API for accessing data, metadata, and catalog information for these datasets. Files are available in CSV, JSON, and XML formats. Some datasets are free, while others require a paid subscription.

Note: If you previously used Quandl, all functionality and APIs are now part of Nasdaq Data Link, but many references to "quandl" remain in URLs and code for backward compatibility.

• Nasdaq Data Link Documentation
• Nasdaq Data Link API Reference

At this time, all of Nasdaq Data Link's data is snapshot-updated. Much of the data is updated daily. However, depending on the dataset, update frequency ranges from hourly to annually.

Data formats

Nasdaq Data Link provides data in two primary formats: Time-Series and Tables (formerly called "Datatables"). Understanding these formats will help you select and work with the right datasets for your needs.

Time-series

Time-series datasets are designed for historical, chronological data—typically financial market data, economic indicators, or other metrics that change over time. Each row represents a unique point in time (e.g., a trading day), and columns contain values for that date. Time-series datasets may be vertically partitioned (one metric per dataset) or horizontally partitioned (multiple metrics per dataset, such as per-symbol instrument data). Metadata for time-series is often limited to column names, with less detail about data types or constraints. For example, daily closing prices for a stock, with columns for date, open, high, low, close, and volume. See Time-Series API Reference for more details.

Tables

Tables (previously "Datatables") are structured, relational-style datasets. Each table has a defined schema with column names, data types, and sometimes constraints or enumerated values. Tables can contain a mix of numeric, text, and date/time fields, and often provide richer metadata than time-series datasets. The Tables API supports complex filtering, sorting, and pagination, making it suitable for large or multi-dimensional datasets. For example, option chains, financial statements, or company fundamentals, where each row represents an entity or event with multiple attributes. See Tables API Reference for more details.

Note: Not all tables are listed in the public catalog. See the Nasdaq Data Link Tables documentation for more details and usage examples.

Connect to Nasdaq Data Link

To access datasets on Nasdaq Data Link, follow these steps:

1. Obtain an API Key
   Sign up for a free account at Nasdaq Data Link and get your API key from your account dashboard.
2. Authenticate Your Requests
   Include your API key in every request using the api_key parameter.
   • Example (cURL):

     curl "https://data.nasdaq.com/api/v3/datasets/XNAS/AAPL/data?api_key=YOUR_API_KEY"
     

   • Example (Python):

     import requests
     url = "https://data.nasdaq.com/api/v3/datasets/XNAS/AAPL/data"
     params = {"api_key": "YOUR_API_KEY"}
     response = requests.get(url, params=params)
     

   • You can also use the API key in browser URLs or with tools like Postman.
3. Free vs. Premium Datasets
   Some datasets are free, while others require a paid subscription linked to your account and API key. Attempting to access premium content without a subscription will result in an error.
4. Error Handling
   If your API key is missing or invalid, requests will fail with an HTTP 400 error. Always check that your API key is correct and has the necessary permissions for the requested dataset.

Quickstart

The following is a sample sequence of command line commands to import end-of-day data for U.S. stocks starting in 2018. These commands assume a typical Deephaven installation, and that iris_exec is on the user's PATH.

Create the table schema file

Run the following command in your home directory to generate a schema file:

iris_exec quandl_schema_creator -- -ns Quandl -qd XNAS -qt AAPL -gc ticker -pc data_set -ak <Quandl API Key>

Where:

• -ns: Namespace for the schema (e.g., 'Quandl').
• -qd: Database code (e.g., 'XNAS').
• -qt: Table or dataset code (e.g., 'AAPL' for Apple Inc.).
• -gc: Grouping column (e.g., 'ticker').
• -pc: Partition column (e.g., 'data_set').
• -ak: Your Nasdaq Data Link API key.

The generated schema file lives in the Quandl directory:

ls -l Quandl/*.schema

Deploy the schema

Import the generated schema into Deephaven. For example, if your generated schema file is called Quandl_XNAS_AAPL.schema:

sudo -u irisadmin /usr/illumon/latest/bin/dhconfig schema import --file ~/Quandl/Quandl_XNAS_AAPL.schema

Run the importer to download and import data

sudo -u dbmerge iris_exec quandl_import -- -dp localhost/2018-09-26 -ns Quandl -qd XNAS -qt AAPL -ak <Quandl API Key> -om REPLACE -aa "date.gte=20180101 qopts.export=true"

Where:

• -dp: Destination path (e.g., 'localhost/2018-09-26').
• -om: Output mode (e.g., 'REPLACE' to overwrite existing data).
• -aa: Additional arguments for filtering or export options.

Schema creator

The schema creator for Nasdaq Data Link generates a table schema from a Nasdaq Data Link data table or data set(s) using the metadata provided by Nasdaq Data Link.

Note: The command-line tools and directory names may still use 'quandl' for backward compatibility (e.g., quandl_schema_creator, quandl_import). These will continue to function with Nasdaq Data Link credentials and datasets.

Command reference

iris_exec quandl_schema_creator <launch args> -- <schema creator args>

The following arguments are available when running the schema creator:

Import Query

There is presently no support for Nasdaq Data Link imports via an Import Query.

Import to in-memory tables

Importing from Nasdaq Data Link by script differs somewhat from the file-based imports. Nasdaq Data Link data can be pulled into in-memory tables using the techniques described here.

Deephaven Quandl functions are contained in the QuandlHelpers class. This class is not imported into the Deephaven Groovy session by default, so use of these methods requires either explicit imports, or full qualification of method names.

Explicit import

import static com.illumon.iris.quandl.QuandlHelpers.*

quandlAPIKey("4qxnjGszorJKZ565sJP7")

Fully-qualified

com.illumon.iris.quandl.QuandlHelpers.quandlAPIKey("4qxnjGszorJKZ565sJP7")

Methods

There are three main methods for creating in-memory tables from Nasdaq Data Link (Quandl) data:

• getQuandlTimeSeries – connects to the time-series API, and creates a table from the resultant data retrieved.
• getQuandlTable – connects to the datatables API, and creates a table from the resultant data retrieved.
• getQuandl – will try both APIs – datatables first, then time-series if datatables doesn't have the requested data set.

Each of the getQuandl methods throws an IOException, which is normally caused by a problem parsing CSV data downloaded from Nasdaq Data Link. If the method is used through the Deephaven console, the console itself will set up a handler for the IOException. If the method is used in Java code, the method will need to be called from within a try/catch block to ensure the IOException is caught if thrown. RuntimeExceptions normally result from other problems that can occur during the download and the parsing of Nasdaq Data Link data (e.g., a timeout, or an incorrect property value).

Each of the methods noted above has several forms in which it can be used. The simplest form takes two arguments: a Nasdaq Data Link database name, and a Nasdaq Data Link dataset or datatable name. (Note: These names are shown in Nasdaq Data Link's online data catalog. The XNAS database provides daily end-of-day prices for NASDAQ-listed US equities. Replace AAPL with any valid NASDAQ ticker as needed.) For instance, using the database "XNAS" and the datatable "AAPL" provides daily end-of-day stock data for Apple Inc. (NASDAQ-listed).

A slightly more complex form for calling these functions takes three or more arguments. Like the simplest form, the first two arguments are the Nasdaq Data Link database name and the Nasdaq Data Link dataset name. However, the last argument is a variable argument (vararg), and it can be used to include one or more Nasdaq Data Link API arguments (separated by commas) to limit the scope of the data requested from Nasdaq Data Link.

For example:

myTable=getQuandlTable("XNAS", "AAPL", "date.gte=20100101", "qopts.export=true")

In this case, "date.gte=20100101" tells the Nasdaq Data Link API to filter the XNAS/AAPL datatable so only those dates that are greater than or equal to January 1, 2010 are included. The next argument, "qopts.export=true", tells the Nasdaq Data Link API to package the results in a zip file for download. (Note: Deephaven automatically handles checking for the zipped results to be ready for download, downloading them, and unpacking them.) The option to zip results is only available for datatables, not for time series. Also, for large datatables, if the results are not zipped, only the first 10,000 rows will be returned.

Another argument that can be passed to these methods is a StatusCallback object. This progress parameter allows the getQuandl methods to update process status as data is downloaded and parsed. This can be used to poll status, or to register a callback that can be used to update a status message and/or a progress bar. If the version of a method that includes a StatusCallback parameter is called, with the parameter value explicitly set to null, this will result in progress update info messages being logged to the console log. If a version of the method that does not include a StatusCallback parameter is used, then no progress messages will be logged. A custom StatusCallback object can also be created by the calling application and passed in, to allow logging and/or other progress tracking.

Retrieve data from any Quandl API: getQuandl

When using getQuandl commands, the Quandl API key can be provided using the following methods:

• Set it in a configuration (.prop) file: The setting Quandl.APIKey can be included in a configuration file.
• Set it before calling getQuandl: The quandlAPIKey property of the QuandlHelpers class can be used to set or retrieve the current API key being used by the class instance.
• Specify it as part of the optional arguments to a getQuandl call: Including api_key=[your_api_key] as one of the arguments will result in this API key being used for the call.

The order or priorities for an API key are:

1. (Highest) API key set by optional argument.
2. API key set through quandlAPIKey property.
3. API key set in configuration file.
4. If none of these is set, an empty API key will be used, and the request will most-likely fail with an HTTP 400 status code.

Retrieve Quandl time-series data: getQuandlTimeSeries

For time-series data, the dataset (second) argument to the commands also accepts a comma-separated list of dataset names, as well as wildcard patterns (using * as a wildcard character).

For example, the following query will download the time-series datasets for Apple, IBM, and Google end-of-day data, and merge them into a single table. When providing a list or pattern of datasets, a QuandlDataset column will be added to the table, and populated with the name of the dataset that provided the rows. Rows that came from the WIKI/IBM dataset will have IBM in their QuandlDataset column. Note: This column is only added when a list or pattern is used for datasets. A single dataset is returned exactly as seen on Quandl's site, with no extra column.

myTable=getQuandlTimeSeries("XNAS", "AAPL, MSFT, GOOGL")

Similarly, the following query will download and concatenate all time-series WIKI datasets that start with AT:

myTable=getQuandlTimeSeries("XNAS", "A*")

The Quandl WIKI time-series datasets are no longer available. For current US equities, use XNAS and specify the desired ticker symbol (e.g., 'AAPL'). However, other Nasdaq Data Link databases offer datasets with disparate column sets. If a selection includes datasets that use different schema from the first dataset downloaded, the command will fail.

List available Quandl datasets: getQuandlDatasets

One other command available to make it easier to use time-series data is getQuandlDatasets. For example, the following command returns a table of dataset IDs and their respective descriptions for the YALE database (Yale University Department of Economics):

yaleCatalog=getQuandlDatasets("YALE")

Or, with an API Key:

yaleCatalog=getQuandlDatasets("YALE","*","api_key=4qxnjGszorJKZ565sJP7")

Import from command line

QuandlImporter assumes (like other importers) that the target namespace and table already exist, and can then:

• Request and download zipped or non-zipped Quandl data files.
• Extract zipped data files.
• Import data to the Intraday space for the Deephaven table.

Command reference

iris_exec quandl_import <launch args> -- <quandl import args>

Quandl import arguments

Additional examples

Importing a different ticker

iris_exec quandl_schema_creator -- -ns NasdaqDataLink -qd XNAS -qt MSFT -gc ticker -pc data_set -ak <Nasdaq Data Link API Key>

Using advanced filters (e.g., date range)

sudo -u dbmerge iris_exec quandl_import -- \
  -dp localhost/2023-01-01 -ns NasdaqDataLink -qd XNAS -qt MSFT -ak <Nasdaq Data Link API Key> -om REPLACE -aa "date.gte=20220101 date.lte=20221231 qopts.export=true"

Importing to a custom namespace and table

iris_exec quandl_schema_creator -- -ns CustomNS -qd XNAS -qt GOOGL -gc ticker -pc data_set -ak <Nasdaq Data Link API Key>

sudo -u dbmerge iris_exec quandl_import -- \
  -dp localhost/2023-01-01 -ns CustomNS -qd XNAS -qt GOOGL -ak <Nasdaq Data Link API Key> -om REPLACE

Troubleshooting

Common error codes

Authentication errors:

• Ensure your API key is included and valid. Missing or incorrect keys result in HTTP 400 errors.
• Double-check for typos or whitespace in your API key.

Permission errors:

• Some datasets require a paid subscription. Make sure your account has access to the dataset you are requesting.

Data not found:

• Verify database and table codes (e.g., 'XNAS' and a valid ticker like 'AAPL') are correct and available in Nasdaq Data Link.
• Use the Nasdaq Data Link Dataset Catalog to look up current database and ticker codes. For US stocks, use XNAS as the database and a valid ticker (e.g., AAPL for Apple Inc.).

Slow imports or timeouts:

• Large datasets may take a long time to process. Use the -lp flag for progress logging.
• Increase the dt (download timeout) argument if needed for large requests.

Where do I find my API key?

• Log in to your Nasdaq Data Link account and visit your dashboard to view or regenerate your API key.

Can I use these tools with legacy Quandl datasets?

• Yes. All existing Quandl datasets are now part of Nasdaq Data Link. Use your Nasdaq Data Link API key.

Related documentation

• Getting Started with Nasdaq Data Link
• Authentication Guide
• Binary logs
• CSV
• JSON
• JDBC
• Parquet
• XML
• Custom Importer
• Deephaven Schemas