Data Upload
The Data Grid (TDG) provides organizations multiple ways to upload data to their custom database. To upload data, a user must be a part of the organization they want to upload data to, and be logged in.
Overview
The two ways to upload data on TDG are 1 uploading pre-formatted spreadsheets on www.thedatagrid.org, and 2 uploading programmatically using the API. 1 is the main supported method, optimized for common data collection workflows that use spreadsheets and spreadsheet software like Excel and Google Sheets. 2 is the behind the scenes method, which is useful for those who want to stream data into TDG using their own external software. If you aren't sure which method to choose, it's likely that you should use 1, since this is the method designed for general purpose use.
Spreadsheet Upload
For most organizations, this is the only method that is needed in order to completely and successfully upload data to their database. This method is made to leverage the widespread use and understanding of spreadsheets, allowing users to do in-field data collection with the spreadsheet software of their choosing, and upload those spreadsheet files (.xlsx) to TDG after the fact. Formatting spreadsheets from scratch is tedious, and organizations have already declared their data format when creating their database, so TDG automatically generates pre-formatted spreadsheet templates for organizations to download, input their data into, and reupload. No formatting of columns, data validations, sheets, or formulas is required!
Steps
The following steps detail how to upload data using the spreadsheet upload method:
- Go to the 'Audit Dashboard' page and download a pre-formatted spreadsheet template for an upload type and feature
- TDG will prompt you to download a .xlsx file, which is an open spreadsheet format supported by Excel, Google Sheets, and many others
- Input your data into the spreadsheet with the software of your choosing
- Go to the 'Audit Dashboard' page and upload your completed template
API Upload
API upload is meant for organizations who want to upload data to the database programmatically. Some examples of where this can be useful are:
- An IoT device which collects and uploads in-field data automatically
- A stream where data is constantly uploaded to TDG in order to be as close as possible to the ground truth
- A service which allows users to upload data to TDG as part of a larger system
- A workflow where dealing with raw JSON files is preferred to .xlsx files
Members of organizations who want to use this method must generate a secret API key, which is a passcode that authenticates that they are authorized to upload data. They then use that key to send an HTTP POST request to api.thedatagrid.org with their data.
!Warning: do not share your API key with anyone outside of your organization, post it publicly, or include it in public source code (like websites). Anyone who has your key is able to upload to your organization.
Steps
- If you don't already have an API key, go to the 'Audit Dashboard' page and generate an API key. TDG will show you your key, which you must copy and store securely, since this is the only time TDG will show it to you.
- Collect your data and format it into the submissionObject format below, including your API key in the request header where specified below.
- Send an HTTP POST request to the URL below. See the HTTP response code and body to understand if the upload has been processed.
Object and Endpoint
The submissionObject is the JSON object which is used to upload data via the api. Its structure is specified below:
{
items: {
create: [createItemObject, ...],
update: [updateItemObject, ...],
delete: [deleteItemObject, ...],
requestPermanentDeletion: [deleteItemObject, ...]
},
observations: {
create: [createObservationObject, ...],
update: [updateObservationObject, ...],
delete: [deleteObservationObject, ...]
}
}
Note the flexibility of using the API endpoint. Instead of only allowing a single upload type and single feature per upload like when using spreadsheets, API upload allows for any combination of upload types and features. You can even have dependencies within your submission where one of the uploaded items depends on another.
The endpoint to make the API request to is the following:
https://api.thedatagrid.org/audit/submission
The API key should go in the HTTP header as follows:
X-API-KEY: api-key-goes-here
! You should not be making this API request in a browser with fetch(), this will expose your private API key. If you want to use a web platform, you must use a proxy and make the actual request from a server which can keep the key secret.
The API upload section is a work in progress, more information as well as detailed object documentation will be released soon.