Different methods of data download
There are different ways to download data from Meteo-Hub portal:
Web Interface
A registered user can obtain data from one or more datasets of the platform using the Data Extraction feature on Meteo-Hub Web Interface.
How to use Data Extraction Feature
In the first step, the user must select one or more datasets (multiple selection is allowed only with datasets of the same category: observation, forecast, radar).
In the second step, the user can filter respect the provided parameters that are specific for the dataset category:
Category | Filter |
Observation | Reftime, Level, Product, Timerange |
Forecast | Reftime, Area, Level, Origin, Proddef, Product, Run, Timerange |
Radar | Reftime, Area, Level, Origin, Proddef, Product, Run, Timerange |
If the user is allowed to use post-processing tools, they can be applied in the third step. The set of post-processing tools are specific for the dataset category:
Category | Post-processing |
Observation | Derived Variables, Time Post Processing, Quality Control Filter, Format Conversion |
Forecast | Derived Variables, Time Post Processing, Space Post Process |
Radar | Derived Variables, Time Post Processing, Space Post Process |
In the fourth and last step, the user must enter a name for the current request and submit the request to the system. Before submitting the request, it is also possible to schedule it using the “Schedule” button.
After submitting a request, the user is led to the “My Requests” page where, once the extraction has been completed, the output data can be downloaded.
A video tutorial illustrating the four steps of construction and submission of the request is available on MISTRAL project YouTube channel:
A tutorial (Italian language) with further information on data extraction is available:
How to download packages of open data
For all users, even those not logged in, it is possible to download packages of open data. The packages are produced daily and are available on the Meteo-Hub main page.
Currently this feature is available for the following datasets:
- COSMO-2I: COSMO at 2.2km – Italy area
- COSMO-5M: COSMO at 5km – Mediterranean Region
- Multimodel ensemble forecast by Arpa Piemonte
- Surface Rainfall Intensity from Radar-DPC
- Italy Flash Flood
The open data from the observational datasets can be downloaded from the map of Observations webpage.
Meteo-Hub Client
A client application that allows users to perform the main operations on the Meteo-Hub platform from the command line is available in this GITLab repository.
The application allows users to download data extracted by an immediate or scheduled request prepared interactively on the Meteo-Hub web interface. It also provides a wait-and-download functionality for users having access to an AMQP data-ready queue. To have further information on how to have access to an AMQP data-ready queue please contact the Mistral technical team at mistral-support@cineca.it
At the moment, the command-line client does not allow to perform or schedule requests for data extraction that must be prepared interactively through the web interface.
A tutorial (Italian language) on how to use the Meteo-Hub client is available:
API
Users can directly use Meteo-Hub APIs to create and schedule requests of data extraction and to download the output data. The API are documented using the swagger definition at this URL.
How to download the open data using the API
In order to get the list of all the packages of open data for a specific dataset this API has to be used:
GET https://meteohub.mistralportal.it/api/datasets/{dataset_name}/opendata
Where the query parameter dataset_name is the name of the dataset of which the packages list is referred. Currently this functionality is available for all the forecast datasets and the multim-forecast and radar_dpc datasets. Examples on available parameters are:
- COSMO-2I : refers to COSMO at 2.2km – Italy area dataset
- COSMO-5M: refers to COSMO at 5km – Mediterranean Region dataset
- multim-forecast: refers to Multimodel ensemble forecast by Arpa Piemonte dataset
- radar_dpc: refers to Surface Rainfall Intensity from Radar-DPC dataset
- ICON-2I_all2km: refers to ICON at 2.2km – full domain area
- ICON-2I_ita2km: refers to ICON at 2.2km – Italy area
To ask for download a specific package of open data this API has to be useed:
GET https://meteohub.mistralportal.it/api/opendata/{filename}
Where the query parameter filename is the name of the package to download
To ask for download one or more packages of open data related to a specific dataset, a specific reference date and/or a specific run this API has to be used:
Where the query parameter dataset_name is the name of the dataset of which the packages download is requested.
The date for the reference_date parameter can be expressed with the following data formats: YYYYMMDD
or YYYY-mm-dd
where YYYY
stays for year, mm
for month and dd
for day.
The run parameter must be a valid time expressed with the following time format HH:MM
where HH
stays for hour and MM
stays for minutes. The time for run parameter is intended as UTC.
No authentication is required to use these previously described APIs,unless the request of open data download refers to a dataset with restricted access.
How to get an access token
In order to use the authenticated API the user has to provide an access token. The token is obtained as a response to the login API.
POST https://meteohub.mistralportal.it/auth/login
Example of the request body:
{
"username": "user@example.com",
"password": "string",
}
How to create a request of data extraction using APIs
In order to create a request of data extraction the following API has to be used:
POST https://meteohub.mistralportal.it/api/data
Example of the request body:
{
"request_name": name of the request,
"reftime": {
"from": "2022-01-10T11:04:54.615Z",
"to": "2022-01-10T11:04:54.615Z"
},
"dataset_names": [
list of datasets
],
"filters": {
"area": [{}],
"level": [{}],
"origin": [ {}],
"proddef": [{} ],
"product": [{}],
"quantity": [{}],
"run": [{}],
"task": [{}],
"timerange": [ {}],
"network": [{}]
},
"output_format": "json"/null,
"only_reliable": true/false,
"postprocessors": [ object ] //list of requested postprocessors with their specific configuration
}
In the request body the metadata related to the desired data has to be specified. The filters are specific for the dataset category (see table in the previous chapter “Download data from web interface”). Output_format and only_reliable are parameters specific for observations data: the first specifies if the user wants the ouput data in json format instead of the default format (BUFR). The second one specifies if the user wants to download only validated data
To use the previously described API an authentication is required. The access token get from the login API has to be included as Authorization in the request Header .
How to create a scheduled request of data extraction using APIs
In order to create a scheduled request the following API has to be used:
POST https://meteohub.mistralportal.it/api/schedules
The request body, where all the metadata related to the request are specified, is similar to the one described in the previous chapter “How to create a request of data extraction”. Additional parameters has to be added in order to specify the scheduling interval. The available scheduling methods are:
Periodical request: data are extracted at specific time intervals
"period-settings": {
"every": number, // specifies the time interval of the extraction,
"period": string // specifies the time period. Choose one of "hours”,"days",”months”,”years”
}
At-time request: data are extracted on specific days at a specific time
"crontab-settings": {
"minute": number // range between 0-59,
"hour": number // range between 0-23,
"day_of_week": number // range between 0-7 where 0 and 7 refers to sunday, 1 to monday, and so on,
"day_of_month": number // range between 0-31,
"month_of_year": number // range between 1-12 where 1 refers to january and 12 refers to december
}
On-data-ready request: data are extracted as soon as they are available on the platform. This option is available only for COSMO-2I and COSMO-5M datasets.
"on-data-ready": true,
To use the previously described API an authentication is required. The access token get from the login API has to be included as Authorization in the request Header.
How to download data using APIs
In order to get the list of all the submitted requests and their status this API has to be used:
GET https://meteohub.mistralportal.it/api/requests
If the data extraction has been completed the fileoutput parameter in the response indicates the name of the output file.
To download the output data the following API has to be used:
GET https://meteohub.mistralportal.it/api/data/{filename}
Where the query parameter filename specifies the file to download.
To use the previously described API an authentication is required. The access token get from the login API has to be included as Authorization in the request Header.