The standard Visual Crossing Weather query page makes it easy to create weather API queries and download CSV data for multiple weeks or months and multiple locations. However, sometimes you need to retrieve more weather data than can easily be fetched via simple browser query. If you need thousands of rows of results, many locations, or a very complex summary query, you may find find that your query hits the limits of a web browser UI or the browser’s fetch timeout. Of course, this can always be solved by writing a small script or a bit of code. However, if you are looking for an even faster, easier way to get large weather result sets without writing any code at all, this article will show you how to do that by using the command-line tool cURL.
What is cURL?
If you are using Unix or Linux, you are probably already familiar with cURL. It is a very useful command-line tool that wraps the programming library libcurl. Short for “Client URL,” cURL makes it easy for anyone to write a simple command that makes web queries and fetches their result data. Most Unix and Linux systems already have a working version of cURL installed and ready to use. If you don’t see it available to you and you have admin privileges, you can install it from any standard repository. If you are not an administrator, however, check with your local system administrator to find out how to get access to cURL.
More recent version of Windows (Windows 10 build 1803 and later, for example) come with a version of cURL installed. To find out if you have it, simply open a command prompt, type “curl,” and hit enter. If you get a help message specifically for the cURL command, you are all set. If not or if you are on some other OS platform, it is easy to obtain cURL for just about any system. There are many web-based binary sources available, and one of the most common can be found here: https://curl.haxx.se/download.html. Simply select your OS and follow the appropriate download link. (Note that this link is to a 3rd-party site over which Visual Crossing has no control. So, as with any internet download, please scan all executables for viruses or other concerns before installing.)
How can I create the URL for my weather query?
As a command-line tool, cURL is designed to automate querying and fetch via various internet protocols. In this article we’ll specifically use “POST” functionality to maximize the size of our query. Note that we could use the more basic “GET” functionality instead, but since GET query length is more limited, and the goal of this article is to download bulk data, which typically involves larger queries, we’ll use POST.
First, we need to build our weather query. We can do so either entirely in Visual Crossing Weather’s web-based query UI or manually as a string. The easiest starting point in either case is in the query UI. To get started, log into Visual Crossing Weather, configure your locations and query parameters, and then click on the “Query API” button at the top of the results page.
On the Query API page, we can see the exact query URL that will fetch the weather result data. Since we want to fetch bulk data using POST, we need to change the radio button below the URL to be “POST” and also change the format radio button below the Data section to be “Raw.” While “Formatted” mode is easier to read, “Raw” mode combines all of the data parameters together for easy use in a tool such as cURL.
The “Base query” area near the top contains the URL that shows where to submit your weather data query. The Data area near the bottom shows the details of the query. This includes the time window for your query, your API key, and your query locations. We will use these two components, Base query and Data, when setting up our cURL command line.
Note that the parameters in the Data section can easily be edited and adjusted to change or expand the query. For example, we can add additional locations simply by adding pipe separated location values into the “locations” portion of the string. This is very common if you want to expand your example query generated by the UI into a bulk query that returns data for hundred or thousands of location. For more information on how to edit these parameters manually, please see our query API documentation.
How do I fetch the weather data for my bulk query using cURL?
Now that we have our weather query defined, we can run it in cURL. First, we need to take the Data portion and save it as a text file. We can either copy-and-paste the data directly from the Data section on the Query API, or we can use an edited version as discussed above. For this example we’ll name this text file “post_data.txt”.
Next we need to compose the cURL command line. While cURL command lines can become somewhat complex if you need to make specialized queries, in this case, we only need to specify the URL and a single command-line flag that points to our file. That flag is “-d” (which is a shortened form and synonym of “–data”).
So, assuming that we saved our Data string into “post_data.txt”, and we are using the standard base weather forecast query URL, we can use cURL to run our query with the following command line.
curl -d @post_data.txt https://weather.visualcrossing.com/VisualCrossingWebServices/rest/services/weatherdata/forecast
Note the ampersand (@) before the text file name. This is required to tell cURL to use the contents of that file instead of just the file name itself.
If you run your query using the command line above, you will notice that the weather results are dumped as CSV text output into the console window from which you are running. While you could manually copy them from the console window into a text file, that is usually not what you want. Luckily, it is easy to get the results directly into a file either via redirecting the “standard output” from cURL using the “>” sign or via the “-o” parameter flag (which is a shortened form and synonym of “–output”).
So, our final cURL command-line takes the general form as follows:
curl -d @<DATA_FILE_NAME> -o <OUT_FILE> <VISUAL_CROSSING_WEATHER_BASE_URL>
Simply replace <DATA_FILE_NAME> with the actual path to and name of your data file, <OUT_FILE> with the name of the results file that you wish to created, and <VISUAL_CROSSING_WEATHER_BASE_URL> with the base URL for your query. Note that this base URL will be different for different types of queries (history, forecast, etc.). So please be sure to copy it directly from the API Query page or the Visual Crossing Weather API documentation. Finally, it is worth mentioning one final time, make sure not to forget the ampersand before the data file name. This is the most common mistake when setting up a POST cURL command line.
What if I need an extremely large weather dataset?
While the cURL method described above will work well for medium and large result datasets up to tens of thousands of rows, some use cases require hundreds of thousands of rows or more. If this is your use case, you have two options.
The first option is to request that Visual Crossing generate the dataset for you offline and send you the results as a file in your desired format (CSV or JSON). This is the easiest path for most people, especially those who only need to run the query a single time. Examples of this use case include “priming” a business intelligence data warehouse with years’ worth of historical weather data or training a machine learning algorithm. Simply reach out to support@visualcrossing.com with the details about your query, and we’ll give you a cost-effective quote based on your query needs.
The second option is to use the above procedure in a loop running one chunk of your entire query with each iteration. This would require a bit of scripting or code to control the loop, chunk the query into reasonable blocks, and then combine the results into a single output file (if necessary). As this requires some shell script, Python, or other scripting tool at a minimum, it is outside the scope of this article. However, it is not difficult if you have some basic knowledge of any scripting environment. You could either script the use of cURL as described in this article or you could use the native URL POST query capability of your coding environment. Please consult your script or code language documentation for specifics.