Open Source Weather Data?

[freshJet]

Anyone know of some open source weather data, whether it be raw data or a PHP class/XML module?

I'm looking specifically for temperature and wind data, particularly winds aloft at certain altitudes and waypoints.

Grateful for any help.

[BaRBeR]

Dont know whether this is any help to you http://phpstarter.net/2008/12/5-sources-of-free-weather-data-for-your-site/ but might be worth a try

[Tom]

I used this http://aviatorjoe.net/go/wx/

[Vangelis]

I use vatsim weather

[freshJet]

Should state that this is not for METAR/TAF retrieval

[Zach]

• The NOMADS Distribution System, fetching which current winds aloft at waypoints.
  
• Wesley Ebisuzaki, wgrib2 tool used to process the GRIB2 files provided by the NOAA-NOMADS system
  

Take a look at these links @itrobb

Enjoy,

[simpilot]

There is a textual version of the data available for the NOMADS system that I use for my live dispatch system that might help you here -> http://aviationweath...ucts/nws/winds/

[freshJet]

Hmm this could be tricky. Does NOMADS only cover the US?

[Zach]

I've got these links from the simbrief.com website.

http://www.simbrief....ome/?page=about

I believe this is world-wide data, because simbrief can do it worldwide.

Example:

PLUGA
S3536.4
E15441.3
400 238/071 -53
380 237/069 -51
360 239/066 -51
340 240/063 -51

I'd be interested on how you go about pulling this info into phpVMS/your briefing. I have built a dispatch for my VA, and one of only things its missing is Winds aloft, at the waypoints on the route.

[freshJet]

Ah thanks a lot, never noticed they had that page.

[MESTXRX]

Hello!

Did anybody figure out a way to retrieve and use (show) the winds etc from NOMADs?

I've been looking at this since the links were posted, but I just can't seem to figure out how to actually use this...and I'm sure I'm not the only one

Any help would be highly appreciated

Thanks

[SimBrief]

Hi, is there still interest in this? I'm willing to put together a mini-guide on downloading NOMADs winds, but only if there is still interest.

Let me know,

[freshJet]

Hi, is there still interest in this? I'm willing to put together a mini-guide on downloading NOMADs winds, but only if there is still interest.

Let me know,

Absolutely, that would fantastic.

[Zach]

Would be great thanks!

[ln-asm]

That'd be absolutely amazing if you could. Our VA would really love to try to implement this somehow in our systems

[SimBrief]

Alright here goes nothing then. Grab a chair because this will require some reading!

The process for using the NOMADS grib2 files has certain minimum requirements:

• You must be able to install/run the wgrib2 tool on your web server. How this is achieved depends on your OS, I will cover the ones I have worked with later.
  
• Ideally, you need to be able to work with cron jobs if you want the winds to be updated regularly.
  
• If you wish to do this in PHP, you must be able to perform exec() commands. I believe some PHP setups block these commands for security reasons.
  
• The ability to use Google is essential, as I will refer to it often since explaining everything myself would probably make a better book than a forum post!
  

Through the course of this tutorial, I will assume at least a basic understanding of PHP. I will tell you what you need to do in PHP and where, but I won't actually paste specific code that you should be able to come up with on your own.

With that out of the way, lets start with the basics. To obtain the GRIB files that actually contain the winds data, you must download them from the NOMADS-NOAA Distribution system: http://nomads.ncep.noaa.gov/ .

A GRIB file (GRIdded Binary or General Regularly-distributed Information in Binary form) is a concise data format commonly used in meteorology to store historical and forecast weather data. It cannot be read or parsed by text, the data it contains must be extracted by specially designed tools. The NOMADS-NOAA system provides a variety of atmospheric data and models in GRIB format.

For our purposes, we will be using the GFS (Global Forecast System) model. It is available free of charge. The NOAA releases a new complete GFS atmospheric model every 6 hours, in varying degrees of resolution (0.5x0.5,1.0x1.0,and 2.5/2.5 degree grids). For example, the 0.5x0.5 degree grid gives data for every half a degree of lat/long on the earth's surface. This is more precise, but obviously, it means the file will be much larger as it contains more data.

Every GFS cycle includes a forecast of up to 16 days in the future. This is achieved by releasing a group of GRIB files, one for the current time of release (0 hours forecast), right up to 16 days in the future (384 hours forecast), in 3 hour intervals (so a separate GRIB file for 0 hours, 3 hours, 6 hours, 9 hours, and so on). Point is, if you want to be including forecast winds, you're going to be downloading more than 1 GRIB file at a time, and you'll need to parse each one individually.

With that out of the way, let's move onto actually downloading the files. If you were to download the raw GRIB file for a given GFS cycle and hour forecast, it would be rougly 15-20MB. Now because that is quite large, and because we won't be using most of that data anyways (do we really need to know the height of the planetary boundary layer?), the NOAA has provided a handy Perl script. Using this, you can download a GRIB file containing only the fields that you intend to read from it, thus reducing the file size considerably.

That tool can be found here: http://nomads.ncep.n...n/filter_gfs.pl . Note that this is for the 1.0x1.0 degree GFS model, you can access other models by clicking the "grib filter" link for a specific model on this page: http://nomads.ncep.noaa.gov/ . One great feature from this tool is that it can either download the file with the options you selected, or it can output the link to download the file, which is great for programming your software to automatically download files from this site. For example, if I were to download the wind vector for the 500mb layer for the latest GFS model at the 0 hour forecast, the link would look like so:

http://nomads.ncep.noaa.gov/cgi-bin/filter_gfs.pl?file=gfs.t12z.pgrbf00.grib2&lev_500_mb=on&var_UGRD=on&var_VGRD=on&leftlon=0&rightlon=360&toplat=90&bottomlat=-90&dir=%2Fgfs.2014031712

By using this tool, you will be able to generate several links for different GFS models, forecast periods, and parameters, and by examining those links you will be able to program your application/script to generate them as required (for example, the link above would download data for the March 17th 12Z model, you'll need to program PHP to insert the proper date into this link). More on what options to choose a bit later.

If you want to have the winds automatically updated on your server whenever there is a new GFS model available, then you'll probably have to run a script every 6 hours (preferably using a cron job, but that's up to you). SimBrief has a PHP script that runs every 6 hours, it is layed out basically as follows:

• Generate the link for the latest GRIB file containing only the options you want. For example, insert lines such as (floor(gmdate('H') / 6) * 6) into your link to have it point to the latest GFS release that's available (either 00,06,12,or 18 Zulu).
  
• Actually check to make sure that file exists (!!!). This is definitely necessary, sometimes files aren't available as early as you expect them, and you should handle that error. Normally using get_headers() on your link can reveal any problems before actually downloading the file.
  
• If the link isn't available, look for the next latest available file 6 hours earlier.
  
• When found, download the file and save it onto your server.
  

So let's now assume your PHP script downloads and places the latest grib file onto your server every 6 hours. Now you need to actually read data from it, this is done using the wgrib2 tool. http://www.cpc.ncep..../wesley/wgrib2/ . It can be downloaded for several operating systems from various places on the internet, a simple Google search for your server's OS should find it easily (i.e. wgrib2 CentOS).

Installing it is normally fairly straight-forward. In FreeBSD, it was as simple as dropping the wgrib2 binary file onto the server. In CentOS/Redhat it must be installed using an .rpm file, which requires root access. In windows, I believe it requires cygwin.

With that installed, you can now extract data from the GRIB file on your server. How and when you do that is really up to you. You can either parse the entire file every 6 hours (using another cron job, or even the same one if you'd like) and dump the wind data into a database or text file on your server, or you can have wgrib2 read the GRIB every time your system wants to get wind information. wgrib2 isn't very fast when called repeatedly for small requests, so I don't recommend the second option unless you don't plan on using the winds too often.

In my case, I have been using PHP's exec command to execute wgrib. For example, if I wanted to get the data at coordinates 45N 075W, this would be my PHP command:

exec('WIN32wgrib2.exe -s grib/master_1373479200f09.grb -lon -75 45',$output);
$output = implode("\n",$output);
print $output;

To break the command down, "WIN32wgrib2.exe" is the location of my GRIB binary (on a Windows OS in this case. In FreeBSD I was writing simply "wgrib2"). "-s" tells wgrib2 to print out a simple inventory. "grib/master_1373479200f09.grb" is the location of the GRIB file on my server. "-lon -75 45" is the lat/long I'm looking for. Note that the longitude is placed before the latitude for this option.

In turn, that command returns this:

1:0:d=2013071018:TMP:150 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=213.3
2:64634:d=2013071018:UGRD:150 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=28.49
3:257012:d=2013071018:VGRD:150 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=14.89
4:431947:d=2013071018:TMP:200 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=223.3
5:502533:d=2013071018:UGRD:200 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=41.29
6:705030:d=2013071018:VGRD:200 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=21.77
7:893367:d=2013071018:TMP:250 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=234
8:964523:d=2013071018:UGRD:250 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=44.5
9:1074043:d=2013071018:VGRD:250 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=20.9
10:1182352:d=2013071018:TMP:300 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=239.9
11:1253170:d=2013071018:UGRD:300 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=29.7
12:1366222:d=2013071018:VGRD:300 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=24.5
13:1477910:d=2013071018:TMP:400 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=253.1
14:1548096:d=2013071018:UGRD:400 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=28.9
15:1658925:d=2013071018:VGRD:400 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=7.7
16:1768551:d=2013071018:TMP:500 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=263.7
17:1840399:d=2013071018:UGRD:500 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=23.5
18:1949209:d=2013071018:VGRD:500 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=3.3
19:2056759:d=2013071018:TMP:600 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=272.4
20:2131428:d=2013071018:UGRD:600 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=22.6
21:2239982:d=2013071018:VGRD:600 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=6.8
22:2347427:d=2013071018:TMP:700 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=279.3
23:2428877:d=2013071018:UGRD:700 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=21.1
24:2540109:d=2013071018:VGRD:700 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=7.3
25:2650371:d=2013071018:TMP:850 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=288.4
26:2744036:d=2013071018:UGRD:850 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=13.8
27:2858643:d=2013071018:VGRD:850 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=1.5
28:2973369:d=2013071018:TMP:1000 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=294.6
29:3064198:d=2013071018:UGRD:1000 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=5.1
30:3270712:d=2013071018:VGRD:1000 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=1.27

Don't panic, it isn't as complicated as it looks. Because of the "-s" option, it has returned every data value for every height that is contained in the GRIB file at the lat/long I specified. In this case, when I had downloaded this GRIB file I had specified the TMP, VGRD, and UGRD for the 1000, 850, 700, 600, 500, 400, 300, 250, 200, and 150 mb (millibar) level. Furthermore, I had downloaded the 9 hour forecast GRIB file, as indicated. Let's look at a single line now:

1:0:d=2013071018:TMP:150 mb:9 hour fcst::lon=285.000000,lat=45.000000,val=213.3

Each datablock is separated by a colon ":". There is a lot of meaningless gibberish to sift through in this case, but the important ones are "TMP", which signifies this is a temperature value, "150 mb" indicates the level, or altitude this temperature is for, "lon" and "lat" are self-evident, and "val=" gives the actual value for these parameters, in this case 213.3 Kelvin (converts to -60C or so).

VGRD and UGRD are the vector components for the wind at a given location. This is how GRIB files store wind information, so you'll need to derive the wind speed and direction from these vector components using a bit of trigonometry. Google this if you require a more detailed explanation. Just note that this will give a wind speed in m/s, which will need to be converted to knots. Note also that the angle produced will indicate where the wind is going, not where it's from, so you will need to subtract 180 degrees for aviation use.

As far as altitudes go, GRIB files store these as pressure levels, so you will need to convert them to an actual altitude. Generally speaking, the 1000mb level equates to sea level, 850mb equates to 5000ft, and so on. A Google search will turn up a conversion chart you can use. It's worthwhile to note that the pressure levels aren't at fixed altitudes; they move up and down all the time (at the heart of a strong hurricane, 850mb might be closer to the ground than it is to 5000ft), so any conversion charts you find on the internet will be intended as ballpark figures. You have 2 options, either use these ballpark figures and simply assume that 850mb is at 5000ft (much simpler, and it's no big deal if the wind speed you report at 5000ft is actually the wind speed at 3000ft. This is only a forecast after all and the winds rarely vary that much in only a couple thousand feet), OR you can return the HGT parameter for each millibar level and interpolate the winds.

In closing, there are many many levels and parameters you can choose to include in your GRIB files. I can't define them all, but there is a chart available on this website: https://icdc.zmaw.de/gfs.html . This page might also be good for some further reading, I covered a lot of it already but it might still shed some light on the finer points: http://www.cpc.ncep....rib_filter.html .

So that took longer than I expected. Let me know if you have any questions (I'm sure there will be a few), but please bear in mind that I can't write your script for you. You will probably need to do a lot of trial and error and make a few mistakes to get this running properly on your specific system and server configuration.

Best of luck!

- Derek Mayer

- SimBrief.com

[freshJet]

Thanks a lot Derek, I hope this acts as a breakthrough in the development of my redesigned OFPs.

[SimBrief]

No worries, if you have any questions let me know!