Urban Heating Effects

#!/bin/bash

# wget ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt
H=`pwd`
cd ftp.ncdc.noaa.gov/pub/data/ghcn/daily
## New York state:
#stations=(`awk '$5=="NY"{print $1}' ghcnd-stations.txt`)
## New York city:
stations=(`awk '$0~/NY\ NEW\ YORK/{print $1}' ghcnd-stations.txt`)

## This requires having downloaded & unpacked 1.7Gb
# wget -Nr --exclude-directories=grid ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd_all.tar.gz
Tstations=()
for S in ${stations[@]} ; do
   Tstations=(${Tstations[@]} `awk '$1~/TMAX/{print FILENAME; exit}' all/${S}.dly`)
done

## Extract & compose annual temperatures for each station
awk '{ID=substr($1,1,11);stations[ID]=1;\
      Yr=substr($0,12,4);\
      Mo=substr($0,16,2);\
      }\
      $1~/TMAX/{l=length($0); d=1; \
         for(i=22;i<l;i+=8){\
            mx=substr($0,i,5)/10.0;\
            if((mx<100)&&(mx>(-100))){\
               tmx[ID,Yr]+=mx; Nx[ID,Yr]++; \
               if(D[Yr,Mo,d]==""){\
                  D[ID,Yr]++; Day[Yr,Mo,d]=1;\
               }\
            }\
            d++;\
         }\
      }\
      $1~/TMIN/{l=length($0); d=1; \
         for(i=22;i<l;i+=8){\
            mn=substr($0,i,5)/10.0;\
            if((mn<100)&&(mn>(-100))){\
               tmn[ID,Yr]+=mn; Nn[ID,Yr]++;\
            }\
            d++;\
         }\
      }\
     END{printf("Year ");\
         for(I in stations){printf("%s ",I);}\
         printf("Average\n");\
         for(Y=1900; Y<=2009; Y++){\
            printf("%d ",Y); NYavg=0; NYn=0;\
            for(I in stations){\
               if((tmn[I,Y]!="")&&(D[I,Y]>240)){\
                  printf("%f ",((tmx[I,Y]/Nx[I,Y])+(tmn[I,Y]/Nn[I,Y]))/2.0);\
                  NYn++; NYavg+=((tmx[I,Y]/Nx[I,Y])+(tmn[I,Y]/Nn[I,Y]))/2.0;\
               }else{\
                  printf("nan ");\
               }\
            }\
            printf("%f\n",NYavg/NYn);\
         }\
     }' ${Tstations[@]} | sort -n > ${H}/NewYorkTempByStation.txt

Trend is 1.3C/century.  As you can see, New York City had a single entry in GHCN’s temperature data until 1945, USC00305801.  This station is Central Park, and its temperature peaks out from under the average as more stations come online.  The peaks in 1945, 1948, and 1951 are probably due to my weak quality control which keeps data when a station reports more than 240 days of the year.  If a station begins recording in the spring, then records continuously the remainder of the year, this would shift the temperature upward in that year.  Generally, the stations’ temperatures tend to go up and down together.  Let’s see what happens if I require 360 days of data per year…

Trend is 1.2C/century, not significantly different from the data with 240 days/year required.  The long term station at Central Park is visible in this graph.  The stations seem to be in better agreement about each year’s temperature in this workup.  Let’s contrast New York City with the other stations around the globe between 40 and 41 degrees North.

The trend along the rest of 40N is 0.6C/century.  If NYC is the ultimate example of UHI, at least 0.6C/century could be removed from that city’s warming.  I wonder what else is at 40N…   The next comparison is enlightening:  the greater New York City area ( latitude= 40N to 41N  longitude= 73W to 74W) versus all stations in 40N to 41N and everything in 35N to 45N latitude.  Remarkable how representative of the entire 10 degree band the 1 degree strip is.  By contrast, the NYC area is warming at nearly 2C/century.  I no longer wonder why city dwellers believe the whole planet is warming.

## You need to run the script that makes TAvg_1x1.txt first!

## Gather up all the sectors in the 40.0 to 40.99 Latitude range
## Odd side-effect of my gridding method,
## NY NEW YORK CNTRL PRK  lat=40.7800  long=-73.9700
## ends up in the long=-74 sector.
awk 'BEGIN{print "Year 40N 35-45N NYC"}\
     (($1==40) && ($2!=(-74)))){Avg[$3]+=$4; N[$3]++}\
     (($1>=35)&&($1<=45)){Zavg[$3]+=$4; Zn[$3]++}\
     (($1==40)&&($2==(-74))){NYt[$3]+=$4; NYn[$3]++}\
     END{\
         for(Y in Avg){print Y,Avg[Y]/N[Y],Zavg[Y]/Zn[Y],NYt[Y]/NYn[Y]}\
     }' TAvg_1x1.txt | sort -n > 40N_latitude3.txt

Decided to process a few more sprawling urban areas.  Here’s the same analysis style applied to the Los Angeles area.  Since Los Angeles spans a latitude boundary 33.9N to 34.1N, I’ve graphed LA’s temperature using both of the applicable 1×1 degree sectors.  Clearly, a linear regression on the annual average temperatures of the 25N to 35N band is inappropriate, but is shown here as a contrast to the trend within LA.  Within the LA area, temperature is trending upward at 1.8 or 1.3C/century.  Temperature records in southern LA start in 1910.  Removing data points such that Northern LA’s records started in 1910 did not change the trend at all.

I wonder where that oscillation comes from…  Next stop, Tokyo!  Tokyo’s data does not start until 1949, so the graph starts in 1949.  Trend 2.7C/century.

Here’s Atlanta, Georgia and St. Louis, Missouri:

Now that is surprising.  Certainly wasn’t expecting two of the United States largest inland metropolitan areas to have no urban heat island (UHI) effects.  I think I will explore this odd phenomenon further.

Added 1-1-2010 Chasing around UHI, I computed the slope for a linear regression on the annual average temperature for each station.  Unfortunately, some stations have data at both ends, 1800’s & 2000’s with nothing between.  I’ll re-run with some quality control for time distribution of the temperature data.  Here’s a sampling of the stations with the highest positive slope (aka temperature increase over time):

#!/bin/bash

### Crunch through all the daily files and get a linear regression slope
### for everyone with temperatures.  Basic  100 > T > -100
H=`pwd`
cd ftp.ncdc.noaa.gov/pub/data/ghcn/daily/
echo -n "" > ${H}/Regressing.txt
for Station in `ls -1 all` ; do
   ## Going to try & compute a linear regression slope for each station;
   ## slope = (N*(sum(Y*T)) - sum(Y)*sum(T))/(N*sum(T^2) - (sum(T))^2 )
   awk '{St=substr($0,0,11);Yr=substr($0,12,4);Mo=substr($0,16,2);}\
     $1~/TMAX/{l=length($0); d=1; \
         for(i=22;i<l;i+=8){\
            mx=substr($0,i,5)/10.0;Mflag=substr($0,i+5,1);\
            Qflag=substr($0,i+6,1); Sflag=substr($0,i+7,1);\
            if((mx<100)&&(mx>(-100))){tmx[Yr]+=mx; Nx[Yr]++;\
               if(D[Yr,Mo,d]==""){\
                  D[Yr]++; Day[Yr,Mo,d]=1;\
               }\
               if((Qflag==" ")&&(Sflag!="S")){Qtmx[Yr]+=mx; QNx[Yr]++;}\
            }}}\
     $1~/TMIN/{l=length($0); d=1; \
         for(i=22;i<l;i+=8){\
            mn=substr($0,i,5)/10.0;Mflag=substr($0,i+5,1);\
            Qflag=substr($0,i+6,1); Sflag=substr($0,i+7,1);\
            if((mn<100)&&(mn>(-100))){tmn[Yr]+=mn; Nn[Yr]++;\
               if((Qflag==" ")&&(Sflag!="S")){Qtmn[Yr]+=mn; QNn[Yr]++;}\
            }}}\
     END{for(i in tmx){\
           if((tmn[i]!="")&&(D[i]>240)){ \
               if((i>=1961)&&(i<=1990)){nyears++};\
               T=(tmx[i]+tmn[i])/(Nx[i]+Nn[i]);\
               sumYT=sumYT+(i*T) ; sumY+=i ; sumT+=T ; sumY2=sumY2+(i*1.0)^2; N++;\
           }\
         }\
         if(nyears<30){exit};\
         printf("%s %0.8f %f %d\n",substr(FILENAME,5,11),(N*sumYT - sumY*sumT)/(N*sumY2 - sumY^2),sumT,N );\
      }' all/${Station} >> ${H}/Regressing.txt
done
sort -nr +1 -2 ${H}/Regressing.txt > ${H}/Regressing.csv
## Add in latitude & longitude from ghcn-stations.txt
awk 'FILENAME~/Regress/{\
      info[$1]=$2" "$3" "$4;\
     }\
     FILENAME!~/Regress/{\
      info[$1]=info[$1]" "$2" "$3\
     }\
     END{\
      for(ID in info){print ID,info[ID];}\
     }' ${H}/Regressing.txt ghcnd-stations.txt |\
     awk 'NF==6{print}' |\
      sort -nr +1 -2 > ${H}/Regressed.txt

Station    Slope=C/yr #yrs Latitude Longitude
CH000051087 0.11664303  53 46.9800   89.5200
CH000053463 0.09438920  58 40.8200  111.6800
CH000050564 0.08136308  56 49.4300  127.3500
MG000044231 0.07951558  49 49.6300  100.1700
RP000098430 0.07928921  48 14.6300  121.0200
CH000050353 0.07859015  56 51.7200  126.6500
MX000032032 0.07751636  41 21.6700 -103.3500
CH000050915 0.07616788  54 45.5200  116.9700
CH000050434 0.07532965  53 50.4800  121.6800
SZ000008440 0.07411992  49 46.2500    6.1300
CH000052818 0.07323816  55 36.4200   94.9000
CH000053192 0.07237252  57 44.0200  114.9500
CH000054208 0.06978685  56 42.1800  116.4700
USC00419966 0.06961902  69 31.7000 -106.3200
CA005022788 0.06880061  32 50.1200  -97.3300
MG000044218 0.06875581  49 48.0200   91.6500
CH000054102 0.06753824  57 43.9500  116.0700
CH000050527 0.06745366  59 49.2200  119.7500
USC00503933 0.06704211  46 59.5700 -154.4700
SW000002080 0.06693051  48 68.4500   22.5000

The winner is CH000051087 in Fuyun, China with a trend of 0.11C/year = 11C/century. Reviewed the chart for this station at KNMI.  There are several winter lows around -40C and -30C in the late 1950’s, whereas recent winters bottom out around -20C.  Otherwise, the KNMI  data (via GHCN) looks fairly flat since 1965.  I’ll go pull this station’s raw data & do an annual average plot.

Fuyun, China’s slope is driven by the partial data at the beginning.  Let’s plot out CH000053463.

This city’s slope is driven by partial data in the 2000’s.  Check KMNI to see a nice plot of this city’s monthly temperatures.

Advertisements

Responses

  1. […] Urban Heating Effects […]

  2. Interesting stuff Eugene. Keep up the good work. I’m trying to drum up some interest in your site to get some good feedback – don’t seem to be doing a great job though.

  3. […] Urban Heating Effects Posted by: genezeien | December 29, 2009 […]

  4. Nice work so far.

    One thought on why some urban areas show more warming than others: the siting of the actual sensors. The sensors for St. Louis and Atlanta may be properly sited?

    Just a thought.

    • The method I’ve applied uses all the temperature stations within a 1×1 degree sector. This would dampen out a single rogue station that reads 2C above actual temperature on a regular basis. At this point, I am more inclined toward blaming smog (aka low cloud cover) Hmm, average wind speed may correlate nicely with UHI by city. I may have to extract that metric, if it exists in the GHCN files.

      http://surfacestations.org has a nice setup for checking sensor locations, if you’d like to contrast the Atlanta area with Los Angeles.

  5. Is it possible to search for differences in neighboring stations? Let’s say you could make a graph which plots difference of station coordinates vs. difference of temperatures? That approach should be able to nail UHI, not?

    Of course it isn’t as straightforward as I make it sound, but maybe worth a thought about where and how to make use of it?

    Anyway, good work. I like what I read.

    • Ah, I see. You’re looking for a temperature change over some distance. For instance, site A is 50 miles north of site B and site A’s temperature averages 1C lower than site B’s. This graph https://justdata.files.wordpress.com/2009/12/rawtempbylatitude_23.jpg on Step by Step covers that, in a crude fashion. As expected, the further one goes from the equator, the faster the temperature drops. There is a negligible difference from the equator to 10N or 10S, but the average drops 10-15C from 10N to 40N. I would expect east-west differences to be driven by topography; a nearby ocean, lake, forest, or difference in altitude.

  6. Eugene,

    The older and larger a city is, the earlier it maxed out on UHI. There is only so much extra temperature that can be added to a normal weather station. (What is not measured by a max/min thermometer is whether the hot time lasts longer each day. That’s a different concept).

    It’s quite tricky to prove it, but I’d propose that the big cities of the world had achieved a near plateau of UHI max temp effect by the 1920s.

  7. [quote]Ah, I see. You’re looking for a temperature change over some distance. For instance, site A is 50 miles north of site B and site A’s temperature averages 1C lower than site B’s.[/quote]
    Yeah, that would be what I meant. I’m not sure if this approach would work, but my guess is that it should be useful if applied at least to areas densely populated by weather stations, where coincidently we would expect higher absolute UHI differences. (Trends might show stronger in developing world, but absolute values should be higher in urbanized countries.)

  8. I think it is interesting that NYC actually does look linear, far more linear than the equivalent latitude. In fact NYC doesn’t appear to be affected by what is going on at that latitude. Tokyo is a bit similar.

    Most of the other cities seem to follow the same cyclic pattern as the equivalent latitude.

    Hohhot is interesting. It almost appears to be rising exponentially. It could be due to the shorter temperature history (so we only see part of a natural cycle) or to a combination of increasing urbanization and industrialization (or all three).

    So why does NYC appear linear – location? Independence from surrounding land temperature? Dominant ocean temperature reduces cyclic effect for this combination of geography / urbanisation? Suggestions?

  9. Interesting stuff. The cooling from 1940-1970 is very obvious in your latitude band temps, as is the fact that temperatures were often higher than today between 1910 and 1930 (and varied much more). Are these effects real or missing data etc?

    • Are these effects real or missing data etc?
      I think some aspects of the temperature variance are real. Even though there is a lot of noise in the 615 1×1 degree sectors, a 60 year cycle is evident. Only sectors which were continuously represented by at least one thermometer were used. Despite the UHI, thermometer additions/drop-outs, and changes in equipment, the past 10-15 years show a tendency to repeat the downward cycle of the 1940s. I did not eliminate any stations, from any sector, for any reason. Temperatures above 100C, or below -100C were discarded.

  10. One point that I haven’t seen discussed at all is that in North America, at least, there is some correlation between population density and coastal location, especially in the West but also along the Atlantic. Comparing, say, Chicago + St Louis with New York + Miami, is there any correlation of the difference with known Atlantic current phenomena?


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: