Tragedy in Japan and how to help

Like everyone, I am horrified by the loss of life and homes in the recent devastation in Japan. The scale is enormous, and as we hear more about it, many of us feel hopeless to help and scared about what it means for the future. Aftershocks and worries about radiation are all over the news, along with calls for donations, and occasional worries about scams, people promising to help, but taking money anyway.

At the same time, I am consistently impressed by the people who jumped in to help. From the first responders, the military, to the nuclear workers working day and night to prevent greater tragedy. There are people on the ground in Japan doing great work, mapping transportation routes, helping people find loved ones, providing food and shelter.
The Google Crisis Response team has been putting together many of these resources here: http://www.google.com/crisisresponse/japanquake2011.html, including ways for people to donate money, maps of the situation, and links to the Japan Person Finder app.

The Humanitarian OSM Team is also on the ground, with more information here: http://openstreetmap.jp/crisis/ about how to help with their mapping efforts.

Millions have already been donated. Millions more will be needed. Please help out now. If you want to know more about what Google is doing, follow @earthoutreach on twitter.

If you have other creative ways to contribute, please feel free to leave them in the comments.

Ruminations on Place

Sunday, I was on a panel with Steve Chase, Hurricane Chase, and Murata Takehiko at the Yahoo! Japan Geolocation Conference (#geoconf). Yahoo! Japan (a distinct entity from Yahoo! Inc. btw), put on an excellent conference, and had announced that they would be contributing data to Open Street Map, in a similar fashion to Bing and MapQuest. There was a lot of excitement in the room, with many OSM participants present (by design I imagine). And there were inevitable questions about Google's future involvement in the project - something I'm interested in personally but can make no commitment to on behalf of Google of course. But more importantly, to this blog post at least, a question came up about whether Google would contribute Street View imagery to OSM, and whether there were any OSM related projects developing open versions of Street View. As a panelist, and someone who can't avoid talking, because I'm just that way, I had to answer of course. My answer went something like this, speaking as myself not foreshadowing a future Google product:

I don't think in the future, Google should have to be driving all the streets in the world. Ideally, we will move toward a time in which everyone is providing data about the world in open formats, in ways that are search-able and crawl-able. What makes Street View so compelling, I think, is that it gives us a sense of place, a sense of the essence, be being-ness of the location we're looking at. And in fact, in places where we don't drive cars, you can still see geolocated photos. Look at Moscow, for instance, with Pegman selected. There are photos all over. Ideally, this is something self-perpetuating, something people want to open up.

That got me to thinking, especially after reading Ed Parson's recent post about a 1930's video of Teddition. He, and the commenters, talks about what can be seen there, and what has stayed the same. Ed ends with this statement:

"We are lucky to have rare video like this for it’s completeness but at the same time it’s disappointing that for future generations Google Street View which could offer a similar resouce has had to be mutilated to accomodate privacy concerns."

Indeed, there is something we miss out on. I am not arguing for doing away with privacy, I'm just ruminating here on pictures and meaning. The word "place" has many meanings in English, but they are interrelated. The reason satellite imagery, and particularly street view imagery is so compelling is that it relates something that standard maps can't, a sense of the place or essence of a location. And that place-ness, that sense that is tied to location, time, people, trees, buildings, smell, sounds, all of that is incredibly powerful, and I think Street View is only starting to touch on what that means to people.

I've been in Japan this last week, and at two GTUG meetings, I did demonstrations of custom Street View. In particular, people were interested in this simple holiday greeting from Digitas. In it, the developers added some panoramas to their Street View application through the API, which of course anyone could do. The panoramas lead off street into the digitas office where employees hold up signs and there are thought and word bubbles around them. This conveys a sense of the place, a sense of celebration and fun and associates it with Digitas. I hope it really is that fun to work there, they seem like really nice folks in the picture!

The point I'm trying to make is that by this simple set of panoramas and a few lines of code, they were able to extend the sense of place that is the city around them, and change it to create a view of the place that is their company.

I know there are lots of academic writings on this subject, but frankly since 1994 I get bored when post-modernism or marxist dialectics get mentioned, so my forays into those studies have been unsuccessful at moving me. I read the works of China Meivelle, or Armistead Maupin instead, each of whom conveys vividly this sense of place

Today, like most days I have free in cities not my own, I walked. I walked around Kyoto. Not to find temples or castles, but to find that sense of place. I walked around, a lot, often going back over the same streets, visiting occasionally the same shops. I tried my extremely limited Japanese, and people graciously helped me out. I walked not just the big streets, but the small ones, gathering into myself my own impressions, my own sense of Kyoto-ness. Of course, this is very different from living here, what the place is like, and especially different from being a native. I tried to take pictures (posting them later) that captured this sense. How will technology solve this? Street View, satellite imagery, they are a start, but I believe augmented reality applications are going to be the best bet for capturing the "-ness" of a place. And I believe at some point, most of us will be involved in that capturing. The popularity of check-in applications, the rise of geotagged photos and videos, providing reviews, Foodspotting, and more are already starting it. It's only been 6 years since Google Earth and the Maps API came out, I wonder what we're going to see in another 6 years.

Private Tables in Fusion Tables: Better privacy for your data

Back in June, I wrote about "Techniques for protecting your data." Yesterday, Google announced another way to protect your data, Protected Map Layers.

Protected map layers, a part of the Google Maps API Premier package, allow you to create a data table in Google Fusion Tables and keep that table private. Maps API Free allows you to use only tables marked Public. Protected map layers are rendered as clickable rasters. You still have to be careful what data you put into the infowindow, make sure it's only what you want public.

Of course, Maps API Premier also allows you to make maps that are password protected, so this will be really useful for enterprises that want to keep their data entirely private but still take advantage of the performance improvements and spatial queries that are allowed by using a FusionTableLayer.

JSON in KML Templates

Since Google Earth 5.0, the description balloon has handled pretty close to full JavaScript. There's some interesting things you can do with this. In this example, I'm showing a very basic use of JSON using KML balloon templates.

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2">
  <Document>
    <Style id="testStyle">
      <BalloonStyle>
        <text><![CDATA[
          <script type="text/javascript">
             function loadData(){
               var foo = $[foo];
               var myObj = foo.myData[1].value;
               document.getElementById("example").innerHTML = myObj;
             }
           </script>
           <body onload="loadData()">
             <div id="example">test</div>
           </body>
          ]]>
        </text>
      </BalloonStyle>
    </Style>
    <Placemark>
      <styleUrl>#testStyle</styleUrl>
      <ExtendedData>
        <Data name="foo"><value>{"myData":[
          {"value":"1"},{"value":"50"}]
          }</value></Data>
        <Data name = "bar"><value>23094</value></Data>
      </ExtendedData>
      <Point>
        <coordinates>0,0</coordinates>
      </Point>
    </Placemark>
  </Document>
</kml>

Balloon templates are designed to allow you to use a single <Style> element for all or a set of your <Placemark;gt; elements, and then just insert <Feature> specific data into the template. Named <Data> elements inside a <ExtendedData> element provide name/value pairs. However, this can be a little flat. You can put JSON elements inside the <value> element (a child of <Data>, which gives you a lot more flexibility.

The good folks at Secorra provide ObsKML, an ocean observations format, using this technique.

Interactivity with FusionTableLayer and mouse clicks

The FusionTablesLayer in the Google Maps API is a bit of a black box. Fusion Tables generates a clickable tile overlay to use place in your Maps API application, but you don't get direct to change anything. If you want to change data, you have to do server side calls. However, the FusionTablesLayer does give you access to row data on click, which allows you to do interesting things server side.

In my sample interactiveftlayer, I use a Fusion Table of Google's corporate addresses. The table has a simple Color column, with values of 0 or 1. In the Fusion Tables map visualization for the table, I configured the Marker Icon such that values of 0 are green, and 1 are blue. The color means nothing of course about the actual office.

In interactiveftlayer, I suppress the InfoWindow by adding a suppressInfoWindows: true option to the layer initialization. I then add an event listener to the layer that captures the click event on the layer. This gives me row information about the feature clicked. I then send a XMLHttpRequest to the server (in this case App Engine) with the other color (1 if the color is 0, 0 if it is 1). After the response from the XMLHttpRequest, I reload the layer.

There's one tricky part, and hopefully we can find a way to improve this on the Maps API end. The FusionTableLayer is cached by the browser. Which means that even if the data has changed, the layer stays the same, at least at zoom levels visited while it was visible. This includes, BTW, not only the images but the row data associated with them. In order to defeat that caching, I append a Where parameter to the layer selecting for a Id greater than a random number from -1000 to 0. Since I know all Ids are greater than 0, I can do this. I'm not proud of that strategy, but it works.

Two Thumbed Closure Slider for Time Slider

I've posted a couple of times about sliders - Playing with Closure UI Library: Slider and a basic Basic Time Slider in Closure. My final slider example uses the same data set as the last, and incorporates a Two Thumb slider, meaning you can use the slider to set a range of values, not just a less-than or greater-than value. I also corrected something that impacted performance in the last sample, that is I checked whether the query changed by the different events, and change the query on the map layer when the query changes from the last. This is an artifact of the events that I'm listening form, MOUSEUP, MOUSEOUT, and KEYUP. You may remember in the first sample, I realized that firing off a query change whenever the slider had a CHANGE caused too many queries to hit the overlay server, causing the Map to show the missing overlay error. Listening for mouse and key events caused fewer events to fire, but still more than one per change. The change simply tests to see if the new query is different from the old query, and only re-query the layer server if the query changes.

It's 2011: Time to think about the future of Geo

It's fairly traditional to start a year with either a retrospective of the previous year, or a look forward. Or, of course, both. In that tradition, I'd like to start the year thinking about some trends that I think are going to be important for Geo this year.

Powerful Easy Analysis Tools

This is the year. In 2005, the Google Maps API and Google Earth broadened the use of geography tools far beyond the traditional GIS crowd, sparking a debate between so-called neo- and paleo-geographers that lasted for years. In the last year or so that debate seems to have calmed down a bit, as traditional GIS tools adapted to the web, and neo-geographers, were everywhere. So there's room for a fresh controversy.

The Google Maps API, followed shortly by a host of other APIs from Yahoo!, Microsoft, OpenLayers and others, allowed developers to easily place maps on their site. But as I pointed out in my Ignite Spatial talk in September, developers and GIS professionals aren't the only ones who want to share spatial data. In fact, I'm guessing the vast majority of spatial data, by volume if not quality, is in tabular form. Geocommons has recognized this for years, providing easy tools for uploading, combining, and sharing spatial data. With the addition of Google Fusion Tables, and easy mapping of spreadsheets and sharing of data, powerful tools for data analysis are in the hands of anyone with a Google account. I won't call them "low-end" tools, though certainly they lack the power of ESRI's tools, or any of a host of other proprietary and open source GIS applications. I predict that this year will see a lot of people migrating to Fusion Tables, and others using the API to back-end store the data. Which leads us to

Cloud


OK, I almost had to slap myself for saying "Cloud." After all, of all the buzzwords going around, I think it is the least penetrable to people not "in-the-know" and perhaps has the most number of definitions. To make matters worse, Microsoft has diluted the term even more with this wacked commercial campaign.  However, it is being used a lot, so let me be clear, I think this is the year of Cloud Data Hosting.

I predict more and more spatial data will go into "The Cloud." We're already seeing that happening with services like SimpleGeo, Microsoft Azure's support for spatial data, and many other services. Fusion Tables of course has an API which I anticipate will be useful for a number of spatial data storage services.

Cloud Analysis


It's still early days on this. I think that 2011 will be the year of early adoption. In particular, tools like Google Earth Engine will allow you to run high-end analysis in cloud data centers. Some of these tools are already out there, but the introduction of Earth Engine allows you to do things we're used to in the spatial world, namely raster data analysis, but do it faster and cheaper than before.

Location


Well, really, who am I kidding? 2010 was the year of local and location. Facebook's Places and Places API were a very powerful entrance into the location and local scene. Google Places with Hotpot is a big entry into the local market, but it came pretty late in the year. 2010 was about local, 2011 local will become mainstream, such that everyone will have forgotten that it wasn't part of our sites. Remember when there weren't maps everywhere? That was only 6 years ago, now it's taken for granted. Local will get that way by 2012. When of course the world ends, right? That's what the movies tell us anyway.

Prediction posts are fun, because rarely are you held to them. But really they tell you more about where you are now. The things we can't talk about, or don't know about yet, those are the real surprises. Happy New Year everyone.

Basic Time Slider in Closure

Earlier this month I was playing with a the Google Closure library and using the slider. I wanted to play with it for developing a time slider too. Since I'm going to History Hack Day at the end of January in London, I thought I'd show off some samples there and this was a perfect hack for that.

Anyway, it's pretty simple. You set minimum and maximum values on the slider that correspond to the millisecond values used by JavaScript to represent time. Then use the JS Date object to convert those to readable date and time for display under the slider. You can see the sample here. I set the slider to between Wed Dec 31 2003 23:59:59 and Fri Jan 01 2010 00:00:00 since I knew all dates in the underlying table fell between them. I used another Wikileaks table, the Afghan War Diary, 2004-2010 table.

The tricky part is that Fusion Tables doesn't document the date formats that it accepts for queries. We just haven't gotten around to changing the docs, but I happen to know that we accept these formats:

MM/dd/yy
MM-dd-yy
MMM-dd-yy
yyyy.MM.dd
dd-MMM-yy
MMM/yy
MMM yy
dd/MMM/yy
yyyy

AGU 2010: Data Visualization

So, this was the first year without a digital globes track. I think two things happened:

1. Organizers got tired of doing it every year
2. More important: Digital globes are now all over the American Geophysical Union's annual meeting. There's really no need to separately call it out.

So this year, I presented to Visualization Aided Data Analysis: Tools and Techniques for the Geophysical Sciences. Smaller than my other years at AGU, this was still a vital record of the different tools needed for data analysis. I was particularly impressed by Hank Childs' presentation on VisIt: A Tool for Visualizing and Analyzing Very Large Data, which really seemed to be a tool for visualizing anything in any possible way. Wow, awesome tool. I was also really impressed by Vis Research: Advances in Visualization Research by Ken Joy, U.C. Davis. He had really impressive visualizations of flow patterns.

My own discussions of Fusion Tables, Earth, and Earth Engine seemed a bit more basic, but perhaps that's the point I've been trying to make, that visualization tools can be powerful and hard to use, putting incredible versatility and detail in the hands or professionals, or easy to use and basic for everyone. And when they are easy and basic, everyone can use them effectively.
 Anyway, here's my slides:

Freebase Meetup and JSONP

So, I was invited to speak at a Freebase meetup by Kirrily Roberts. I thought "Cool, Freebase. Oh, I should learn that!" Especially now that Freebase is part of Google! So apologies if you're looking for my intended next sample which was closure as a time slider, that will come.

I did a couple of maps mashups to try and learn the query language. But at the same time, I wanted to learn more about Google Closure. So, here's a couple of samples that I did. Let me explain in further detail:

<link href="http://code.google.com/apis/maps/documentation/javascript/examples/standard.css" rel="stylesheet" type="text/css" />
<script type="text/javascript" src="http://maps.google.com/maps/api/js?sensor=false"></script>
<script src="goog/base.js"></script>
<script>
   
goog.require('goog.net.Jsonp');
var obj;
var map;
 
function initialize() {
  var latlng = new google.maps.LatLng(0,0);
  var mapOptions = {
        zoom: 2,
        center: latlng,
        mapTypeId: google.maps.MapTypeId.ROADMAP
  };
 
 
  map = new google.maps.Map(document.getElementById("map_canvas"), mapOptions);
  getEvents(); 
} 

function getEvents() {
  var envelope = 'http://www.freebase.com/api/service/mqlread?query={%20%22query%22:%20[{%20
        %22type%22:%20%22/time/event%22,%20%22id%22:%20null,%20%22name%22:%20null,
        %20%22included_in_event%22:%20[{%20%22name%22:%20%22world%20war%20ii%22%20}],
        %20%22locations%22:%20[{%20%22geolocation%22:%20{%20%22latitude%22:%20null,
        %20%22longitude%22:%20null%20}%20}]%20}]%20}';
  getData(envelope,'');
} 

function getData(dataUrl,data) {
  var jsonp=new goog.net.Jsonp(dataUrl);
  jsonp.setRequestTimeout(100000);
  jsonp.send('',setObject);
}

function setObject(response){
  obj = response['result'];
  addMarkers();
}
function addMarkers(){
  for(var i in obj){
    for(var j in obj[i].locations){
      var latLng = new google.maps.LatLng(obj[i].locations[j].
            geolocation.latitude,obj[i].locations[j].geolocation.longitude);
      var marker = new google.maps.Marker({
          map: map,
          position: latLng,
          title: obj[i].name
      });}
      console.log(obj[i].name);
  }
}

So what I did there was use goog.net.jsonp from Closure to load the JSON from the query results and then create markers. The first sample finds events during World War II and find their geolocation. Since some events have more than one geolocation, it gives each location a marker with the same title, which will show up as a tool tip on hovering over the marker. In this case, the event contains a /location/location node which has /location/location/geolocation nodes. Someone comment if I'm not getting that quite right.

For those that don't know, I used the JSONP utility to get around the same domain policy of browsers, which prevents you from loading scripts directly from domains other than the one you are loading from. Server side scripting would have solved that issue for me easily, but I decided to set myself the additional challenge of finding that workaround. JSONP takes advantage of the caveat to the same domain policy that allows you to load scripts into <script> tags in HTML. In the case of the Closure library, it creates a temporary <script> tag, uses it to get the JSON object, and then gets rid of the script tag. One must exercise caution when using this approach, of course, that the source of your JSON is trusted to prevent attacks on visitors to your site.

My second sample took a /location/location, in this case Berkeley California (where I grew up) and found all /location/location nodes that had a containedby relationship to Berkeley California. And then mapped them. Then I added in animation to, so if you click on one place it bounces. Just for fun.

It was pretty basic. Figuring out the query language is not easy. Actually, before I spoke at the meetup, Jamie Taylor did a great talk about the schema which really cleared up some things for me. Unfortunately, we experienced a complete A/V failure in the room the meetup was in> Jaime did his talk with a white board, which I think made him happy. I jokingly referred to my talk as interpretive dance. I described my approach, why I was using JSONP, what tools were available, and then we went and had a beer.

Update

I should have credited this article on JQuery and JSONP with part of my inspiration.