This blog shows an example how to work with Google Prediction API on a standard classification example: it is based on the currently running “Give me some credit exercise”from kaggle.com. The example case wants to know (based on a set of defined input variables, how big the probability is that a certain person will go into insolvency during the next two years.

This first part covers data preparation, upload & training preparation, the second part will cover training & prediction with Google’s API and the third part will contain a summary & a recommendation on when & how Prediction API can be used.

Steps needed to use Prediction API

0) Prepare your infrastructure

You need access to two Google APIs: Prediction API & Cloud Storage: Cloud Storage is used to store the data, which are analyzed. There is no option to use another cloud storage provider.

Point your browser to the API Console at https://code.google.com/apis/console/ and create a “project”: the project is a container to cover all API, billing etc. needs for one application. The API Console Project also bundles all oAuth requirements for the Google APIs.

Under the menu item “Services” you can find the list off all available APIs for your project: make sure, that Prediction API and Google Cloud Storage are enabled. (Enabling of Google cloud storage requires you to enter billing information, however the free quota is 5GB (according to https://code.google.com/intl/de-DE/apis/storage/docs/pricingandterms.html )

For easier interaction with the cloud storage it makes sense to use Google’s GSUtil tool (http://code.google.com/intl/de-DE/apis/storage/docs/gsutil.html ): please put special attention to correct authentication setup, as detailed in http://code.google.com/intl/de-DE/apis/storage/docs/gsutil_install.html#authenticate .

The Storage manager web application is an alternative way to interact with the Cloud Storage: further details can be found at http://code.google.com/intl/de-DE/apis/storage/docs/gsmanager.html#using .

1) Methodology

In challenges like mentioned Kaggle challenge the methodology looks like the following:

• Check which information should be predicted (Regression with a function-like outcome y= f(x) vs. classification (choose from a list of discrete values)
• Classification/regression information is developed on a small training set. (normally based on a sample of 30% of the total rows)
• The created algorithms are tested on validity with a separat test dataset (normally based on the other 70% of the rows of the dataset.
• The prediction should match the value of the variable to be checked.

For this example I put the training & prediction part to Google prediction API in order to later test how good the prediction was.    

2) Prepare your data set

Many data sources have some documentation attached, like in the Kaggle example:

The documentation is the start point for the data scientist to see, on what basis he works: likely he will need further explanation & information from the business process owners to get the complete picture of both the task ahead and the information available right now: sometimes more information is needed.

The file cs-training.csv contains 150k rows of data in the structure also described above:

One important precondition to be met for working Prediction API is to have the value to be predicted in the first column: the first column is automatically seen as the only variable, which is to be predicted.

3) Upload your data

Google Cloud Storage can be seen as a large file storage, which organizes groups of related files in buckets: so we need first to create a bucket:

Daniel-Kollers-MacBook-Pro:~ daniel$ gsutil mb -l eu gs://kaggle-exercise
Creating gs://kaggle-exercise/...
Daniel-Kollers-MacBook-Pro:~ daniel$

In this example I named the bucket kaggle-exercise, the “-l eu” to gsutil that I want to have the bucket stored in a EU data center. “gs://” is the standard prefix used to create URL-like bucket names.

“gsutil ls” gives us all existing buckets:

Daniel-Kollers-MacBook-Pro:~ daniel$ gsutil ls
gs://cs-training/
gs://kaggle-exercise/

Now let’s upload the CSV file to cloud storage (with “gsutil cp cs-training.csv gs://kaggle-exercise”):

Daniel-Kollers-MacBook-Pro:Downloads daniel$ gsutil cp cs-training.csv gs://kaggle-exercise
Copying file://cs-training.csv...
Uploading: 7.21 MB/7.21 MB

“gsutil ls gs://kaggle-exercise” gives us a kind of directory listing of the bucket:

Daniel-Kollers-MacBook-Pro:Downloads daniel$ gsutil ls gs://kaggle-exercise
gs://kaggle-exercise/cs-training.csv

So we can see: the csv file is now uploaded.

3) Initiate the training process

The training process can be started on command line interface or a web interface: the web interface is handy for this task and can be seen at http://code.google.com/apis/explorer/#_s=prediction&_v=v1.4 .

As we are not going to use a public dataset we need to “switch to private access”: an authorization dialog checks the user account we want to use.

The main interface of the API explorer shows the available methods:

We select the insert call and need to provide further information in the request body:

The training ID is “credit-training” and th edataStorageLocation is “kaggle-exercise/cs-training.csv” (without the leading “gs://”).

The lower part of the window contains the call result after the click on the “Execute”-button.

Response

200 OK

{

“kind”: “prediction#training”,

“id”: “credit-training”,

“storageDataLocation”: “kaggle-exercise/cs-training.csv”,

“selfLink”: “https://www.googleapis.com/prediction/v1.4/trainedmodels/credit-training“

}

Now the training process started… the status can be checked with the trainedmodels.get call:

The result reveals that the training is still running …hold on to the next part, where training process is monitored and predictions can be made.