RSS

Manage Google Cloud Storage with Python

07 Apr

Download and install GSUtil.

GSUtil is bundled in a single archive named gsutil.zip. Unpack the archive in a convenient location, such as C:\GSUtil, see also https://developers.google.com/storage/docs/gsutil_install

image

Open a cmd window, and cd to the installed folder:

cmd> cd gsutil
cmd> python gsutil config

Follow the instruction in the command window, and copy the URL as suggested and paste it in the web browser:

image

image

Click continue. Paste the authorization code obtained above in the cmd window. Follow the rest instructions in the cmd window.

image

Visit http://code.google.com/apis/console, and click the services tab as shown below:

image

Then click the Google Cloud Storage

image

Copy the x-ggo-project-id back to the cmd window

image

Boto config file will be created and saved to “C:\Users\YourAcoount\.boto”. You might have a look at it with any text editor if you like, and for advanced users, you can customize the configuration as well.

image

image

Testing gsutil

in cmd window, type:  cmd> “python gsutil ls”
Then all you existing bucket in Google Cloud bucket is listed. Success!

image

Program using python

Now it is time to run your python code to automate your task. Test the code in https://developers.google.com/storage/docs/gspythonlibrary, it is fun.

You may encounter a few problems, for instance: “No module named oauth2_plugin” or sth like that. And the workaround is to add your gsutil installation path to the system variable “PYTHONPATH”.

image 

image

Close your old cmd window and open a new one. This is important since the old cmd window does not know that the PYTHONPATH has changed, and to reflect this update, you have to spawn a new cmd.

Rerun below python code, and you will find two bucket is create in your google cloud storage.


import StringIO
import os
import shutil
import tempfile
import time
from oauth2_plugin import oauth2_plugin
import boto

# URI scheme for Google Cloud Storage.
GOOGLE_STORAGE = ‘gs’
# URI scheme for accessing local files.
LOCAL_FILE = ‘file’

now = time.time()
CATS_BUCKET = ‘cats-%d’ % now
DOGS_BUCKET = ‘dogs-%d’ % now

for name in (CATS_BUCKET, DOGS_BUCKET):
# Instantiate a BucketStorageUri object.
uri = boto.storage_uri(name, GOOGLE_STORAGE)
# Try to create the bucket.
try:
uri.create_bucket()
print ‘Successfully created bucket “%s”‘ % name
except boto.exception.StorageCreateError, e:
print ‘Failed to create bucket:’, e

Cute! It works! You can now create, delete cloud buckets and copy/move files to and from local file system and remote cloud storage systems!

image

image

 
Leave a comment

Posted by on April 7, 2012 in Programming

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: