Create your own datasource with Python, Heroku and mLab

Introduction

In a previous post I have implemented an AngularJS client capable to browse the NYT Best Sellers through the NYT API. Back in time I struggled to find an open REST datasource that can be used to play with different technologies, so I have decided to implement my own CRUD services on top of a Mongo database sitting in the cloud. To do so I took advantage of mLab, which provides DB-as-a-service, and Heroku, which provides Platform-as-a-service.

Resources

All the source code for this project is available on GitHub, so you may also consider to watch, fork or star the project! The application is also under continuous integration on TravisCI and all the tests are available at this link.

Structure of the Project

CRUD services have been implemented in Python through the use of the Flask micro-framework. The project has been structured as follows:

The code is divided into a source (test_data_crud_services) and a test package (test_data_crud_services_test). The source code is then organized into the core, rest and config packages, with a self-explanatory content.

The app.py script is the core of the application, and it is in charge to start-up the Flask framework, register the blueprints, initialize CORS filters and finally run the service. The full script is available below:

The root service return the JSON Hyper Schema that describes the available services, their inputs and related outputs.

Setup the DB with mLab

The first thing to do is to create a free account on the mLab signup page. After that, it is possible to create a new database. To have a free hosting you want to select Single Node and then Sandbox (shared, 0.5 GB) in the creation page. Finally, choose a name for the new instance and hit the Create new MongoDB deployment button.

For this project I have created two instances, one for the production environment and one for the tests.

Once all the instances have been created, it is possible to go back to the MongoDB Deployments page. The website will redirect you to a management page by clicking on any of the instances. This page provides tools to create the users who can access the instance, manage the collections, import/export the DB and so forth.

The most important part of this page is the box on the top that states the MongoDB URI required to connect to the DB. This URI looks like something like:

mongodb://<dbuser>:<dbpassword>@ds059145.mlab.com:59145/kalimadata

We will need this URL in the next step: time to implement the business logic!

Business Logic

The datasource I am implementing is very simple: each collection contains only one kind of object (e.g. a book) and there are no relationships between the different collections.

The business logic is implemented by a single DAO which provides the CRUD functionalities we need. The next table summarizes these functions:

Method Inputs Output
create
  • collection_name
  • item
The id of the new item.
get
  • collection_name
All the items in the collection.
get_by_id
  • collection_name
  • item_id
The item corresponding to the given id.
update
  • collection_name
  • item_id
  • item
Status of the update operation.
delete
  • collection_name
  • item_id
Status of the delete operation.

The dao.py script also exposes another method: get_dao(environment). This method is used to configure the datasource for the DAO, either production or test. Through this differentiation it is possible to run the tests on the DB and keep the production environment consistent. The connection parameters (username, password, MongoDB URI and so forth) are stored in two dictionaries in the settings.py script.

Blueprints

The application has a single DAO, but, through the use of blueprints, it is capable to operate on separate collections. It is possible to define a blueprint for each collection, and keep the same business logic. The following example shows the (partial) blueprint for the books collection:

Each blueprint is then registered on a different endpoint in the app.py script as follows:

Each service requires the environment parameter, in order to select the appropriate datasource, as seen in the previous section.

Deploy on Heroku

At this point, the application is almost ready to be deployed. In order to do so through Heroku, we need to provide two more files. The first one, requirements.txt, is a simple list of the project’s dependencies:

The second file, Procfile, tells Heroku how to run the application, and it contains a single line:

web: gunicorn app:app

Register a free account using the sign-up page of Heroku, and then create a new app. In the app page, it is possible to link the GitHub account in order to deploy straight from there. Finally, just visit to https://YOUR_APP_NAME.heroku.com/ to test your CRUD services!

Conclusions

This post describes how it is possible to create a simple datasource to provide test data with a cloud DB and a simple Python application. The business logic is implemented by a single DAO. Such DAO is then used by multiple blueprints in order to access several collections in the same DB. The MongoDB is hosted by mLab, while the application is deployed on Heroku.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s