github twitter linkedin email rss
Visibility of Status Endpoints
Dec 19, 2011
5 minutes read

Following last month’s work to introduce more standard status endpoints on the services I ended up working on a way to make it more visible.

Initially I looked into existing open source dashboards to have an idea of possible cases and requirements, the best one in my opinion was stashboard (working example here).

It reveals a REST based solution of the status endpoints and to me it brings the following advantages:

Clear and clean UI that feeds of the Service (splitting UI from status endpoints logging and logic). Service allows for OAuth. You can POST into the service and add new projects, it defines statuses as three resources - Services, Events, Statuses. Based on Amazon’s status page. I found it to be a great way to define statuses and decouple the way you add data to it from anywhere, you can easily bring in a new way to define the current status.

Unfortunately it didn’t suite our requirements but I was able to spike an initial project (Github) that worked the same way by actually recreating very similar endpoints based on theirdocumentation.

OAuth could only use one consumer, which was hard coded in the config - In this feature I decided to ignore OAuth completely for now. It misses resources like Environment (Systest, UAT, Live, Dev, etc) - this was completed. Needs the ability to add extra configuration information like Host Headers - this was also completed. No way to filter out services or group them, in our case we have dozens of machines spread through different applications and environments and would be useful to show just Live or just API deploys - the Application resource was added. It was developed in Python and running on the Google App Engine - I really wasn’t happy with having to be restricted to this closed environment even if I can write clients on any language. In this project the concept of Service represents a specific deploy of an Application on an Environment.

Technology-wise I decided to use Sinatra which I had used before as a quite light and fast developing RESTful approach to services and MongoDB for the data just as a way to try out NoSQL.

As a side-note on MongoDB I can see why so many people are joining this sort of database, it’s very well integrated with Linux and Ruby and I found it much easier to set-up than something like MySQL - It just works out of the box with a couple lines to connect and the schema is built during the usage - How well it would behave in the long run is a completely different question.

In the Rakefile I added a few tasks for tear down of all data and build default types for Status, Environment - these tasks are just POST requests and output the JSON response from the service. This is how simple it is to set-up a new environment:

rake clean_all

rake build_statuses

rake build_environments

Clean All does connect to the database and deletes every record, I thought it was too dangerous to add DELETE at this point.

And now I’ll build a new API application into it with has a live service (I have this scripted in an uncommitted file that calls for configs.rb and builds.rb):

build_application 'API'

build_service 'OK Example', 'API', 'Live', '',''

build_service 'Bad Request Example', 'API', 'Live', '',''

build_service 'Down Example', 'API', 'Live', '',''

As explained build_service, build_statuses, build_application are just POST requests, here is an example of one of the build_statuses entries.

Net::HTTP.post_form(uri, 'name' => 'BadResponse', 'description' => 'service is not responding OK', 'image' => '/images/exclamation.png')

At this moment we have everything set up and can start adding status updates (Events) - Because everything is decoupled we can just create a script that does this and posts updated, I added a very simple extension to the same project, you can see it here.

When you run this one tiny 50 line script what you get is a one minute loop what GETs all services from the main ‘Board’ and uses the URLs and Host Headers to curl the status endpoints, then it POSTs an update to the board with the current results or errors if it’s not HTTP OK.

In the same manner the updates are being done we can make a specialised Web Application, again, separated from the main ‘Board’ which is just a dumb reader for status and shows current state (code here - I apologise to Stashboard for stealing images and CSS but I’m not that great at it).

These two extensions are just examples on how new logic for updates or display can be built completely detached from the main database/update service and I think it’s a good example on how the single responsibly principle really affects the complexity and expandability of a project.

# Some ideas for future work:

Add OAuth - currently this has to be internal but we can make the Web UI reader external without compromising the data - OAuth could actually be a separated middle service that is allowed to connect to the Board but available externally. Update logic should be more careful and not insert every single update - probably also have some control on the number of POSTs allowed.

Back to posts