Hi, Ravi Ithal here. Even though my job as chief architect here at Netskope is to tackle the challenge of enabling safe and compliant use of cloud apps for our customers, I also see it as my duty to share the cool tools and practices we use to make our jobs easier and lives better. In my blog posts, you’ll see a lot of content like this. Don’t be alarmed – we’re not all about touting our wares here at Netskope!
So here goes:
If you are a developer or a dev ops engineer at a cloud company, chances are that you will need to spin up a new internal web service from time to time. We do too. We are not talking about million packets per second user facing systems but internal systems that perform lightweight functions without getting (much) in the way of existing services.
The list below enumerates steps to quickly develop a deployable service. It is just one way to do it that has worked very well for us, with very specific choices of technology components. If you use other technologies, you may find parallels to these components.
Just a few years ago, the tools we refer to here didn’t exist and this would take weeks if not months. With the open source frameworks and tools, it’s a matter of hours if not minutes.
1. Set up an isolated runtime using virtualenv
By creating an isolated runtime, you keep the flexibility of running multiple services on the same host to optimize infrastructure, while still achieving the isolation needed in a typical service oriented architecture. We use python heavily and virtualenv works great for us. With virtualenv, you can have two different services on the same host that use different versions of the same library and not have any conflicts as long as they are running under their own virtualenvs. Once you have a virtualenv created, running your favorite process under that virtual environment is simply a matter of activating it and running the process under it. Write a wrapper script for this so we can use this later for a one-line invocation of the process.
#!/bin/bash source $VENV_PATH/bin/activate && exec $@
When you want to run, say myservice.py, you can simply issue: myenvrun.sh myservice.py
2. Build on top of a micro framework such as flask
By choosing a micro framework, you avoid installation bloat, minimize time to get started and defer dependencies to a time when they are really needed. In our particular case, Flask is fantastic since it is made for REST APIs, which is the lingua-franca for SOA these days. Sure you can achieve the same result using a more heavyweight framework like Django but why would you use a sledgehammer to break a peanut? Sure you can start writing your own gevent based service from scratch but why would you embark on emptying a bucket using a spoon?
Here’s how easy it is to get a hello world REST API.
from flask import Flask app = Flask(__name__)
@app.route('/') def hello_world(): return 'Hello World!'
if __name__ == '__main__': app.run()
3. Bring your service to life using smart and hard work
It’s time to bring in life to your service. The goal is to spend as little time as possible in other phases so you can focus on this step. Further, flask also has integrations with many commonly used components such as SQLAlchemy, Celery etc so you can further focus on building business logic rather than spend time integrating.
4. Set up monitored execution using supervisord
You created an isolated environment, built your service on a framework that can grow with the service and grew the service to meet your needs. It’s time to figure out how to keep this running in production without somebody waking you up in the middle of the night to just have you restart your service because you didn’t take care of handling that stupid exception (alright, you need to fix the exception but you shouldn’t have to babysit your process until the fix is in). This is where monitored execution comes to play. By using something like supervisor, you can continuously monitor your process and restart them when they die. You can also rotate your logs without having to modify your logging code inside the app. It even has an XML-RPC interface so your processes can be monitored and controlled from another service.
Here’s a simple conf file to get your service up and running under supervisor. Just fill the paths and drop it under the supervisor config directory.
[program:myprog] command=/usr/bin/myenvrun.sh <path to flask app> process_name=%(program_name)s numprocs=1 priority=999 autostart=true autorestart=true startsecs=10 startretries=3 exitcodes=0,2 stopsignal=TERM stopwaitsecs=10 redirect_stderr=true stdout_logfile=<path to log file> stdout_logfile_maxbytes=1MB stdout_logfile_backups=10 stdout_capture_maxbytes=1MB environment=VENV_PATH=<path to virtualenv> serverurl=AUTO
This was meant to be a short post to highlight how you can get up and running with a new service in a relatively quick and painless way using some nice open source tools out there. In the real world, services need much more work so they can do more than just say hello world and that’s your business logic. How we grow the service by adding features like application specific monitoring is the topic for another post.