sábado, 14 de fevereiro de 2015

typokiller: my new tool to exterminate typos in software documentation

Last year, during the Django Sprint we had in Kraków, I was working on finding and fixing typos in the Django Documentation.

It was fun to automate spellchecking the docs, but it was also very laborious to use my rudimentary tools.

This week, in a span of 2 nights and 1 morning, I worked on a minimally usable version of typokiller. The first outcome of this work was this Pull Request made just a few hours ago.

Even though typokiller is still an alpha-quality baby project, I am pretty happy that it has an interactive terminal-based UI, thanks to the awesome termbox library:

With typokiller running, a bit of patience and about 1 hour of careful inspection I was able to find 45 typos in OpenShift Origin's documentation. As you can see in the screenshot I had to go through 1583 potential typos. Thank God I implemented replace all, edit all and ignore all commands :-)

The idea behind typokiller is to have several small programs each responsible for one of these steps:

  1. Collect documentation excerpts
  2. Check spelling
  3. Interactively fix typos
Right now typokiller only supports inspecting the documentation of Go code, but I might include support for more languages in the future (or you can create and issue to tell me which language you need, or you can code it and make a pull request).

You start with a Git repository with some project containing potential typos in its documentation. You might want to create a new branch. Then you can run:

$ ./killtypos $(find /path/to/project -type d -not \( -name Godeps -prune -o -name .git -prune  \))

This will find all directories inside the project path, ignoring the subdirectories Godeps and .git, and pass that through the 3 stages of typokiller, leaving you in an interactive mode. After inspecting the potential typos, type "a" to apply the changes to the files. Review the files with Git, commit, push and make a PR.

The design was made such that I can pipe the output of one command into another, and also easily integrate with other unix tools like head, grep, etc.

That's all for now, let me know if you enjoy typokiller.

domingo, 8 de fevereiro de 2015

Falcon, Django, PyPy & Docker

Late last year I learned about the Falcon Framework, one more to the list of many Python frameworks out there. Falcon focus on performance, having quite impressive benchmarks.

So I was wondering how can you have super fast API endpoints in your Python web service while still serving some legacy code in Django, or just any other WSGI-compliant framework.

The outcome of my experiment lives in this GitHub repo: rhcarvalho/django-plus-falcon-pypy.

sábado, 7 de fevereiro de 2015

StarterKit - automated development environment

I've been working at Base for 439 days. That's one year, two months, and two weeks of very intensive work.

I joined in 2013 to give life to a newly formed team of Python developers, effectively introducing a new piece to our existing infrastructure, previously Ruby-based. By the way, we're hiring.

Since then we've completed a few projects and keep working on end user productivity in our product. That's pretty exciting as we're at the core of our mission of helping sales teams become 10x more productive.

High productivity is not restricted to sales people. We, as software developers, rely on productivity-boosting tools as well. How can we ship quality code fast? Sure there're a bunch of things to it, but today I'll focus on having a good development environment. We call ours "StarterKit".

Getting Base up and running requires starting tens of services written in multiple languages, frameworks, with all kinds of dependencies, so it asks for a good deal of orchestration to get it all right. 

Our most recent and best attempt to automate it all started with Mirek and Gabriel, who sat together to build StarterKit, further improved by them and several other Base developers. StarterKit is very specific to Base and our internal needs, but the tools we use to build it are open source and you can leverage them to build something that fits your needs.

StarterKit is a Vagrant virtual machine combined with Ansible playbooks to setup a working environment to run Base services in isolation of your development environment that you might be running on your host (editor, vim config, etc).

That means you get a ready-to-use Linux virtual machine, with nginx, Docker and Docker images for MySQL, Elasticsearch, RabbitMQ, etc. You'll have everything pre-configured and with the latest versions of daily used tools.

Each web service that empowers Base runs in a Docker container inside the aforementioned virtual machine. Docker is such a cool tool that deserves a separate post.

In essence, StarterKit aims at making the process of setting up a new web development machine as painless and simple as possible. So what tools do you use to automate your development environment?

927 days

Time is passing, life is changing, and there's been a long while since my last post here.

How did I fill these last 2.5 years? I'm living abroad since the second half of 2012, first in Beijing, China and now in Cracow, Poland. It would take far too long to tell all that have happened, so I'll just move on to writing new posts.

quarta-feira, 25 de julho de 2012

Upgrade your Python's SQLite module on WebFaction

I needed to deploy an application that uses some "new" SQLite features, such as Full-Text Search support. The sqlite3 library available on the system and the standard Python module sqlite3 were too old.
To see the installed versions:

$ python2.7 -c 'import sqlite3; print "lib version:", sqlite3.version, "| SQLite version:", sqlite3.sqlite_version'
lib version: 2.6.0 | SQLite version: 3.3.6

To test FTS support you can run this on the command line:

$ python2.7 -c 'import sqlite3; db = sqlite3.connect(":memory:"); db.execute("create virtual table test using fts4()"); print db.execute("select count(*) from test").fetchone()[0]'
Traceback (most recent call last):
  File "", line 1, in
sqlite3.OperationalError: near "virtual": syntax error

It should print "0", and give no error.

After talking with WebFaction's support team (these guys are great!), they recommend installing pysqlite2 from source. That is the project from where the standard sqlite3 is taken with 1 minor modification that we will reproduce soon.

The steps to get your sqlite3 upgraded:

1. Download and extract the pysqlite2 sources:
$ wget http://pysqlite.googlecode.com/files/pysqlite-2.6.3.tar.gz
$ tar xf pysqlite-2.6.3.tar.gz

2. Build using the most recent version of SQLite:
$ cd pysqlite-2.6.3
$ python2.7 setup.py build_static

Note: do not use "setup.py build", use "build_static" instead. It will download the newest version of SQLite and link against it. (Learned from this question on StackOverflow)

3. Install to your home directory:
$ mkdir -p $HOME/lib/python2.7
$ PYTHONPATH=$HOME/lib/python2.7 python2.7 setup.py install --home=$HOME

4. (optional) Override the standard sqlite3 module:
$ echo "from dbapi2 import *" >> $HOME/lib/python2.7/pysqlite2/__init__.py
$ ln -s $HOME/lib/python2.7/{pysqlite2,sqlite3}

This will let your application work without having to change your code (to update import lines).

 We should be done by now:
$ python2.7 -c 'import sqlite3; print "lib version:", sqlite3.version, "| SQLite version:", sqlite3.sqlite_version'
lib version: 2.6.3 | SQLite version: 3.7.13
$ python2.7 -c 'import sqlite3; db = sqlite3.connect(":memory:"); db.execute("create virtual table test using fts4()"); print db.execute("select count(*) from test").fetchone()[0]'