GaeCouch - CouchDB compatible API service on GAE

last week I started my small prototype project, GaeCouch. I'll describe why I started on this post.

One concrete thing why I started GaeCouch is not technical but ideological.

Why CouchDB? - Relax

Please read the 'Why CouchDB' chapter in CouchDB Defenitive Guide. It's all for CouchDB. And the first section, 'relax' is the most important.

You know Ruby Rails says 'Web development that doesn't hurt'. Definitely I think it is. And beyond the rails, we need to get the fun of development, which CouchDB gives us.

I have developped several applications with CouchDB for 2 or 3 years in my job or holidays. CouchDB is fun in the life of my development.

Why CouchDB? - Application Platform and WWW

Many web developers (including me) watch the NoSQL movements for scalable database. Someone tries one NoSQL database., others tries the another one. In many case, they evaluate only the database itself, not including application development environments. And they say "This is scalable!", 'This one is not", "I don't know the scale?".

I got to think it is non-sense after learning CouchDB. I'm not database administrator, but a web developer!

CouchDB has the unique feature that other NoSQL databases does not have. It's a application framework, which forces us to think NoSQL application as "document orietned" applications. NoSQL is ambiguous, but "Document-Oriented" is not. WWW - the set of hypertexts and hyperlinks - is a one of document oriented application.

In the other words, CouchDB forces us to be WWW developers. Yeah, we are web developers!

GAE solves one missing thing.

These are why I'm interested in CouchDB. so why I started to port it to GAE?

I've introduced CouchDB (and CouchApp) in Japan several times. People who get to know CouchDB always says two impressions:

- It's fun and convenience about RESTful APIs.
- Why does CouchDB itself work on the only single node? Why do I need Lounge for scale-out? It is bother for me.

OK, I think GaeCouch could empower the former, and solve the latter. I would expect GaeCouch makes it is easier for people to try CouchDB environment.

making CouchDB as yet another MRTG.

I prototyped a CouchApp example for resource monitoring system in this weekend. My prototype is available on my server.

http://www.yssk22.info/stats/_design/site/_show/history


I summarized pros & cons for using CouchDB as resource monitoring system. If you are insterested in, ping me on twitter.com/yssk22 so that I'll publish on my github.

Pros:

• easy to define metrics because CouchDB is schema-less database.
• easy to calculate metrics using CouchDB views.
• easy to notify alerts by using _change & _filter endpoints (not implemented yet).
• easy to render charts using jQuery (and jqplot plugin).
• using the replication feature, monitoring repository clusters could be implemented, which could be more scalable than existent
• no mysql and php required (it's important for me)

Cons:

• CouchDB has no scheduling system (though Lotus Notes has) so that deployment of monitoring agents is not so easy yet.
• There are many matured tools as MRTG, Nagious, Caccti, Zabbix, ... and so on.

To be done:

• Cumulative resources such as /proc/stat should be calculated diffs using list functions.
• Real-time chart rendering using CouchApp's Evently plugin.

CouchApp !code macro recursion.

I wrote the patch about !code macro behavior several months ago. and now an official topic branch was created by jchris's gread help.

Please check it out from http://github.com/couchapp/couchapp/tree/recursive-macros.

This should be helpful for making your javascript library more flexible.

Here is an example:

approot/lib/a.js is:

// !code b.js
function a(){};

approot/lib/b.js is:

function b(){};

and approot/show/test.js is:

function(doc, req){
   // !code lib/a.js
   ..
}

The current version of couchapp deploys the approot/_show/test as:

function(doc, req){
   // !code b.js
   function a(){};
   ..
}

b.js is not extracted by !code macro.

After applying my patch, the deployed code is:

function(doc, req){
   function b(){};
   function a(){};
   ..
}

Secure programming in CouchDB applications - View protection.

I wrote secure programming notes for CouchDB applications. This is written in Japanese, so I've started the translation into English.

--

The first topic is about "View protection" in CouchDB.

For example, we prepare the view as follows.

function(doc){
  emit([doc.owner, doc.created_at], doc.protected);
}

The result of this view will be as follows.

{ "total_rows": 1234, "offset", 0, "rows": [
{"id" : "...", "key": ["user1", "2009/01/01"], value: {...}},
{"id" : "...", "key": ["user1", "2009/01/01"], value: {...}},
{"id" : "...", "key": ["user1", "2009/01/02"], value: {...}},
{"id" : "...", "key": ["user2", "2009/01/02"], value: {...}},
{"id" : "...", "key": ["user2", "2009/01/02"], value: {...}},
...
]}

If we have a security requirment that only document owner can view the doc.protected field, this view should be handled carefully. The current CouchDB view implemetation has no authentication function so that authorization function cannot be implemented.

So how do we solve this requirement?

The answer is:

• to use a reverse proxy to forbid direct view access.
• to use _list functions for view result processing.

And we need to check parameters in our list function.

function(head, req){
  // !code vendor/json/json2.js
  var startkey = JSON.parse(req.query.startkey || "[]");
  var endkey   = JSON.parse(req.query.endkey || "[]");
  if(startkey[0] == endkey[0] &&  
     startkey[0] == req.userCtx.name ){
     while(row == getRow()){
        // row.key[0] == req.userCtx.name is always true.
     }
  }else{
     // not authorized     
  }
}

First entry.

This is a test.