Wednesday, February 25, 2009

Python Accessing Caller **locals() from callee method using sys._getframe

This is my first attempt at memcaching html page using cheetah
since cheetah render needs locals() i use getCallerInfo() to get the locals() and send to memcached
let me know if it is possible to better do this
notice getCallerInfo

utils.py
@log_time_func
def renderpage(key, htmlfile, deleteafter=3600):
from globaldb import mc
try:page = mc.get(key)
except:
page=None
clogger.info('except error mc.get '+ key)
if not page:
clogger.info(key+ ' rendering cheetah page')
terms = getCallerInfo(1)
#print terms
page = str(web.render(htmlfile, asTemplate=True, terms=terms))
try:mc.set(key, page, deleteafter)
except:
clogger.info('except error mc.set '+ key)
return page

@log_time_func
@memcachethis
def mcrenderpage(key, htmlfile, deleteafter=3600):
terms = getCallerInfo(2)
#print terms
return str(web.render(htmlfile, asTemplate=True, terms=terms))

def getCallerInfo(decorators=0):
'''returns locals of caller using frame.optional pass number of decorators\nFrom Dig deep into python internals http://www.devx.com/opensource/Article/31593/1954'''
f = sys._getframe(2+decorators)
args = inspect.getargvalues(f)
return args[3]

Usage
key=facebookstuff.APP_NAME+'newstart'+str(uid)
return utils.renderpage(key, 'pick.html')

Saturday, January 10, 2009

Facebook sytle Notifications in couchdb

Goal: To prepare a facebook style notification using couchdb
schema
class Notification(Document):
unread = BooleanField(default=True)
activity = TextField()
appname = TextField()
image = TextField()
type = TextField(default='notification')
fromuser = TextField()
touser = TextField()
link = TextField()
message = TextField()
created = DateTimeField(default=datetime.datetime.now())
time =TimeField(default=datetime.datetime.now())
date = DateField(default=datetime.date.today()

def savenotifications(fromuser, tousers, link, message, activity, image, appname, *args, **kwargs):
for touser in tousers:
if fromuser == touser:continue
notidoc = Notification( fromuser = fromuser,touser=touser, link=link, message=message, activity=activity, image = image, appname = appname)
notidoc.store(fbcouchdb)
for touser in tousers:
r=fbcouchdb.view('_view/pop/unread',key=str(touser), count=1)
for i in r:i

Bold: I even try to prepare the views when some one gets a new notifications by precalling the view unread ahead of time when a new notification for that user is inserted inorder to cause forced view regeneration


def getnotifications(self, all=False):
try:
from couchmodel import fbcouchdb
if not all:
return [ (x.id, x.value) for x in fbcouchdb.view('_view/pop/unre
ad',key=str(self.uid), count=5) ]
else:
return [ (x.id, x.value) for x in fbcouchdb.view('_view/pop/noti
fications',key=str(self.uid)) ]
except:return[]


Problem:but whenever i do getnotification it is really slow.. And the page doesnt open forever. Anyways to make couchdb usable with this db? the situation is that notifications are constantly inserted into the db all the time as things happen to the user in real time. is this not a good use case for couchdb

Wednesday, January 07, 2009

Using couchdb to build a EMAIL messaging solution with threaded inline replies

Couchdb based Messaging system
The goal is to build an anonymous inbox with inline threaded replies and count of number of items in the inbox and sent. It works well, the only part that is hard is that the views are pretty slow the first time they are constructed

Schema

class Anonmessage(Document):
subject = TextField()
read = BooleanField()
fromread = BooleanField()
toread = BooleanField()
msg = TextField()
background = TextField()
type = TextField()
reply = TextField()
fromuid = LongField()
replycount = IntegerField()
touid = LongField()
created = DateTimeField(default=datetime.datetime.now())
time =TimeField(default=datetime.datetime.now())
date = DateField(default=datetime.date.today())


Permanent view for inbox:
function(d){ if(d.type != 'reply'){ emit(d.touid, d); if(d.touid != d.fromuid){ if (d.replycount>0) emit(d.fromuid, d); } } }

Count of number of messages in USER inbox

map:function(d){
if(d.type != 'reply'){
emit(d.touid, 1);
if(d.touid != d.fromuid){
if (d.replycount>0)
emit(d.fromuid, 1);
}
}
}

reduce:function(keys, values) { return sum(values)}

Count of Sent items
map
:function(doc) { emit(doc.fromuid, 1) }
reduce:function(keys, values) { return sum(values)}

View to get replies
map:function(d) { if (d.type=="reply" ) emit(d.reply,d); }

Usage from python
inbox_messages = [ (x.id, x.value) for x in anondb.view('_view/truthbox/inbox',key=z.uid) ]

Saturday, January 03, 2009

Ten Reasons why Couchdb is better than Mysql

  1. No schema - schema less db, which means you can develop at the speed of your thought, you dont need to do a db update everytime you add a column
  2. everything is over http, simple http, get post, put, delete requests which means works with varnish/squid out of the box (i prefer varnish because it is clean and simple)
  3. Attachments - you can store file attachments for eg., your greeting card to grandma can carry images, music, flash
  4. Map Reduce - no more sql queries, use amazingly scalable map-reduce based views. Views once saved are lighting fast
  5. Sexy Futon javascript interface, comes with a cool js interface for displaying and editing data
  6. javascript server using mozilla spidermonkey to construct views - means no need php in flash
  7. zero config replication - work from home with no internet
  8. python couchdb library
  9. bulk updates, deletes - you can store 100000 docs in one post request
  10. Each couchdb document is just a simple JSON compatible doc, no cruft just simple
  11. (Bonus) Uses Erlang, which means it is scalable for multicore multiprocessor machines
  12. (Extra Bonus) Low memory requirement Takes 150MB compared to 8Gig taken by Mysql for a similar db setup
  13. similar to zodb, but much more cleaner and intuitive
  14. Extremely friendly community and developers - Damien, Jchris, paul davis, noah slater, chris anderson, Jan