My script:


a = []
>>> def memory_hog():
... for i in range(1000):
... a.append(a + ["what"])
>>> memory_hog()
>>> memory_hog()
.
x times
.
>>> memory_hog()


Dreamhost was pretty prompt in killing my python processes as soon as the python process hit 512MB of RAM with the following message.

“Yikes! One of your processes (python, pid 22279) was just killed for excessive resource usage.
Please contact DreamHost Support for details.”

It was depressing that Godaddy hosting services only offered python 2.4 but it does not matter for this test. On the specific Godaddy server that I was testing, there was a total of 12G of memory and I was able to easily hog memory for well over 2G and my process was not killed. I did not test my limit to see if the server goes down since python 2.4 was already not worth for me to explore more.

Webfaction claims to have a memory limit of 256MB which is quite constrained.

My hope is to use a combination of AWS and a few of these other shared hosting to deploy the site. Maintaining db server, memcache and lb on aws and deploying multiple webservers on shared hosting seems like a possibility.

I explored Storm-Starter which has some examples storm topologies to help get your feet wet. Most java examples worked fine for me but I got issues in running the WordCountTopology. The WordCountTopology example uses the python language to write its bolt. I kept getting the error below:

Cannot run program “python” (in directory “/tmp/cf2bf9a …

The problem was solved by adding the “multilang” directory in Storm-Starter project to the class path.

Usage:

1) Remove ‘a’ tags: Type the commands as shown in the image below in a python terminal

https://addons.mozilla.org/en-US/thunderbird/addon/linux-growl/

Screenshot:

In settings.p, you will require the following changes.

TEMPLATE_LOADERS = [
-'django.template.loaders.filesystem.Loader',
-'django.template.loaders.app_directories.Loader',
+'django.template.loaders.filesystem.Loader',
+'django.template.loaders.app_directories.Loader',
'forum.modules.template_loader.module_templates_loader',
'forum.skins.load_template_source',
]
TEMPLATE_CONTEXT_PROCESSORS = [
'django.core.context_processors.request',
'forum.context.application_settings',
'forum.user_messages.context_processors.user_messages',
-'django.core.context_processors.auth',
+'django.contrib.auth.context_processors.auth',
]

Boto is a python package that provides a number of python interfaces to interract with Amazon Web Services (AWS) like EC2, RDS, ELB, etc. Boto interface for Route53 is pretty simple. Boto allows adding ‘A’, ‘AAAA’, ‘TXT’, ‘CNAME’, ‘MX’, ‘PTR’, ‘SRV’ and ‘SPF’ records to Route53.

Each zone that is created on Route53 has a zone id which can be obtained by using the command line utility provided by amazon.

% ./bin/route53 ls
============================================
| ID: ZXXXXXXXXXXXXXX
| Name: yourdomain.com.
| Ref: xxxxxxx-xxx-xxxx-xxxx-xxxxxxxxxxxx
============================================
{}

Basic authentication on boto for Route53

The following step is required for all boto operations with route53. In python terminal use the following commands:

>>> import boto
>>> from boto import route53
>>> key = ‘AAES5WCQQBW5XZ2DAA’
>>> access = ‘DMJsjdlkr0OHnrwqEsGkksjweosbg/yeISOPEOdnsK’
>>> route53 = Route53Connection(key, access)
>>> results = route53.get_all_hosted_zones()
>>> results
{u’ListHostedZonesResponse’: {u’HostedZones’: [{u'CallerReference': u'6000r000-b00-0000-8000-46000000c', u'Config': {}, u'Id': u'/hostedzone/ZXXXXXXXXXXXXXX', u'Name': u'yourdomain.com.'}], u’IsTruncated’: u’false’, u’MaxItems’: u’100′}}

Adding A record on Route53 with Boto

>>> import boto
>>> from boto.route53.record import ResourceRecordSets
>>> conn = boto.connect_route53(key, access)
>>> changes = ResourceRecordSets(conn, “ZXXXXXXXXXXXXXX”)
>>> change = changes.add_change(“CREATE”, “yourdomainname.com”,”A”)
>>> change.add_value(“45.34.22.455″) #your ip address
>>> changes.commit()

Adding CNAME on Route53 with Boto

>>> import boto
>>> from boto.route53.record import ResourceRecordSets
>>> conn = boto.connect_route53(key, access)
>>> changes = ResourceRecordSets(conn, “ZXXXXXXXXXXXXXX”)
>>> change = changes.add_change(“CREATE”, “subdomain.yourdomainname.com”, “CNAME”)
>>> change.add_value(“some_dns_name”)
>>> changes.commit()

Adding TXT Record on Route53 with Boto

Note: All text records will require you to add strings as value. Note the extra quotes around the text value, else expect to see an error as below:

Invalid Resource Record: FATAL problem: InvalidTXTRDATA encountered at “\”v=spf1 ip4:64.64.9.245 a mx ~all\”

>>> import boto
>>> from boto.route53.record import ResourceRecordSets
>>> conn = boto.connect_route53(key, access)
>>> changes = ResourceRecordSets(conn, “ZXXXXXXXXXXXXXX”)
>>> change = changes.add_change(“CREATE”, “yourdomainname.com”, “TXT”)
>>> change.add_value(“\”v=spf1 mx ptr include:mail and other text a ~all\”“)
>>> changes.commit()

Deleting CNAME on Route53 with Boto

>>> import boto
>>> from boto.route53.record import ResourceRecordSets
>>> conn = boto.connect_route53()
>>> changes = ResourceRecordSets(conn, “ZXXXXXXXXXXXXXX”)
>>> change = changes.add_change(“DELETE”, “subdomain.yourdomainname.com”, “CNAME”)
>>> change.add_value(“some_dns_name”) #Note: This value has to match exactly with the record
>>> changes.commit()

After installing the plugin in solr and changing the required entries in solr.xml and solrconfig.xml, a spatial query resulted in the following error.

HTTP ERROR: 500

Invalid shift value in prefixCoded string (is encoded value really a LONG?)

java.lang.NumberFormatException: Invalid shift value in prefixCoded string (is encoded value really a LONG?)
at org.apache.lucene.util.NumericUtils.prefixCodedToLong(NumericUtils.java:206)
at org.apache.lucene.search.FieldCache$10.parseDouble(FieldCache.java:294)
at org.apache.lucene.search.FieldCacheImpl$DoubleCache.createValue(FieldCacheImpl.java:635)
at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224)
at org.apache.lucene.search.FieldCacheImpl.getDoubles(FieldCacheImpl.java:602)
at nl.jteam.search.solrext.spatial.lucene.LatLongLocationDataSetFactory.buildLocationDataSet(LatLongLocationDataSetFactory.java:35)
at nl.jteam.search.solrext.spatial.lucene.SpatialQuery$SpatialWeight.scorer(SpatialQuery.java:186)
at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:297)

The solution was to delete the data folder (aka the indexes) and index the data again.

Short example of mozilla components:

For sake of argument, assume that there is no global scoped function called “alert()” in javascript which generates a popup message. The only way of creating a popup box is to access a mozilla component as below:

var sample = Components.classes["@mozilla.org/embedcomp/prompt-service;1"].
getService(Components.interfaces.nsIPromptService);
sample.alert(null, “Window Title”, “Alert Message”);

So in a world where “alert()” was never an openly callable function, mozilla components like the one mentioned above which is aptly named as “prompt-service” expose functionalities that can be accessed from outside.

XPCOM was developed so that different components can be developed in different languages and can all be exposed to each other via common interfaces. Hence you can imagine developing mozilla components in Javascript, C++, python and other languages as long as they maintain XPCOM interfaces appropriately. XPCOM interfaces for mozilla components are based on IDL which is an interface description language. Learn more about IDL here .

If you want to know all the mozilla components available on your firefox, you can install the XPCOM Viewer plugin. The following image shows the mozilla components installed in my firefox installation. All these components expose functions that can be accessed as we did in the above example. Since it is cross platform you will be able to find ways to access them across many languages.

Short Memcache Tutorial

Dynamic websites are those in which web pages are generated on the fly using some mainstream scripting
languages. Unlike static html websites that do not change, dynamic websites are characterized by their ever changing “dynamic” content. Almost without doubt a dynamic website must use some kind of data storage to update its content over time. One of the simple cases is a sql server. When a high traffic dynamic website requires to load a popular page a million times, it is wasteful of resources to run similar data acquisition queries on the sql server. To help such situations, a caching mechanism can act as a blocking layer that can store information regarding a page without having to re-fetch it from the data store of the mysql server. Memcache server is one such caching application that help your applications store / cache some information. Memcache is an in memory application which means that it stores the cached content in RAM. You will be able to interface with memcache using telnet but not quite practical for applications written in various languages to interact in that manner. Hence, most languages provide a wrapper that can act as in interface between your programming language and the memcache server.

Memcache has a simple interface which is basically a key value store. Let us consider a single web page as an item and the page url as a unique identifier to the page. Memcache can be easily imagined as a dictionary where the index is the url and the value is the content (html) of the page. Off course it need not work in the way that I just mentioned, but a way to think about it. Usually the keys are hashed to minimize conflicts.

Towards the end of this article we will see a way to interface python and memecache server.

Memcache installation ubuntu:

On an ubuntu box, memcache installation is easy.

sudo apt-get install libmemcache-dev libmemcache0

That is it ! Check if memcached is running by using

ps aux | grep memcached

To start memcached server, just go ahead and use the command (Note:- if you did not manually start memcache, find the memcache start command details using “ps aux | grep memcache”) :

memcached -d -m 64 -p 11211 -u memcache -l 127.0.0.1

Options:
-d runs the server as a daemon
-m specifies the memory that memcache is allowed to use
-p port to which memcache listens to
-u user
-l hostname that would like to bind to (127.0.0.1 prevent access from outside your server)

What is happening now:

After you have started memcache, you can interact with it in some nice different ways. Let us first interact with it through telnet. We gave the port number while startup so lets telnet to port 11211 as below

# telnet 127.0.0.1 11211

You should expect the following result :
Trying 127.0.0.1…
Connected to 127.0.0.1.
Escape character is ‘^]’.
|

This essentially has setup an access to memcache via port 11211 which the memcache server is listening to. Type in your commands and hit enter to get a response form memcache. Find a list of valid memcache commands to interact with it.

How to store value in memcache

If you had followed this far, you are currently at :

Trying 127.0.0.1…
Connected to 127.0.0.1.
Escape character is ‘^]’.

Type the set command which has the following syntax:

set  <keyname> <flag> <time until expiry of data> <bytes of data you will store>

Trying 127.0.0.1…
Connected to 127.0.0.1.
Escape character is ‘^]’.
set testkey 0 120 4 (press enter and in the next line provide the 4 byte data)
data
STORED (Response from memcache. You just stored “data” to key “testkey”)

How to get value from memcache

Assuming that you had already stored a value as mentioned above, access the stored data using the get command which has the following syntax:

get <keyname>

The following is the result of the issued command:
Trying 127.0.0.1…
Connected to 127.0.0.1.
Escape character is ‘^]’.
VALUE testkey 0 4
data (You have got back the value that you had stored !!)
END

Make sure you do the set and the get within a span of 120 second as you have asked memcache to store the key value pair just for 120 seconds in the set command. Increase that to a really long time as desired. We have come far in understanding the functioning of memcache server and I hope it helped.