find and replace in place

With our current project structure we are constantly having to jump across versions. Simplest way for in place replacement.

For e.g. to find and replace all snapshot versions in pom.xml

find . -name 'pom.xml' -print0 | xargs -0 sed -i 's/3.1.0-SNAPSHOT/3.1.4-SNAPSHOT/g'

or

find . -name 'pom.xml' -exec sed -i 's/3.1.4-SNAPSHOT/3.1.5-SNAP/g' {} \;

Git log

while git log is standard, quite often I found the need to have a stripped down version of the logs. Git alias to the rescue again

This gives you a one line representation of the hash tag, the author and message

vi ~/.gitconfig
[alias]
lg = log --pretty=format:'%Cred%h %Cblue%an:%Creset%s'

If you want the entire history this works great
lgall = log --graph --oneline --decorate --all

git remove untracked files

I do a lot of branch switching and files which are ignored in one branch show up as untracked files. This gets annoying during merge conflicts as then its difficult to keep track of whats actually conflicted and new and whats not.

After trying around a few commands this alias works great for me

vi ~/.gitconfig

[alias]
remove-untracked = "!git ls-files --other --exclude-standard | xargs rm;"


Now just run git remove-untracked and the eyesore is gone!

convert csv to sql

Generally you can use either awk or sed to generate sql files from a csv input. However on my current project we are using oracle as our db and needed the ability to write a commit instruction every few hundred lines.

After a bit of reading about, awk actually keeps track of lines read and was well suited to generate such a sql.

Here is my shell script. Usage : ./csv.sh inputfile.csv

#!/bin/bash
csvName=$1

awk -F',' '
BEGIN {
lines=0;
}
{
lines ++;
print "insert into db.table values ("$1","$2", "$3");"
if ( lines%100==0 ){
print "commit;"
}
}
END {
print "commit;";
}
' < $csvName > insert.sql

Git and windows

I've missed git since joining overstock, so while starting on a new personal project, I decided to start off with git again.

Now it's been a while since I've programmed on Windows and once you're used to 'nix machine you miss all the tools that's built in. After keeping forgetting that ls, vim or ssh won't work on the stupid windows prompt, I was looking at alternatives.

Git Bash is a brilliant tool for anyone using git on bash. One problem is since it doesn't share the same registers with the base windows OS, copy pasting isn't the easiest. However I noticed that Shift-Insert works to solve that problem. Also windows system variables don't get translated well into git bash - I solved it by redefining them in ~/.bash_profile

If you don't have a git config file, I would recommend using git config initially

git config user.name "anoop"
git config user.email "anoop.kulkarni@gmail.com"

Once you execute this in your project, git creates a file called config under the .git directory.
Here are some of my favorite aliases as well to add to that file:

[alias]
st = status
ci = commit
cia = commit --amend
co = checkout
br = branch
sb = show-branch
cp = cherry-pick
staged = diff --staged
di = diff
rb = rebase
sm = submodule
head = !"git log -n1"
whois = "!sh -c 'git log -i -1 --pretty=\"format:%an <%ae>\n\" --author=\"$1
\"' -" #takes in name of person or email address
whatis = show -s --pretty='tformat:%h (%s, %ad)' --date=short #takes in comm
it name
edit-unmerged = "!f() { git ls-files --unmerged | cut -f2 | sort -u ; }; vim
`f`"
add-unmerged = "!f() { git ls-files --unmerged | cut -f2 | sort -u ; }; git
add `f`"
lg = log --graph --oneline --decorate --all

Push in Grails

There are a few push applications for grails (horrible documentation) and we had to tweak a few things for our application.

Asynchronous Notification or Push follows the typical publish/subscribe design pattern. One of the things about push is it broadcasts to everyone while we had to come up with a way to push messages to specific clients so as not to waste resources on unnecessary updates.

ICEPush:
We started with ICEPush as one advantage it offered over comet is it did not keep a thread open for each request. However we noticed that plugin did not seem well designed to keep track of URL mappings and context paths. I tried to get some feedback on their forums but it doesnt seem like the technology is under active development
forum post

Atmosphere:
Atmosphere seems to have a bigger community following offering comet push and long polling options. Setting it up in grails is pretty simple

grails install-plugin atmosphere

You can then create a service as an atmosphere service

grails create-service clientCountBroadcastService

class ClientCountBroadcastService {
 
    static transactional = false
 
    static atmosphere = [mapping: '/atmosphere/clientNotify']
 
    def onRequest = { event ->
    }
 
    def onStateChange = { event ->
    }
 
}

With this service, UI clients can register with this service and get response on state change. For us we needed to push notifications on a per 'Client' entity and obviously not push notifications for unrelated clients.

At application start up, we then created client specific spring beans and append to the application context in BootStrap.groovy

class BootStrap {
 
    def init = { servletContext ->
        def clientCountBroadcasts = Client.getClientIds().collect { clientId -> ["/atmosphere/clientCountBroadcast/${clientId}", clientId] }
        if (clientCountBroadcasts) {
            def parentContext = servletContext.getAttribute(GrailsApplicationAttributes.APPLICATION_CONTEXT)
            def bb = new grails.spring.BeanBuilder(parentContext)
 
            def beanName = "clientCountBroadcast"
            def beans = bb.beans {
                clientCountBroadcasts.each { clientCountBroadcast ->
                    "${beanName}${clientCountBroadcast[1]}Service"(com.icrossing.cmp.kwbt.ClientCountBroadcastService) {
                    }
                    "${beanName}${clientCountBroadcast[1]}ServiceGrailsHandler"(com.odelia.grails.plugins.atmosphere.GrailsHandler) {
                        targetService = ref("${beanName}${clientCountBroadcast[1]}Service")
                        servletContext = servletContext
                    }
                }
            }
            beans.registerBeans(parentContext)
            def handlers = servletContext.getAttribute(StratosphereServlet.ATMOSPHERE_PLUGIN_SERVICE_HANDLERS)
            clientCountBroadcasts.each { clientCountBroadcast ->
                addMethod(parentContext.getBean("${beanName}${clientCountBroadcast[1]}Service"))
                handlers << [mapping: "${clientCountBroadcast[0]}",
                             handler: parentContext.getBean("${beanName}${clientCountBroadcast[1]}ServiceGrailsHandler")]
            }
            servletContext.setAttribute(StratosphereServlet.ATMOSPHERE_PLUGIN_SERVICE_HANDLERS, handlers)
        }
    }
 
    def destroy = { }
 
    private addMethod(source) {
        source.metaClass.getBroadcaster = {->
            def _broadcaster = [:]
            servletContext[StratosphereServlet.ATMOSPHERE_PLUGIN_HANDLERS_CONFIG].each {
                _broadcaster."${it.key}" = it.value.broadcaster
            }
            _broadcaster
        }
    }

Each client atmosphere bean has a dynamic method to broadcast messages to subscriber. Our client count broadcast service now has the following structure.

class ClientCountBroadcastService {

    static transactional = false

    def onRequest = { event ->
        event.suspend()
        println "onRequest"
    }

    def onStateChange = { event ->
        println "onStateChange"
        def response = event.resource.response
        response.writer.with {
            write event.message
            flush()
        }
        event.resume()
    }

    def broadcastCount(def clientId, def clientCount, def clientVolume) {
        println "broadcastCount"
        def broadcaster = getBroadcaster()
        def clientCountBroadcaster = broadcaster["/atmosphere/clientCountBroadcast/${clientId}"]
        clientCountBroadcaster.broadcast("{\"successeful\":\"true\",\"clientCount\":\"$clientCount\",\"clientVolume\":\"$clientVolume\"}")
    }
    
}

On the UI side we decided to go with long-polling. So a request comes in to the onRequest method where it is suspended and on state change it broadcasts the client count and volume. As we went with long polling, we have to resume the thread on broadcast (which essentially closes the thread and the UI needs to register again).


The UI on load will register with a specific client based broadcaster using long polling and on callback update the text.

...   

//using the jquery atmosphere plugin in head

     
     


.....


    
    function callback (response) {
        $('#updateCount').text(response.clientCount);
        $('#updateVolume').text(response.clientVolume);
    }

    $(document).ready( function (){
        setTimeout(function () {
            $.atmosphere.subscribe(
                KWBT.app.root + '/atmosphere/clientCountBroadcast/${params.clientId}',
                callback,
                $.atmosphere.request = {
                    fallbackTransport : 'long-polling',
                }
            );
        }, 250);
    });


....

    

Expando to the rescue

As we incorporate more plugins in our environment, one of the issues we faced was on how to test code executed by plugins. Most well written plugins can be auto injected into your controllers and the implementing class is generally not exposed. So without knowing the implementing class, mocking it out would not be possible.

We came across something called Expando and the more we use it, the more I'm falling in love with it. Lets give a concrete example using the Background Service plugin

Say we have a service using the background service plugin.

class SomeService {

def backgroundService

...

def doSomething(){
backgroundService.execute("Asynch process", {
doAsynch(....)
})
...
}
}

Now to test this, add expando to your test class and execute the closure

def expando = new Expando()
expando.execute={ info, clos ->
assertEquals "log message match expected", "Asynch process", "${info}"
clos() //This will execute the closure so in effect calling doAsynch
}

someService.backgroundService = expando

So if you are ever considering choosing Groovy on Grails vs Java, Expando is grails' x-factor to tilt the decision.

Connect to In-memory grails hsql database using java swing

Grails comes with a Swing Database manager application to connect to your in-memory database.

Start the grails app : grails run-app

Add the hsql db to your classpath. I would just add it your bash_profile/bash_rc file
export CLASSPATH=$CLASSPATH:/usr/local/grails/lib/hsqldb-1.8.0.10.jar

Run the swing app
java org.hsqldb.util.DatabaseManagerSwing

Enter the following settings
Type - HSQL Database Engine In-Memory
Driver - org.hsqldb.jdbcDriver
URL: jdbc:hsqldb:file:/Users/akulkarni/workspace/ares/kwbtMaster/devDB (Point to your local project location)
User: SA
Password: --blank--

GrouponCheck.com

I've been playing around with appengine for a while and finally I've decided to publish something. My website gets deals from groupon for all cities and displays it on http://www.grouponcheck.com.

Obviously its nothing much to look at right now, but hopefully I get some time to spruce it up. If you log on to the site, it prints out deals from just a few cities and not from the aforementioned all cities. Apparently I found out google has a 30 second limit on requests and since I load the json deals from groupon as a cronned servlet request, its hitting the 30 second limit. I might have to look at task queues and see if they have a solution for me.

Another, not quite obvious, stumbling point was how to set up masked domain forwarding. I bought my domain on google apps through enom and the domain forwarding part isn't as straight forward as my other domain forwarding (anoopkulkarni.com) from yahoo small business domains. I finally happened on to a knol which contained the required instructions with the main part in

• Add a Host Record with @ as the Host Name, the url of the home page of your Site as the Address, and URL Frame as the Record Type
• Change or add a CNAME of www so it points to the symbol @

If google is serious about google apps and google domains, they might look into making their services slightly more straightforward to use.

Inter-vm RPC communication

Service Oriented Architecture

Borrowing the wikipedia definition: Service-oriented architecture (SOA) is a flexible set of design principles used during the phases of systems development and integration in computing. A system based on a SOA architecture will provide a loosely-coupled suite of services that can be used within multiple separate systems from several business domains.

SOA also generally provides a way for consumers of services, such as web-based applications, to be aware of available SOA-based services. For example, several disparate departments within a company may develop and deploy SOA services in different implementation languages; their respective clients will benefit from a well understood, well defined interface to access them

Now if we go ahead with SOA architecture principles, one of the major issues to be solved is how best to address inter-jvm data communication.
Speed is inversely proportional to the data size and one thing to consider is how do the respective communication protocols handle similar data. 

Caucho has a nice article on RMI, Hessian, Spring and CORBA communication (Technically JSON can also be used as a SOA protocol, but they dont have specs comparison for it)

More details on the study can be found protocol comparison

As you can see for sending back the same data, Hessian and RMI comprehensively beats SOAP and Corba, which unnecessarily slows down the speed of response (may not matter much on a gigabit network, but as the size of data adds up, this compounds the response size)

Spring remoting

If you dont want to go all the way into socket programming and maintain an abstraction layer over the actual communication protocol, spring has a few remoting options.
Lets take the example of Hessian (smallest memory footprint). Incase we build services in spring in a single jvm the normal spring bean definition would be

<bean id="accountService" class="example.AccountServiceImpl">

<!-- any additional properties -->

</bean>

Now if we decide to remote this using spring remoting, the new server side definition will be

<bean id="accountService" class="example.AccountServiceImpl">

<!-- any additional properties -->

</bean>

<bean name="/AccountService" class="org.springframework.remoting.caucho.HessianServiceExporter">

<property name="service" ref="accountService"/>

<property name="serviceInterface" value="example.AccountService"/>

</bean>

The client will then call

<bean id="accountService" class="org.springframework.remoting.caucho.HessianProxyFactoryBean">

<property name="serviceUrl" value="http://remotehost:8080/remoting/AccountService"/>

<property name="serviceInterface" value="example.AccountService"/>

</bean>

With minimal configuration change, spring will support either intra-vm or inter-vm communication over http. Ofcourse for additional speed, the remote host will hopefully be an intranet http call to reduce number of ip hops from client to server.

Another obvious advantage of Hessian compared to RMI is it can be called by non Java clients making it much more suitable for heterogenous environments. There are some issues concerning serialization of lazy-initialized hibernate objects.

Hessian Supported Languages: C++,C#,D,Erlang,Flash,Java,Python,PHP and Ruby
Website : spring remoting

TCP/UDP communication

If your app can write to TCP/UDP sockets, there are a few other protocols on the open source market which provide faster ser/deser speeds as well as smaller memory footprint.

See http://code.google.com/p/thrift-protobuf-compare/wiki/BenchmarkingV2
Serialization/Deserialization time

Kryo

Kryo has the advantage of serializing at runtime.

Kryo kryo = new Kryo();

kryo.register(SomeClass.class);

// ...

SomeClass someObject = new SomeClass(...);

kryo.writeObject(buffer, someObject);

Once serialized, you might have to resort to socket level programming to pass the serialized object over the network, or as an alternative use Kryonet (based on top of kryo) for client/server communication. Ofcourse since both client and server need to implement Kryo ser/deser, you are tying yourself to a java based application environment.

Supported Languages: Java
Website : Kryo
Website : Kryonet

ProtoStuff runtime

Google uses ProtoBuf with most of its inter-vm communication and is one of the best java serialization libraries. It unfortunately requires a .proto file that describes the data structure, which gets difficult to maintain over time. Protostuff runtime (which uses protobuf) allows your existing pojo's to be written to different formats (protobuf,json,xml etc.) at runtime

// json serialize

boolean numeric = true;

byte[] json = JsonIOUtil.toByteArray(foo, schema, numeric);

// json deserialize

Foo f = new Foo();

JsonIOUtil.mergeFrom(json, f, schema, numeric);

ProtoBuf Supported Languages: Action Script,C/C++,C#/.NET/WCF/VB,Clojure,Common Lisp,D,Erlang,Go,Haskell,Java,Lua,Mercury,Objective C,Perl,PHP,Python,R,Ruby,Scala
ProtoStuff Runtime Supported Languages: Java

Website : protostuff

Apache Thrift

Thrift works similar to Google's ProtoBuf requiring pre-runtime configuration. It requires a thrift file specifying the thrift interface. However it provides the client server communication protocol with custom server/socket implementation.

TServerSocket serverTransport = new TServerSocket(somePort);

TimeServer.Processor processor = new TimeServer.Processor(someImpl()); //Impl that the client will call

Factory protFactory = new TBinaryProtocol.Factory(true, true);

TServer server = new TThreadPoolServer(processor, serverTransport, protFactory);

server.serve();

transport = new TSocket("localhost", 7911);

TProtocol protocol = new TBinaryProtocol(transport);

Client client = new Client(protocol);

transport.open();

//client can call the impl function

As opposed to kryo/hessian/Proto* which serializes entire objects, thrift has a few common base types

bool: A boolean value (true or false)
byte: An 8-bit signed integer
i16: A 16-bit signed integer
i32: A 32-bit signed integer
i64: A 64-bit signed integer
double: A 64-bit floating point number
string: A text string encoded using UTF-8 encoding

Supported Languages : C++, C#, Erlang, Haskell, Java, Objective C/Cocoa, OCaml, Perl, PHP, Python and Ruby

Website : Apache Thrift

Add Labels