Ruckus Notes

Saaspose: Cloud based document conversion & recognition

Fri, 25 May 2012 11:27:18 -0700

Saaspose: Cloud based document conversion & recognition:

Appears to be a versatile service for barcode recognition and OCR. This could come in very handy.

Protip: Store file imports for better support / debugging

Thu, 17 May 2012 15:50:00 -0700

If you run a service which allows one to import data, say maybe by uploading a CSV file, than do yourself a favor and store the original file in a place where you can access it later for support or debugging.

Our service allows a person to upload a CSV file containing postal addresses and it has about 8 columns that need to be in a certain order.

Before our code starts to actually process the file we shove it up to S3 in its original form. Later during processing if an error is encountered than a support person can look at the original file and see any obvious errors. Maybe you expect a CSV file but get an Excel (or a Apple Numbers file)? Maybe the customer wasnt paying attention and totally re-arranged the columns. Sure, most of these “soft” errors (such as CSV vs Excel) can be caught pretty easy by inspecting the file extension, but as a whole you are guaranteed to run into trickier situations.

Either way, having the original file on hand for debugging is super helpful. Much better than emailing the customer and asking them to re-send you the file. In most cases we can be pro-active and email the customer back and say “Hi, I saw that you just attempted a data import and it failed, it looks like it was due to ….”

Postgres search: normalizing accented characters

Mon, 30 Apr 2012 11:56:31 -0700

As part of my move from Sphinx to Postgres Full-Text Search I needed a way to normalize accented characters. My data contains lots of diacritics, a common example is the varietal name “Grüner Veltliner”.

My users do not want to enter that Umlaut each time they want to search for this varietal.

Fortunately there is an awesome Postgres contribution package called “unaccent” which replaces diacritics with their plain text equivalent, effectively normalizing the data set.

Using the package is pretty straight-forward. We first install the extension and then create a new text search configuration and ensure that we use it in all indexing and searching.

When searching make sure we reference the new configuration instead of the default ‘english’:

Sidekiq: alternative to Resque for background processing

Thu, 26 Apr 2012 13:45:00 -0700

Mike Perham of Dalli fame has written an awesome library called Sidekiq which enables background processing via threads.

Resque is a popular background processing solution and its what we use at Batch. It’s battle tested and for the most part has stood up well.

However, it feels heavy-weight because each Resque worker is a distinct process. Which is where Sidekiq comes in: it uses a pool of threads to process jobs so at most you have one master process.

For smaller apps on constrained systems (read: VPS) I feel this can be a better solution. I’ve been using Sidekiq for a couple of weeks now and I really like it. Sidekiq is Resque compatible so you can write jobs with existing Resque code and then process them using new Sidekiq workers. But in my case I just went straight Sidekiq.

One major change is that in Resque your workers must implement a class method of perform, but in Sidekiq the perform method must be an instance method.

So far I am a fan of Sidekiq. Give it a shot!

Switched from Sphinx to Postgres Full Text

Tue, 24 Apr 2012 14:27:00 -0700

I’ve recently migrated two Rails projects from Sphinx search to Postgres Full Text Search. Mainly because the applications were small and I didn’t see the benefit of running another service, hence another point of failure.

In both cases the number of documents and the level of search activity were not very high, so it was not a question of load on the Postgres server.

In one of the applications searchable content has to be filtered for security and just treating Sphinx as a dumb search content repository was annoying. By moving it all into Postgres I can have the actual fuzzy text searching and the necessary security checks (by checking other tables) all in one place.

My general architecture / flow is:

* For each searchable table: add a search_content column of type tsvector

* Create a GIN index on the search_content column

* If the columns can be indexed as-is and we dont need any other searchable columns then we can use the native tsvector_update_trigger trigger to update the search index:

However, if the search content requires other tables then we need to write a manual function trigger. Don’t forget to use COALLESCE if any of your searchable columns can be NULL. If you forget this and allow NULLs to be creep in then it will make the whole tsvector NULL and you’ll wonder why you have no searchable content.

Now we need to seed the initial search index. In this case I just leveraged the touch method in ActiveRecord.

Contact.all.each { |c| c.touch }

Finally, performing searches is pretty straight-forward. I wish there was a way that we can save from having Postgres parse and construct a query structure twice, but for these projects its a negligible cost.

Quickbooks + Ruby

Fri, 09 Mar 2012 18:37:44 -0800

Recently I needed to integrate my Rails app with Quickbooks. Looking around on GitHub I didn’t find any fresh libraries.

Intuit has their Data Services API (currently at v2) which exposes a REST API to their customer data. Its a pretty clean API. Its annoying that it uses XML but Intuit is hard at work on v3 which will support JSON so thats nice.

But anyways, I didn’t find any good libraries so I decided to roll up my sleeves and write my own: Quickeebooks

It supports reading and writing of basic objects using the v2 API.

I think it’s pretty easy to use, you work with plain Ruby objects which get marshaled to XML and back.

Give it a shot and let me know what you think.

Bundler install error: ArgumentError: invalid byte sequence in US-ASCII

Fri, 02 Mar 2012 10:37:00 -0800

I upgraded my Rails app to 3.2.2 and was doing a deploy via Capistrano and during the bundler install process the deploy bombed out with the following error:

** [out] ArgumentError: invalid byte sequence in US-ASCII
** [out] An error occured while installing will_paginate (3.0.3), and Bundler cannot continue.
** [out] Make sure that `gem install will_paginate -v '3.0.3'` succeeds before bundling.

I think the reference to will_paginate is really a red-herring. Anyways, the solution is to properly set the default environment in Capistrano which gets passed to the environment that Capistrano+Bundler use in the deploy.

Specifically you need to set the LANG attribute.

Add this to your Capfile. Note that the PATH is not strictly necessary, but I had that block in there already.

Writing a basic image filter in Android using NDK

Tue, 21 Feb 2012 20:44:00 -0800

I needed to implement some image filters on bitmaps for my Android project. I first attempted the filters in pure Java but it turns out to be too slow and consume too much memory.

Bitmap handling in Android has always been a pain point and there is too much memory overhead.

To get access to the underlying pixel data from a Bitmap in Java you use Bitmap.getPixels() which returns an integer array of Color objects. There is just too much object creation there meaning too much overhead.

So the solution is to go native C.

For a proof of concept I wanted to try implementing a filter in native C using the Android Native Development Kit (NDK). My test case adjusts a bitmaps level of brightness.

The complete source is available on GitHub: https://github.com/ruckus/android-image-filter-ndk

First off you will need to download the Android NDK and place the directory somewhere handy, I put mine at /AndroidSDK/android-ndk-r7b along side my standard Android SDK.

In your Activity you will need to declare a method using the native keyword, this tells Android that the implementation for this method is in a C library, which we will generate shortly.

In the jni folder you will need any C files, I just have a single file, and a Makefile template.

Compile the library using the ndk-build command in the NDK:

If all goes well then it should compile cleanly and leave you with a libimageprocessing.so shared object.

Crack open jni/imageprocessing.c and lets have a look. Notice that the method name we call is

Java_com_example_ImageActivity_brightness

Which is the complete Java package name plus the method that we annotated with the native keyword in the Activity. The first argument to this method is the JNI environment which Android supplies for us. Any additional arguments are the user supplied ones part of the signature in Java. Looking at the signature in Java:

public native void brightness(Bitmap bmp, float brightness);

We pass in the Bitmap and the brightness value.

Looking back at the Java Activity notice that C receives the Bitmap and writes back the modified pixels to the same object.

The Android NDK gives us some basic bitmap functions that we need to use to get at the actual pixel data:

AndroidBitmap_lockPixels

Its important than any calls to AndroidBitmap_lockPixels are complemented with a AndroidBitmap_unlockPixels when you are done.

In the core filter method brightness the logic is the following:

1) Iterate over each row of pixel data.

2) After we grab the whole row we iterate over each column.

3) The pixel data is a packed integer which contains the actual RGB values

4) Since we need to manipulate each RGB value we extract the components out

5) We multiplty by the brightness factor and then constrain it to a value between 0 and 255

6) Then finally we write the modified pixels back into the structure.

Some before and after screenshots:

Recruiter Fail! Some ProTips: Don’t forget to substitute...

Tue, 07 Feb 2012 09:19:00 -0800

Recruiter Fail!

Some ProTips:

Don’t forget to substitute my real name into the email template placeholders
Make sure all the other recipients are sent their own unique email and not just CC’ed onto mine.

No wonder recruiters garner so much scorn.

Java: Joda DateTime on the wire, long values in the database

Mon, 09 Jan 2012 15:27:00 -0800

I’ve begun development of the Android app for Batch and we use REST/JSON as our API.

Dates in JSON are represented as strings like:

2012-01-07T01:06:37Z

I am also using the wonderful ORMLite framework to persist my objects in a local SQLite database. I would like to store my dates as UNIX timestamps using a type of “long”. By doing this I can do date filtering much faster by just having SQLite compare numbers instead of strings.

Thus, I need an easy and transparent way to marshal my dates from DateTime strings to Longs and back and forth.

Fortunately I am using the Jackson JSON library which has killer custom serialization/deserialization support.

WAL-E for Postgres WAL archiving / backup

Wed, 21 Dec 2011 08:29:59 -0800

WAL-E for Postgres WAL archiving / backup:

Been using WAL-E for Postgres continuous archiving / backup and I have to say I really like it. Super easy to setup and start rolling.

Point-In-Time-Recovery has never been so easy…

Logging to Rails.logger from a Resque job

Tue, 18 Oct 2011 12:06:41 -0700

I had various Rails.logger statements in my Resque background jobs but they were never hitting the the actual log.

Turns out the problem was due to ActiveSupport::BufferedLogger flushing. What happens is that by default the flush period is 1000 statements, but since a given Resque job has such a short lifespan the flush period never gets reached.

The solution is to turn on auto flushing for the duration of that job

Generate Excel friendly CSV from Ruby with UTF8/16 data

Mon, 17 Oct 2011 13:59:42 -0700

I needed to generate an Excel friendly CSV file from a bunch of UTF-8 encoded data, coming from a database. I could generate the file just fine and it looked OK in vi or TextMate but when opened in Excel all of my diacritics (umlauts, etc) would be rendered as gibberish. Which made me think it was more of an Excel issue, because they appeared fine in TextMate, for example.

The answer ended up being two-fold:

Use a tab delimiter instead of a comma
Use Iconv to force your string to be UTF-16

Both of these seemed to be required. Why the tab delimiter was crucial I have no idea, chalk it up to MS.

"I think in general engineers tend to not care as long as people stay out of their way and follow..."

Thu, 28 Jul 2011 11:15:43 -0700

“

I think in general engineers tend to not care as long as people stay out of their way and follow procedure and use sound logic and back things with data.

Engineers hate working on things that data says won’t work, or just don’t make common sense. Engineers, while sometimes creative are very practical and programmatic by nature of their thought process.

The easiest way to piss of an engineer is to have a bad idea, have no data to suggest why it’s not a bad idea and to disrupt the current flow of activity. Do that, and you’re on the shitlist.txt file for life.

Come to the table and explain the logic behind the idea, the data behind it, what needs to be done to bring it to reality, and show you did your homework, and have a timetable that’s practical. More often than not, you’ll get a lot of engineering support.

Engineers love to be productive and work on winning things. Most have an overachiever complex to some degree. They hate to get on board the fail train. Most companies have more work than engineering resources. Think about it.

”

- Robert Accettura - How do engineers at tech companies feel about their non-engineer colleagues?

PostgreSQL trigger to update a tsvector column

Sat, 23 Jul 2011 11:29:00 -0700

One method of implementing full-text search in PostgreSQL is by storing your searchable content in a tsvector column. You can add a GIN index on this column for super fast full-text search, but its not required.

The idea is that you have, for example, a title column that you want searchable. So you make a tsvector column and then store the search index for the title column in the tsvector column.

As content is added to the title column you need to make sure the search index is kept up-to-date. This is where a pgsql trigger comes in to play:

In this trigger we always update the tsvector on an INSERT and only update it on an UPDATE when the text actually changes.

Processing Resque jobs in Ruby that are queued by Java Workers

Thu, 07 Jul 2011 17:52:03 -0700

I am using the awesome Jesque package to process jobs from Resque.

We have a mixture of Ruby and Java workers. All our workers accept a Hash of arguments. In one case I needed to create a job from Java to be consumed by a Ruby worker. To generate the appropriate argument structure I ended up making a dummy worker class like:

and I materialize the Job like so:

Using this technique I can properly generate jobs from Java to be consumed by Ruby workers.

Connection Pooling in JDBC with Postgres and Commons DBCP

Wed, 29 Jun 2011 17:22:05 -0700

I was trying to get connection pooling in JDBC with Postgres going and looked into using the Commons DBCP and Pool libraries.

The problem is that the examples on how to get started dont refer to the released versions of the libraries but instead refer to trunk and the current refactoring.

Thus: the basic “PoolingDataSourceExample.java” doesnt work as advertised, based on the current release of commons-dpcp-1.4.jar and commons-pool-1.5.6.jar.

I was able to create my connection pool by using code like so:

CRON gotcha

Mon, 27 Jun 2011 09:17:00 -0700

Programs and scripts in /etc/cron.{d, hourly,daily,weekly,monthly}/ are ignored if their names contain characters outside a very strict list. Most notably, dots. So if your program ends with .sh, .pl, .py or whatever, cron will silently ignore it.

Origin of Company Policy

Sun, 04 Apr 2010 15:53:00 -0700

Start with a cage containing five monkeys. Inside the cage, hang a banana on a string and place a set of stairs under it. Before long, a monkey will go to the stairs and start to climb towards the banana. As soon as he touches the stairs, spray all of the other monkeys with very cold, high-pressure water. After a while, another monkey makes an attempt with the same result — all the other monkeys are sprayed with cold water. Pretty soon, when another monkey tries to climb the stairs, the other monkeys will try to prevent it.

Now, put away the cold water. Remove one monkey from the cage and replace it with a new one. The new monkey sees the banana and wants to climb the stairs. To his surprise and horror, all of the other monkeys attack him. After another attempt and attack, he knows that if he tries to climb the stairs, he will be assaulted. Next, remove another of the original five monkeys and replace it with a new one. The newcomer goes to the stairs and is attacked. The previous newcomer takes part in the punishment with enthusiasm!

Likewise, replace a third original monkey with a new one, then a fourth, then the fifth. Every time the newest monkey takes to the stairs, he is attacked. None of the monkeys that are beating him have any idea why they were not permitted to climb the stairs or why they are participating in the beating of the newest monkey. After replacing all the original monkeys, none of the remaining monkeys have ever been sprayed with cold water. Nevertheless, no monkey ever again approaches the stairs to try for the banana. Why not? Because as far as they know that’s the way it’s always been done around here.

And that, my friends, is how company policy begins.

(Courtesy of http://hackerboss.com/get-rid-of-templates/ )

"The issue was a company called Zynga, which makes online games, like FarmVille, that have become..."

Sun, 08 Nov 2009 14:06:31 -0800

“The issue was a company called Zynga, which makes online games, like FarmVille, that have become incredibly popular on Facebook among people who are missing parts of their brains.”

- http://www.fakesteve.net/2009/11/why-mainstream-media-is-dying.html