Sails.js without express

A few weeks ago I started working with Sails.js and besides the not (yet) very powerful ORM Waterline I’m quite happy with it. A few days ago I came to a point where I needed to create a worker process which does some housekeeping, listening to RabbitMQ messages. This worker needed access to the Sails.js configuration, services and models, but didn’t require an http(express) server.

The solution is quite simple and I’m sure this will be useful for others too. Here’s the source:

var app = require('sails');

app.lift({

  log: {
    noShip: true
  },

  // Don't run the bootstrap
  bootstrap: function (done){ return done(); },

  hooks: {
    noRoutes: function (sails){
      return {
        initialize: function (cb){
          sails.log.verbose('Wiping out explicit routes to make worker load faster...');
          sails.config.routes = {};
          return cb();
        }
      };
    },
    grunt: false,
    csrf: false,
    cors: false,
    controllers: false,
    policies: false,
    i18n: false,
    request: false,
    responses: false,
    http: false,
    sockets: false,
    views: false,
    pubsub: false,
  }
}, function (err) {
  if (err) throw err;

  // Sails is lifted, do something here
});

MongoDB Aggregation: Group by any time interval

Since version 2.4 MongoDB offers a powerful aggregation framework, which offers great functionality without the hassle of map/reduce. You can group your results by any (calculated) field. Using the date operators you can for instance group your results by day and sum your results.

Assuming you have a collection with a MongoDB Date and a value you field, you can aggregate your data like this:

db.collection.aggregate(
{
'$group' : {
'date' : { '$year' : '$datefield','$month' : '$datefield','$day' : '$datefield' },
'sum_value' : { '$sum' : '$value_field'}
}
});

This is pretty neat when it comes to days, months or years, but what if you want to group your date by an interval of 2hours?

Instead of saving your dates as MongoDB Dates, you could save your dates as unix timestamps and have Mongo calculate the intervals. MongoDB offers a set of arithmetic functions like $subtract and $divide, but unfortunately it doesn’t yet provide any round/floor/truncation functions.  But wait! There’s a solution!

You can (ab)use the $mod operator to remove the decimal digits.

In pseudocode, the formula would look like this.

60seconds * 60minutes * 2hours = 7200

timestamp/7200 – (  (timestamp/7200) mod 1)

In MongoDB your aggregation function would look like this:

db.collection.aggregate(
{
'$group' : {
'date' : { '$subtract' :[ {'$divide' : ['$timestamp_field', 7200 ]}, { '$mod' : [{'$divide' : ['$timestamp_field', 7200 ]},1] } ] },
'sum_value' : { '$sum' : '$value_field'}
}
});

When you receive the resultset, you can simply multiply the returned date field with 7200 and voilá! That’s your final Unix timestamp, grouped by every 2 hours!

 

#2013 – Lost connection to MySQL server during query – OR: MySQL VOODOO!

Today one of my scripts prompted the #2013 MySQL error while querying a huge innoDB table (31GB in ~154million rows). Some queries worked, some just failed.

Looking at the logfiles, I saw the following message:

InnoDB: Page checksum 1840120551 (32bit_calc: 1224736073), prior-to-4.0.14-form checksum 1811838366
InnoDB: stored checksum 3031359782, prior-to-4.0.14-form stored checksum 1811838366
InnoDB: Page lsn 47 623631862, low 4 bytes of lsn at page end 623631862
InnoDB: Page number (if stored to page already) 68664,
InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 22
InnoDB: Page may be an index page where index id is 35
InnoDB: (index “PRIMARY” of table “tracking”.”banner” /* Partition “p4″ */)
InnoDB: Database page corruption on disk or a failed
InnoDB: file read of page 68664.
InnoDB: You may have to recover from a backup.
InnoDB: It is also possible that your operating
InnoDB: system has corrupted its own file cache
InnoDB: and rebooting your computer removes the
InnoDB: error.
InnoDB: If the corrupt page is an index page
InnoDB: you can also try to fix the corruption
InnoDB: by dumping, dropping, and reimporting
InnoDB: the corrupt table. You can use CHECK
InnoDB: TABLE to scan your table for corruption.
InnoDB: See also http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.

I tried to check the table -> mysql failed.

I tired to recover the table -> mysql failed.

The filesystem as well as the RAID were in a healthy condition and even rebooting the machine didn’t to the trick (would have been odd if it did). So, I asked my friend Google, and Google pointed me to an interesting post:

I ran into a problem where, when dealing with HUGE tables (location tables for http://Stiggler.com ), there was an innodb page error, and mysql would try over and over to repair it, and would inform me that it could not repair it (and would then try again, etc).

[…]

I put the force_recovery mode to 1, then restarted mysqld, exported the entire database (i expected to get an error when it got to the bad table, but i never had a problem). After dumping the database, i removed the force_recovery option from my.cnf and restarted the service, and after a few moments, it started back up, and the problem was gone.

I can’t remember where exactly I found this quote (sorry!), but what this guy was basically trying to say: Dump your database and wait for the magic to happen!

So I did, tried my query again and it just works! I dunno what magic voodoo effect dumping a database with mysqldump does, but it just works. Really, this seemed like the most stupid approach to fix this issue, but it did the trick!

 

Next time you hit the “#2013 Lost connection” issue.. try dumping your database!

Creating Website Screenshots on Linux

Today a customer asked for automated screenshots of his website. So first thing to do was asking Google how it could be accomplished on a Linux WebServer. Most of the results referred to installing an X-Server, using Firefox and stuff. This sounded a bit tricky and quite frankly.. over the top.

The solution I came up with is (IMO) much easier and less prone to failures: WKHTMLTOPDF + Imagick Convert.

WKHTMLTOPDF uses QT with a Webkit Widget – it renders HTML pages inlcuding JavaScript like Apple’s Safari Browser. So creating a PDF from a website became very easy:

Usage: wkhtmltopdf -L 0 -T 0 -R 0 -B 0 http://mywebsite.com/ test.pdf

That’s all you need to create a PDF from your website including all images, javascript rendered contents and such. The parameters “-L -T -R -B” define the the border margin of the PDF document. The default value is 10mm – I prefer zero.

Now that we have a PDF of the website, we need to convert it into a JPG file. This can be acomplished using ImageMagick.

Usage:  convert -density 600 test.pdf -scale 2000×1000 test.jpg

In case your webpage is pretty long it might happen that your PDF contains multiple pages. Imagick generates one JPG file per page named test-0.jpg, test-1.jpg and so on. All you need to do is combining the images with Imagick in a second step, which is also pretty easy and straighforward:

Usage:  convert test-* -append single.jpg

That’s it… now you have a single image file from your website.

Creating screenshots with FFmpeg is slow?

Just a quick note for everyone who’s using FFmpeg for creating screenshots from video files. Today I noticed that FFmpeg can be VERY slow on large/long movie files, but there’s a pretty neat trick to speed up the screenshot generation.

I used to create my screenshots this way:

ffmpeg -i /var/www/input.mov -y -f image2 -ss 1234  -sameq -t 0.001 “/var/www/screenshot.jpg” 2>&1

and on huge files it sometimes took minutes. The solution is simple:

ffmpeg -ss 1234 -i /var/www/input.mov -y -f image2  -sameq -t 0.001 “/var/www/screenshot.jpg” 2>&1

Put the -ss parameter in front of the input file and FFmpeg skips to the selected frame almost instantly.

hf! :)

6 ways to kill your server – learning how to scale the hard way

Learning how to scale isn’t easy without any prior experience. Nowadays you have plenty of websites like highscalability.com to get some inspiration, but unfortunately there is no solution that fits all websites and needs. You still have to think on your own to find a concept that works for your requirements. So did I.

Read more

Magento – Performance Analysis

In our office we are currently evaluating the pros and cons of Magento. Besides all the great features it offers it has a huge con – the performance. We read a lot about it but couldn’t find any reliable statements about how fast or slow it really it is. So I put a plain Magento install on my testing machine and gave it a shot..

Read more

Setting up a Testing-Environment

Let’s start with the basic setup for my test scenarios. In our office we had an old and dusty, but still working Intel Xeon 3305 left. It has 2Gigs of RAM and a 1TB SATA harddisc. It’s nothing special, but still good enough for some benchmarks.
Read more

1 2