Cubus Arts Blog

 
Andreas Philippi

Truncate typo3 indexed search tables

Andreas Philippi

Huge table index_rel never stops growing

On one of our webservers we have a lot of typo3-based sites. Some of them make use of the indexedsearch extension, which uses some database tables to store the indexed data. These tables can grow really huge .

Bad performance for search queries

The table index_rel grows very fast, having over 6.000.000 entries for some typo3 installations. Even a simple search takes over 30 seconds to complete. I searched for an elegant solution with very little success.

Our current solution: periodical brute force

Here is a brute force solution: a ruby-script that finds all databases that have a index_rel-table and truncates all indexed search tables if they reach a given size. The tables will be automatically refilled every time a page is accessed.

The script truncates the “cache tables” also, the “indexed search tables” will not be rebuild by public requests if only the cache for a given page is returned by a request.

If you want to refill the search instantly index can write a script the crawls the site using wget —spider or something similar.

The script

We call this via cron once a week – every friday night

Gabriel Târnovan

Simple Audit for Rails Apps

Gabriel Târnovan

simple_audit is a lightweight auditing solution for Rails apps. Instead of auditing all attributes of your models, simple_audit allows you to specify what data is relevant for auditing. This simple idea makes auditing complex model aggregations (through has_many, has_one, habtm associations) trivial.

Audit in 2 minutes:

Result will look like this:
Screenshot of helper result

Why ?

simple_audit is intended as a straightforward auditing solution for complex model associations. In a normalized data structure (which is usually the case when using SQL), a core business model will aggregate data from several associated entities. More often than not in such cases, only the core model is of interest from an auditing point of view (the parent in the has_xxx relations).

One approach would be to audit all aggregated entities sepparately. The downside of this is the high complexity of retracing all changes triggered on each save. Another minus is the increased database workload.

simple_audit takes a simpler approach: you decide for each audited model what data needs to be audited by specifying a serializable representation of all relevant information – including the any relevant aggregated entities (e.g. the housing_unit_bookings in the above example) This data will be serialized on every save. The available helper methods will render a clear audit trail (see above).

Limitations

This is not a solution for versioning. Changed data can not be restored. Use acts_as_audited or acts_as_versioned for that.

Dependencies

The current user is assumed to be available by calling User.current You can use sentient_user or roll your own solution.

More information

Read more on the github page. Contributions are welcome!

Andreas Philippi

Typo3 TMENU_LAYERS generates endless recursion loop

Andreas Philippi

What happens?

You have a typo3-template which uses TMENU_LAYERS. You may notice that you’ll get “too much recursion” javascript-errors on all browsers.

microtime is not enough :)

This errors happens because of a unique-ID-generating-function uses microtime. On some machines the microtime-based function does not provide unique IDs.

Under certain conditions, when using TMENU_LAYERS the layers containing the navigation items get non-unique IDs.

We noticed this error after migrating several sites in a virtualized environment on a very fast machine.

our solution

To correct this error we used the php-function uniqid to give us unique ids.

We proceeded this way:

  • locate the file /typo3/sysext/cms/tslib/media/scripts/tmenu_layers.php
  • search for the string: substr(md5(microtime()),0,6);
  • replace the string using: str_replace(‘.’,‘’,uniqid(‘’,true));

We also filed a bug

Mihai Târnovan

Installing mysql gem with Percona XtraDB

Mihai Târnovan

After installing the mysql gem, we were seeing uninitialized constant MysqlCompat::MysqlRes errors all over the place, although the gem installed fine.

This was on FreeBSD 8.1. Turns out Percona XtraDB binary doesn’t include the mysql client libs, so you need to install those as well. On FreeBSD just do

pkg_add -r mysql51-client

Then reinstall the gem:

env ARCHFLAGS="-arch x86_64" gem i mysql -- --with-mysql-config=/usr/local/percona-xtradb/bin/mysql_config

Mihai Târnovan

Ruby Eval stacktrace position information

Mihai Târnovan

We’ve been often annoyed by warnings or exceptions like this: (eval):231: some error or warning. They make it very difficult to pinpoint the exact location of the problem.

The problem is usually cause by module_eval being called without file and line information. There’s an interesting write-up about eval and position information on Ola Bini’s blog. One of his readers observed that when using module_eval like this you’ll get the incorrect line number:

However, I noticed that if you write the heredoc inline you’ll get the correct line in the backtrace:

__LINE__ + 1 accounts for the line with the module_eval itself. I tested this with Ruby 1.8.6 and Ruby 1.9.1 and it applies for both, tough 1.9 at least puts more lines in the backtrace with the position of the module_eval call.

Passing __FILE__ and __LINE__ is better than skipping them, but can lead to false line numbers unless you use the 2nd form with the inlined heredoc.

Mihai Târnovan

Dear typographers

Mihai Târnovan

The romanian language has some specific diacritics which can radically change the meaning of a word.

The problem

For example țânc vs. tanc, tata vs. țâța vs. țața (kid vs. tank, father vs. breast vs. aunt). The problem with this characters and their implementation in keyboard layouts, charsets and fonts, is well known and has been explained in almost academic detail and extremely well documented manner by Cristian Secară.

Microsoft’s “solution” and its consequences

When Microsoft implemented the first romanian keyboard layout, they mapped the keys which should have produced these characters to similar turkish characters instead. They could have fixed this as early as Windows 2000 (around then the romanian standard SR 13411:1999 was already in place). Instead, they did nothing until 2007.

By then, their core fonts were already the de facto standard for web typography and everyone had fonts installed on their computer which lacked characters for lowercase and uppercase ș and ț (Unicode code points U+0218, U+0219, U+021A and U+021B).

Fixing the problem and causing new ones

It took Microsoft until 2006 to get it right. They released a patch which added the missing characters to Times New Roman, Arial and Verdana. When they released Windows Vista, they still didn’t get it right for all fonts – Verdana was only fixed by a later patch.

Ok, so Microsoft release a patch for XP and did the right thing with Vista. So everything’s fine, right ? Well, not quite. If you use the Romanian keyboard layout on Vista or MacOS, you will write text containing the proper Unicode characters. But a vast majority of romanian users don’t have these characters in their fonts, because they didn’t apply the font patch. So for them, a ș written with the proper romanian character appears as a square.

Possible solutions

  1. Use fonts that never had this the problem in the first place.
    • Palatino Linotype
    • Tahoma

    This is of course too limiting for practical purposes, especially in web design.

  2. Use old keyboard layout.

    I still haven’t figured out how to do this for MacOS without custom keyboard layouts, but it should be fairly easy to do this for Vista and Linux (use Romanian legacy or a similar keyboard layout). You’ll write text with turkish diacritics, but at least all your visitors will be able to display them correctly.

  3. Serve two versions of content, one for Windows XP (and older) users with turkish characters, and one for all the others.

    Due to the complexity of this solution, I’m not sure this is a good idea. (we do this for travelbox with a Rack filter — but that was implemented more to learn about Rack than as a long term solution for this problem)