Welcome to my blog

Are you trying pip install pycrpto or otherwise trying to configure an application and you run into the following errors?

checking whether we are cross compiling… configure: error: in `/tmp/pip-install-2eizv0ma/pycrypto’:

configure: error: cannot run C compiled programs.

You may run into an issue that originates from improved security practices, and your only hint is in the first line of the error: “/tmp/pip-install-2eizv0ma/pycrypto”

Run the following: mount | grep tmp

If you see the option noexce in the mount point for your “/tmp” path, then that’s the issue — no execution of any file residing on that partition is allowed.

In my case, I was trying to install a Python package via pip which required some compilation of C code, so I was able get around the issue by set the TMPDIR environment variable:

TMPDIR=/path/to/alternate/tmp pip install pycrypto

I recently switched a Django project utilizing Haystack from Whoosh as the engine to Xapian. The performance significantly improved, but I was left with some deficiencies in the search results — this turned out to be due to the default settings.

Here are the necessary settings to improve the search result quality:

import xapian

HAYSTACK_XAPIAN_FLAGS = (
    xapian.QueryParser.FLAG_PHRASE |
    xapian.QueryParser.FLAG_BOOLEAN |
    xapian.QueryParser.FLAG_LOVEHATE |
    xapian.QueryParser.FLAG_WILDCARD |
    xapian.QueryParser.FLAG_PURE_NOT |
    xapian.QueryParser.FLAG_PARTIAL
)
HAYSTACK_XAPIAN_STEMMING_STRATEGY = 'STEM_ALL'

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'xapian_backend.XapianEngine',
        'PATH': root.path('data/xapian')(),
        'FLAGS': HAYSTACK_XAPIAN_FLAGS,
    },
}

Before these settings, I was having problem with queries being apparently case sensitive, at least when using AutoQuery, and failing partial searches. These changes improve case insensitivity and stemming, which still having almost 100x better performance than Whoosh.

Hopefully this helps someone out there looking for solutions.

Today, on a production system errors started arising from a Django based application that made use of PostGIS features. All SQL queries that included any geo data were failing. This was on PostgreSQL 9.6 on Centos 7.

Read More

This is an update to Full page screenshots with Python and Selenium updated to work with Python 3, and fix an issue with single pages.

Read More

I was getting PHP `unseralize` errors when working with some WordPress option values.

PHP Notice: unserialize(): Error at offset 4854 of 24578 bytes

Read More

If you ever find yourself using Apache as a frontend server or reverse proxy, as opposed to one Nginx, for example. You may want to be sure to pass the actual host IP and scheme information back to your application so that it may function properly. By default, mod_proxy sets the X-Forwarded-For header. But that header is a list, and is used by other types of proxies. You may want to set the X-Forwarded-Proto and X-Real-IP as well. The following entries into your VirtualHost or similar should do the job:

Read More

If you’re getting “Incorrect format parameter” errors on attempts to import files into phpMyAdmin, be sure to get your post_max_size and upload_max_filesize settings in php.ini. More than likely, they are too small for the file that you’re uploading.

Currently, none of the major Selenium drivers (browsers) support the ability to easily take a screenshot of an entire web page. The following function takes multiple screenshots through the viewport and scrolls between screenshots, then stitches the resulting images into a single PNG.

Read More

I have been writing some automation code a few dozen websites, and I wanted to generate SSL keys, and I couldn’t get the `subprocess` call to work properly, so I thought I would post the final solution. The argument to `communicate()` is critical.
Read More

Today, I had the need to replicate an encrypted query string token to inter-operate with a third-party commercial application. I was able to determine the library, symmetrical algorithm and secret key being used to create the token. Turns out, it was an ASP.net web application using the Chilkat .NET library to do the encryption and decryption. Specifically, it was the Chilkat AES (aka Rijndael) methods being used.
Read More