Full-stack web applications developer

Welcome to my blog

As I discover new things in my field and solve troublesome problems, I will attempt to document my finds here.

MySQL server has gone away

On October 28, 2019, by Arthur Pemberton, 0 Comments

I have a Django project where I was utilizing bulk_create to load a fairly large dataset, and this was some times resulting in the error MySQL server has gone away. Reading though the documentation on this error, I see that one of the possible causes for this is “you send a query to the server that is incorrect or too large” Turns out, I just needed to use the batch_size parameter of bulk_create . In my case, a batch of 56,000 records was triggering the error, so for safety, I switched to using bulk_created(..., batch_size=(10**4)).

Improving Xapian backed Haystack searches in Django

On January 9, 2019, by Arthur Pemberton, 0 Comments

I recently switched a Django project utilizing Haystack from Whoosh as the engine to Xapian. The performance significantly improved, but I was left with some deficiencies in the search results — this turned out to be due to the default settings.

Here are the necessary settings to improve the search result quality:

import xapian

    xapian.QueryParser.FLAG_PHRASE |
    xapian.QueryParser.FLAG_BOOLEAN |
    xapian.QueryParser.FLAG_LOVEHATE |
    xapian.QueryParser.FLAG_WILDCARD |
    xapian.QueryParser.FLAG_PURE_NOT |

    'default': {
        'ENGINE': 'xapian_backend.XapianEngine',
        'PATH': root.path('data/xapian')(),

Before these settings, I was having problem with queries being apparently case sensitive, at least when using AutoQuery, and failing partial searches. These changes improve case insensitivity and stemming, which still having almost 100x better performance than Whoosh.

Hopefully this helps someone out there looking for solutions.

Adding version numbers to your static files

On June 12, 2016, by Arthur Pemberton, 0 Comments

When I started with Django, the version at the time was 1.5. Back then, we prepended STATIC_URL to our static assets to reference them in our templates. With recent release, best practice is to you use the static [1]. I often find myself wanting to append a version number to my static files, at least my CSS and JS files to ensure that browser see my new versions when I push an update. In this post, I make use of the `static` tag and its URL building to easily append a version number.

Read More

Request specific URLs in Django

On September 17, 2015, by Arthur Pemberton, 0 Comments

Do you have a need to use different URL pattern sets based on specifics of an HTTP request? Recently, I needed to choose from a predefined URL pattern (ie. URLConf module) based on the domain name of the request, ie. HTTP_HOST. The URL patterns themselves are not dynamically set, just dynamically chosen.
Read More

Loading posted JSON

On April 24, 2014, by Arthur Pemberton, 0 Comments

If you’re using jQuery, and have an entire JSON document document that you would like to post to Django, the correct way to do it is not through a separate variable, but by posting the actual JSON.
Read More

Varying Django Settings By Environment

On October 21, 2013, by Arthur Pemberton, 0 Comments

If you’ve progressed beyond the exploration phase in your Django journey, you’ve probably come to the point where, at the least, you would like to use one database during your development, and another, once published to your production system. You may even want to

Read More