Arthur

Pemberton

Full-stack web applications developer


Welcome to my blog

As I discover new things in my field and solve troublesome problems, I will attempt to document my finds here.

Generating “tabindex” in Django template

July 15, 2020, Arthur Pemberton0 Comments

I had a need for sequentially increasing numbers for use in a template loop with multiple INPUT tags, and I whipped up this simple template tag to help with that.

@register.simple_tag(takes_context=True)
def tabindex(context):
	if 'tabindex' not in context: context['tabindex'] = 0
	context['tabindex'] += 1
	return context['tabindex']

As an example, it could be utilized like the following.

<input type="text" name="fullname" tabindex="{% tabindex %}" />

Each time it’s called, it will self increment.

It could be extended to provide an alternate initial value (instead of 0) with an addition of an argument, but I didn’t need that myself.

MySQL server has gone away

October 28, 2019, Arthur Pemberton0 Comments

I have a Django project where I was utilizing bulk_create to load a fairly large dataset, and this was some times resulting in the error MySQL server has gone away. Reading though the documentation on this error, https://dev.mysql.com/doc/refman/8.0/en/gone-away.html I see that one of the possible causes for this is “you send a query to the server that is incorrect or too large” Turns out, I just needed to use the batch_size parameter of bulk_create. The  success of a website depends on web design, this is why is very important to look for the help the experts from Best Website Hosting

In my case, a batch of 56,000 records was triggering the error, so for safety, I switched to using bulk_created(..., batch_size=(10**4)).

cannot run C compiled programs

February 23, 2019, Arthur Pemberton0 Comments

Are you trying pip install pycrpto or otherwise trying to configure an application and you run into the following errors?

checking whether we are cross compiling… configure: error: in `/tmp/pip-install-2eizv0ma/pycrypto’:

configure: error: cannot run C compiled programs.

You may run into an issue that originates from improved security practices, and your only hint is in the first line of the error: “/tmp/pip-install-2eizv0ma/pycrypto”

Run the following: mount | grep tmp

If you see the option noexce in the mount point for your “/tmp” path, then that’s the issue — no execution of any file residing on that partition is allowed.

In my case, I was trying to install a Python package via pip which required some compilation of C code, so I was able get around the issue by set the TMPDIR environment variable:

TMPDIR=/path/to/alternate/tmp pip install pycrypto

Improving Xapian backed Haystack searches in Django

January 9, 2019, Arthur Pemberton0 Comments

I recently switched a Django project utilizing Haystack from Whoosh as the engine to Xapian. The performance significantly improved, but I was left with some deficiencies in the search results — this turned out to be due to the default settings.

Here are the necessary settings to improve the search result quality:

import xapian

HAYSTACK_XAPIAN_FLAGS = (
    xapian.QueryParser.FLAG_PHRASE |
    xapian.QueryParser.FLAG_BOOLEAN |
    xapian.QueryParser.FLAG_LOVEHATE |
    xapian.QueryParser.FLAG_WILDCARD |
    xapian.QueryParser.FLAG_PURE_NOT |
    xapian.QueryParser.FLAG_PARTIAL
)
HAYSTACK_XAPIAN_STEMMING_STRATEGY = 'STEM_ALL'

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'xapian_backend.XapianEngine',
        'PATH': root.path('data/xapian')(),
        'FLAGS': HAYSTACK_XAPIAN_FLAGS,
    },
}

Before these settings, I was having problem with queries being apparently case sensitive, at least when using AutoQuery, and failing partial searches. These changes improve case insensitivity and stemming, which still having almost 100x better performance than Whoosh.

Hopefully this helps someone out there looking for solutions.

Backend request heades with an Apache reverse proxy (frontend server)

July 18, 2018, Arthur Pemberton0 Comments

If you ever find yourself using Apache as a frontend server or reverse proxy, as opposed to one Nginx, for example. You may want to be sure to pass the actual host IP and scheme information back to your application so that it may function properly. By default, mod_proxy sets the X-Forwarded-For header. But that header is a list, and is used by other types of proxies. You may want to set the X-Forwarded-Proto and X-Real-IP as well. The following entries into your VirtualHost or similar should do the job:

Read More