I have a Django project where I was utilizing bulk_create
to load a fairly large dataset, and this was some times resulting in the error MySQL server has gone away
. Reading though the documentation on this error, https://dev.mysql.com/doc/refman/8.0/en/gone-away.html I see that one of the possible causes for this is “you send a query to the server that is incorrect or too large” Turns out, I just needed to use the batch_size
parameter of bulk_create
. In my case, a batch of 56,000 records was triggering the error, so for safety, I switched to using bulk_created(..., batch_size=(10**4))
.
Welcome to my blog
As I discover new things in my field and solve troublesome problems, I will attempt to document my finds here.
Setting “application_name” when using Postgres with Django
I have been using pgAdmin 4 as my GUI tool for interacting with my PostgreSQL database servers. I had noticed the “Application” column, and that it was empty for my Django sessions, and wanted to fill that in with useful context..
Read MoreImproving Xapian backed Haystack searches in Django
I recently switched a Django project utilizing Haystack from Whoosh as the engine to Xapian. The performance significantly improved, but I was left with some deficiencies in the search results — this turned out to be due to the default settings.
Here are the necessary settings to improve the search result quality:
import xapian HAYSTACK_XAPIAN_FLAGS = ( xapian.QueryParser.FLAG_PHRASE | xapian.QueryParser.FLAG_BOOLEAN | xapian.QueryParser.FLAG_LOVEHATE | xapian.QueryParser.FLAG_WILDCARD | xapian.QueryParser.FLAG_PURE_NOT | xapian.QueryParser.FLAG_PARTIAL ) HAYSTACK_XAPIAN_STEMMING_STRATEGY = 'STEM_ALL' HAYSTACK_CONNECTIONS = { 'default': { 'ENGINE': 'xapian_backend.XapianEngine', 'PATH': root.path('data/xapian')(), 'FLAGS': HAYSTACK_XAPIAN_FLAGS, }, }
Before these settings, I was having problem with queries being apparently case sensitive, at least when using AutoQuery
, and failing partial searches. These changes improve case insensitivity and stemming, which still having almost 100x better performance than Whoosh.
Hopefully this helps someone out there looking for solutions.
Adding version numbers to your static files
When I started with Django, the version at the time was 1.5. Back then, we prepended STATIC_URL
to our static assets to reference them in our templates. With recent release, best practice is to you use the static
[1]. I often find myself wanting to append a version number to my static files, at least my CSS and JS files to ensure that browser see my new versions when I push an update. In this post, I make use of the `static` tag and its URL building to easily append a version number.
Sample settings.py for a Django microservice project
I present a sample settings.py
for a Django project, setup to accomplish the following requirements.
Read More
Request specific URLs in Django
Do you have a need to use different URL pattern sets based on specifics of an HTTP request? Recently, I needed to choose from a predefined URL pattern (ie. URLConf module) based on the domain name of the request, ie. HTTP_HOST. The URL patterns themselves are not dynamically set, just dynamically chosen.
Read More
Fixing UUID is not JSON serializable
I recently upgraded a Django project to version 1.8, and I am happily using Postgresql and Django’s new UUIDField
. While using django-eztables to do the server-side work for DataTables.net, I ran into an issue with a UUID field.
Loading posted JSON
If you’re using jQuery, and have an entire JSON document document that you would like to post to Django, the correct way to do it is not through a separate variable, but by posting the actual JSON.
Read More
Varying Django Settings By Environment
If you’ve progressed beyond the exploration phase in your Django journey, you’ve probably come to the point where, at the least, you would like to use one database during your development, and another, once published to your production system. You may even want to
Avoiding DeprecationWarning messages in your Apache logs
I recently upgraded a Django project from 1.4.x to 1.5.1, along with that came the anticipated change to ‘django.conf.urls’ which gives a deprecation warning on every request: