In this post I’ll show a few tricks I use to make JSON fit into Django more seamlessly. The first is a lesson on coding that everybody should know.
Parsing external JSON:
Whenever you take in JSON from “strange computers” (which is basically any computer) it works most of the time.
As C3p0 said: R2D2, you know better than to trust a strange computer.
The following example code does two things that are super important when it comes to processing JSON in python.
- Handles the exception in case the JSON is not valid. This happens all the time with unescaped characters, server errors, etc. In the code below raw_data is a string that allegedly contains valid JSON. Not if, but WHEN raw_data is some random invisible UTF-8 character, you want to recover from it, and you want to log what you know about the problem.
- The parser keeps the keys in order. Without the object_pairs_hook=OrderedDict argument, json_data’s internal dictionary keys will be in whatever order your Python interpreter felt like that day. I’ve found for some types of JSON data order does matter and most systems that claim to emit ‘ordered JSON keys’ don’t realize that isn’t how JSON works.
try:
json_data = json.loads(raw_data, object_pairs_hook=OrderedDict)
except JSONDecodeError:
logger.exception('Error when parsing JSON, raw data was ' + str(raw_data))
raise ExternalAPIException('Unable to do my work! Invalid JSON data returned.')
Now that you’ve seen a basic example of processing JSON, the rest of this post will be about storing JSON and integrating it nicely into the Django admin.
Some background on Django + JSON:
If you are using Postgres life is easy with JSON and Django because way back in Django 1.9 (December 2015) the Postgres only models.JSONField() came out. Prior to that you’d be using a TEXT field to store JSON.
MySQL introduced a JSON type in version 5.7 but Django doesn’t support it out of the box as of 2.1. Django requires you to implement JSON storage in a TextField (MySQL type longtext) and leave it at that. I’m ‘guilty’ of using MySQL TEXT to store JSON on a few older projects and it works fine for me.
With the MySQL native JSON field you get additional JSON validation at the database level (which you should be doing at the API level already), optimized storage (yes please!), and ability to use non-standard SQL to index and query values inside the JSON. The last part is interesting but also makes me cringe a little because it is NoSQL wrapped inside an SQL database… Too much rope to get tangled up in.
For MySQL users there is another way…. if you really want to use the native JSON type in MySQL the Django-Mysql package is available.
Consider the following contrived model (models.py):
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import models
from django_mysql.models import JSONField
class BookExample(models.Model):
id = models.BigAutoField(primary_key=True, editable=False)
name = models.CharField(max_length=100)
detail_text = models.TextField()
detail_json = JSONField() # requires Django-Mysql package
class Meta:
managed = True
db_table = 'book_example'
verbose_name = 'Book Example'
verbose_name_plural = 'Book Examples'
For illustrative purposes it has a TEXT based column and a JSON based column.
Running makemigrations + migrate will generate the following MySQL table:
CREATE TABLE `book_example` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `name` varchar(100) NOT NULL, `detail_text` longtext NOT NULL, `detail_json` json NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB;
Django admin wiring (admin.py):
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.contrib import admin
from example.models import BookExample
class BookExampleAdmin(admin.ModelAdmin):
list_display = ('name',)
admin.site.register(BookExample, BookExampleAdmin)
Now let’s fill in some sufficiently complex (contrived) JSON:
{
"title": "Anathem",
"authors": ["Neal Stephenson"],
"publication_year": 2008,
"description_sort": "Anathem is a science fiction novel by Neal Stephenson, published in 2008. Major themes include the many-worlds interpretation of quantum mechanics...",
"chapters": [{
"title": "Extramuros",
"number": 1,
"summary": "The story takes place on Arbre, a planet similar to Earth...",
"page_count": 23
},
{
"title": "Cloister",
"number": 2,
"summary": "Erasmas describes several buildings of the Concent, namely the Scriptiorium...",
"page_count": 14
},
{
"title": "Aut",
"number": 3,
"summary": "Erasmas describes the Mynster, which is a building housing the Concent's clock...",
"page_count": 34
}
],
"language": "English",
"page_count": 937
}
Here is how it would look in the admin using default settings:
Both fields render as <textarea> fields (which are fully editable) and it is really hard to read the contents. The first field is a plain textarea and it will accept any data. The second field has JSON validation wired to it so the form won’t go through unless the JSON is valid.
Let’s make the JSON look better in the admin and make it read only:
When I’m storing JSON in a database it is typically either:
- 3rd party data associated with the record that needs to be kept but is rarely used.
- A document format used by a front end tool or mobile app (designer tool, dashboard layout, or mobile app data sync).
It normally wouldn’t make sense to edit the raw JSON data in the admin. But it would make sense to edit other properties on the row and at the same time see a nicely formatted version of the JSON next to the other fields.
Here is how to make the JSON read only in the Django admin and format nicely.
First in the model, wire up a function to output the formatted JSON. This relies on the python Pygments package.
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.db import models
from django.utils.safestring import mark_safe
from django_mysql.models import JSONField
# new imports!
import json
from pygments import highlight
from pygments.formatters.html import HtmlFormatter
from pygments.lexers.data import JsonLexer
class BookExample(models.Model):
id = models.BigAutoField(primary_key=True, editable=False)
name = models.CharField(max_length=100)
detail_text = models.TextField()
detail_json = JSONField() # requires Django-Mysql package
def detail_json_formatted(self):
# dump the json with indentation set
# example for detail_text TextField
# json_obj = json.loads(self.detail_text)
# data = json.dumps(json_obj, indent=2)
# with JSON field, no need to do .loads
data = json.dumps(self.detail_json, indent=2)
# format it with pygments and highlight it
formatter = HtmlFormatter(style='colorful')
response = highlight(data, JsonLexer(), formatter)
# include the style sheet
style = "<style>" + formatter.get_style_defs() + "</style><br/>"
return mark_safe(style + response)
detail_json_formatted.short_description = 'Details Formatted'
class Meta:
managed = True
db_table = 'book_example'
verbose_name = 'Book Example'
verbose_name_plural = 'Book Examples'
And in the admin file:
# -*- coding: utf-8 -*-
from __future__ import unicode_literals
from django.contrib import admin
from example.models import BookExample
class BookExampleAdmin(admin.ModelAdmin):
fieldsets = [
(None, {'fields': ['name']}),
('JSON', {'fields': ['detail_json_formatted']}),
]
list_display = ('name',)
readonly_fields = ('detail_json', 'detail_json_formatted')
admin.site.register(BookExample, BookExampleAdmin)
Now the admin has a nice easy to read formatted representation of the JSON complete with inline styles:
Other ideas for using JSON with Django:
- The Django Rest Framework Serializer JSONField.
- Working with JSON in Python.
- Emitting JSON from Django with JsonReponse.