Jeremy Satterfield

Resume

2026-03-02T14:50:00+00:00

Full-stack engineer with over 19 years of experience building web applications and APIs primarily using Python and Django. Emphasis on high-quality, maintainable code and reliable, modern infrastructure. Proven leadership in designing systems architecture, mentoring teammates and delivering reliable software for complex business needs.

Skills

Languages & Frameworks: Python, Django, Django REST Framework, Celery, Starlette, JavaScript, React, Angular.js, Stylus, CSS, HTML

Databases: PostgreSQL, MySQL, Redis

Cloud & DevOps: AWS, GCP, Docker, Kubernetes, ArgoCD, SaltStack, Terraform, Scalr, GitLab CI/CD, GitHub Actions, Spinnaker

Monitoring & Tools: Datadog, Kafka, Nginx, Graphite/Statsd, StackStorm

OS & Systems: Linux Administration, Serial Communication, Hardware Integration

Experience

Peloton

Staff Software Engineer (Sep 2022 - Jan 2026)

Senior Software Engineer (Nov 2020 - Sep 2022)

Technical lead for the B2B team, defining software architecture, infrastructure needs, team development processes and security guidelines.
Designed and implemented an API and dynamic data processing workflow for Corporate Wellness partners, supporting dynamic, per-partner data formats and workflows to provide employee eligibility data.
Developed an SSO-based onboarding flow, enabling secure eligibility checks and signup, improving security and user experience.
Led multiple infrastructure migration projects, including transitioning hosting providers from GCP to AWS, Kubernetes cluster replacement, and moving CI/CD pipelines from Gitlab to Github.
Mentored other engineering team members and collaborated with team leadership on technical planning and team roadmaps.

Tech stack: Linux, Python, Django, PostgreSQL, REST, GraphQL, Kafka, Redis, Pandas, Celery, AWS, Kubernetes, ArgoCD, Terraform, Scalr, Datadog

Peerfit

Senior Backend Software Engineer (May 2019 - Nov 2020)

Built API for external gym partner integrations to communicate member participation.
Re-architected the core corporate wellness data processing pipelines managing member eligibility and signups.
Contributed to team development process improvements and system reliability.

Tech Stack: Linux, Python, Django, PostgreSQL, Django Rest Framework, Starlette, Kubernetes, Google Cloud Platform, Gitlab CI/CD

Summit ESP (acquired by Halliburton in 2017)

Senior Software Developer (Dec 2014 - Apr 2019)

Led architecture and development of internal ticketing and ERP systems.
Architected migration from monolithic application to specialized microservices.
Developed PySide/Qt desktop application to provide data entry and syncing with the ERP.
Managed DevOps infrastructure and transitioning to a container-based deployment pipeline.
Involved in planning and design of new features with product managers and end users.

Tech Stack: Linux, Python, Django, MySQL, Javascript, CoffeeScript, Angular.js, Stylus/CSS, SaltStack, Docker, Kubernetes, Spinnaker, Amazon Web Services, Grunt, Gulp, StackStorm, Graphite/Statsd, Hubot

Projects:

ConsumerAffairs.com

Senior Developer (Dec 2010 - Dec 2014)

Converted site from static PHP to a dynamic Python/Django architecture.
Improved site performance and SEO, increasing scalability and site traffic.
Led development on SaaS product which grew to become the company’s primary revenue source.

Tech Stack: Linux, Python, Django, MySQL, Javascript, CSS, SaltStack, Celery

Projects:

OpenTundra

Web Developer (Dec 2008 - Dec 2010)

Built API, web and reporting interfaces for vehicle telemetry and asset tracking applications.
Developed kiosk software for fluid dispensing control and reporting.

Tech Stack: Python, Django, Coldfusion, MySQL, Javascript, CSS, Serial Communication, Hardware Interaction

AT&T Mobility

Analyst/Developer (Mar 2007 - Nov 2008)

Built and maintained for several intranet applications, including custom CRMs and reporting interfaces.
Developed role-based authentication framework internal applications.

Tech Stack: Coldfusion, MS-SQL Server, Javascript, CSS

Community Involvement

TulsaWebDevs (Volunteer, 2010 - 2020)

Organized events, civic hackathons, and meetups to grow the local dev community.

Projects: TRIF, Tulsa Transit GTFS, Django Boundary Service

Code for Tulsa (Delivery Lead, 2014 - 2019)

Coordinated projects and connected volunteers to civic tech initiatives.

Civic Ninjas (Developer, 2013 - 2014)

Built tools for analyzing health demographics in local communities.

Projects: HealthAround.me

Education

Tulsa Community College - Computer Science (2001 - 2003)

Tulsa Technology Center - Computer Science (1999 - 2002)

The Pains of MySQL Perfomance Under Docker

2018-01-15T21:50:21+00:00

Over the course of the last several days I've been plagued with a couple pretty massive performance issues running MySQL under Docker(-compose) in our developer environments. Everyone in our office is running either a Mac or some flavor of Linux. In both cases I spent a couple of days search for a solution, running into several roadblocks, and exhausting many Google search terms. Since I had such a difficult time finding the solution, here are the symptoms I was seeing and what finally worked in our case, maybe it'll save someone else time.

Background

In both cases we were seeing incredibly long query times in certain areas of the site. The particular queries were those which we would expect to see performance issues exacerbated if they existed, large result sets with several joins and sorted. However, running the same queries against the same data on the same machines against MySQL installed natively on the host still ran these queries in fractions of a second, while the Docker instances took several minutes and spiked the CPU usage. This is an existing app with existing data, but only about 6GB on disk, so nothing big.

We are using MySQL's official images on Docker Hub. Using MySQL 5.6 to match what we currently have on our production databases. No customizations to speak of other than setting environment variables for user credentials and mounting named volumes to load and store the data.

The Issue with Mac

This was the fairly simple one to fix. After past experiences with similar CPU pegging performance issues with MySQL, I suspected this to be something to do with IO being limited somewhere. It took a bit of looking around, but I eventually ran across some forum posts talking about fsync flushing issues. I've run across enough similar issues in the past, it's a wonder my mind doesn't start there.

So reading on, I came across this gist which is a script to turn off fsync within the Hyperkit that Docker is running under Docker for Mac. This did the trick. I started up the docker containers and loaded page with the worst offending queries and it loaded in a matter of seconds.

Since this was an issue with something as low level as fsync, you'd be right in thinking that there's an issue with the driver Docker is using to access the disk. Further reading in one of the Github issues I found that they had apparently made changes in 17.11+ to use`raw` format which no longer needed these fsync changes. Docker 17.12 landed in stable a couple days later and appears to have indeed fixed the issue.

The Issue with Linux

This proved to be a much tricker issue to fix and the solution turned out to be one I tried by chance and not anything I found existing on the web. Like Mac I was seeing CPU spike for complex queries and several minutes to return results. It turns out the Mac issue is wide spread enough that it's nearly impossible to do a Google search that gives you Linux specific steps to try to narrow it down.

Here is a short list of many of the steps that did not work or showed minimal improvements:

forcing Docker's storage driver to something other than the default (I ended up on overlay2 because it's recommended, but again there's no noticeable difference)
using a bind (named) volume for storing data files
using a new mount volume from the host machine
using a mount volume to the existing data files for my host machines existing MySQL data files, which worked great with a native instance
upgrading to MySQL 5.7
providing a custom settings file with tuned settings that worked for the native instance
running a standalone Docker instance (read: not compose) with minimal parameters provided
change how my root file system was mounted on boot to include barrier=0

At this point I was 3 days in and pretty frustrated. I decided to change tactics, and I set out to compare the performance of the Docker vs native instance in a better way. I started out by comparing the EXPLAIN results for the same query in both environments, and instantly noticed a pretty big difference, the Docker instance wasn't using a pretty important PRIMARY index which the native instance is using. I dug a little deeper and no matter how the data was stored (bind v mount) if it was going through the Docker instance it wasn't using the index. I verified that the index existing in all instances.

Finally, for no reason that I can particularly pin down, I decided to run MySQL's optimize script on all the tables. That fixed the issue. I have no idea why this fixed anything, but it has done the trick for all the Linux systems in our office. If you have an explanation I'd be happy to hear about it down in the comments.

Python Catalog Data Structures

2017-11-10T15:07:49+00:00

I recently realized I have several open-source projects I'm a maintainer or owner of which I've never really discussed on this site or on social media. Some of these projects have been pretty specific and tied to the other projects I was working on at the time, but several do have the potential for wider general use. I figured now was a good time to start talking about some of these projects and the problems they are intended to solve.

Python Catalogs

This is one of the more general use case projects I maintain. And while the examples and original impetus are for choices in Django, Catalogs could be used for just about any Python project that needs complex mappings. It's a data structure that's very simple to define and very flexible to use.

You can install Catalog yourself via Pypi, pip install pycatalog. As of the latest release it supports Python 2.7 and 3.3+. And you can always submit issues, feature requests and PRs over on Github.

The Choice Definition Problem

One of the niceties that Django brings to the table for both models and forms, introduced in the some of the earliest tutorials, is allowing you to define choices. It's a simple pattern that's not limited to Django, Python or web development, but is weaved through much of Django's out-of-the-box magic. It makes entry easier for users by providing their options to choose from and easier for your code to validate the input.

It starts out easy enough: provide the database value and label for each choice item.

class SomeModel(models.Model):
status = models.IntegerField(choices=((1, 'Open'), (2, 'Closed')), default=1)

Simple, but right off the bat this example exposes one of the first problems. Anytime you want to actually reference the value, default=1 in this case, you have to reference the value that's going into the database. Hard coding these values can make it difficult to refactor as your project grows and also make code checking for these values crypt and unclear. Take, for example, the simplest of checks if sm.status == 1.

A better, more common pattern is to define these values as constants.

class SomeModel(models.Model):
OPEN = 1
CLOSED = 2
STATUS_CHOICES = (
(OPEN, 'Open'),
(CLOSED, 'Closed'),
)
status = models.IntegerField(choices=STATUS_CHOICES, default=OPEN)

That's much cleaner and the code references are clearer too, if sm.status == SomeModel.OPEN.

In most cases this is going to be exactly what your project needs and you're done. Good work.

More Complex Cases

But what if data from this model syncs with a legacy remote API which has it's own archaic idea of how these values should be presented? Not too hard, we can add a mapping for that. Well, actually, two mappings because we need to communicate both directions. No problem.

class SomeModel(models.Model):
OPEN = 1
CLOSED = 2
STATUS_CHOICES = (
(OPEN, 'Open'),
(CLOSED, 'Closed'),
)
API_STATUS_MAP = {
'op': OPEN,
'cl': CLOSED,
}
REVERSE_API_STATUS_MAP = {v: k for k, v in API_STATUS_MAP.items()}
status = models.IntegerField(choices=STATUS_CHOICES, default=OPEN)

And sometimes we'll want to go directly from the API value to the user-readable label. Maybe use a property or method for that? That might obscure the fact that the value is based on the status field, sometimes that's good, but maybe not this time.

Also, we want to use a specific template path for a re-used block that changes based on the choices, add another mapping.

Oh, and maybe a different parser class to abstracts the different parsing details based on the choice, add another mapping.

This is where things are starting to go haywire. More mappings, more properties and methods, more mess.

Enter Python Catalog

This is the snowball I was trying to stop when I started working on Catalog a couple years ago. We had a complex project that required a complicated web of mappings and lookup patterns. So I set out to find a data structure that could handle this case with a couple of goals in mind:

Simple definition signature
Developers could define their own attributes for the choices, to fit their use case
Attributes could store values of arbitrary types
Simple process for mapping items down to an iterable, containing just the attribute values needed at that point
Easily lookup items by any attribute and access any attribute

Starting with the simple signature, Catalog works similar to Python 3's Enum.

class STATUSES(Catalog):
open = 1
closed = 2
STATUSES.open.value # =>1

But that doesn't even handle our simplest case of a database value and a label. We can do better. Let's define more attributes.

class SomeModel(models.Model):
class STATUSES(Catalog):
_attrs = value, label, api_value
open = 1, 'Open', 'op'
closed = 2, 'Closed', 'cl'

Oh, then there's the template paths and parser classes.

class SomeModel(models.Model):
class STATUSES(Catalog):
_attrs = value, label, api_value, template, parser
open = 1, 'Open', 'op', 'some_path/open.html', OpenParser
closed = 2, 'Closed', 'cl', 'other_path/closed.html', ClosedParser

Now let's add those choices to the field. There's an easy method for that.

 status = models.IntegerField(choices=STATUSES._zip('value', 'label'), default=STATUSES.open.value)

Now I want to get the user-readable label, but I've got the remote api value, or parser based database value.

SomeModel.STATUSES('op', 'api_value').label # => 'Opened'
SomeModel.STATUSES(1, 'value').parser # => OpenParser

Conclusion

So there you have it. It's by no means perfect, but after using it in production environments for over a year now, we still find it to be much cleaner and easier to use then some of the other patterns out there.

While the access API is always just as simple, the definitions do get clunkier as you add more attributes to the items. It was never meant to handle dozens of attributes per item and you have bigger problems if that's your use case, but since the attribute values for each item are really just presented as tuples, you can easily throw in parentheses and wrap across lines to help with readability.

Manage All the Languages Using Python Virtualenv

2017-10-28T01:17:32+00:00

A couple of years ago I was working with a couple other languages beside Python and began to get frustrated with their virtualenv equivalents. Not that they were doing anything wrong, just that they didn't work the way I was used to with virtualenvs. On top of that, they didn't work well together. I wanted to see if I could get something working that used the workflow I was used to and managed all the languages I was working with. So after a little poking around I found that many of the newer, "modern" languages had ways of running them from custom paths. I did a bit of work and before long I had a couple of these languages installed in virtualenvs and working side-by-side.

Then, a couple days ago I ran across this conversation on Twitter.

Most common Python issue I see with students is they've installed Python three or four different ways & have all their paths confused. 1/
— Jake VanderPlas (@jakevdp) October 24, 2017

I need to write up how I manage hundreds of projects without this being a problem. It took years to figure it out but it works. https://t.co/xM3fTs7e6u
— Jeff Triplett ✨ (@webology) October 25, 2017

A new development I'm really excited about is not just virtualenv, but also putting any non-Python executables in the virtualenvwrapper bin dir so it's associated with the project too.
— Rachel SKellyton 🔮 (@wholemilk) October 25, 2017

I have flirted with that for non-Python over the years. I liked it but lacked enough Node/Ruby knowledge to pull it off medium term.
— Jeff Triplett ✨ (@webology) October 25, 2017

THIS. It’s always intrigued me that there isn’t a “virtualenv for *all*” - one env, virtualising Py, Ruby, C… whatever.
— Russell Keith-Magee (@freakboy3742) October 25, 2017

And I realized that I had solved this problem for myself a while back and never really mentioned to anyone except a couple coworkers from time to time, and I certainly never shared it on this site or social media. So I'm going to take this opportunity to share how I manage several different languages using virtualenv.

One last note before we begin. I'm exclusively a Linux user and have been for many years. Most of these commands will work on Macs through the magic of Unix, but I have neither the knowledge or tools to troubleshoot Mac or Windows specific problems. Feel free to share solutions to problems you run into in the comments for others, but I will likely not be able to help too much for those OS's. If you're on Windows, please just use the Subsytem for Linux.

Python

With virtualenv being part of the Python tool chain, you're clearly going to need Python up and running. On Linux and Mac, your system probably already has it installed and ready to go. If your project works fine on this version or you're only managing other languages, this version should be fine. If you want to use a specific version, check out pyenv or downloading the source, but they each have docs that can walk you through those processes.

Installing virtualenv

I recommend, and assume for the rest of this post, using virtualenvwrapper and virtualenvs stored at ~/.virtualenv. If you have other preferences, everything here should still work, but you will need to replace my references to ~/.virtualenvs with the path to your environment.

Assuming you already have pip installed, with your Python version, you need to make sure that ~/.local/bin (or /home/<user>/.local/bin or /Users/<user>/.local/bin) is in your $PATH (echo $PATH). If not, add the following to your ~/.profile (or .zprofile, config.fish, etc.)

PATH=~/.local/bin:$PATH

If you needed to make this change, you should probably reboot your machine now to load that value into all your shell sessions.

This is going to allow you to use the --user flag for pip which installs the packages specified at a "global" level just for that user. This is useful for packages like virtualenv, virtualenvwrapper, ipdb, etc which can be used to manage Python and your environments. This should NOT be used for packages used within your project.

So with that set, install virtualenv:

pip install --user virtualenv virtualenvwrapper

Now you want need to edit your ~/.profile (or equivalent) and add the following and restart your shell.

export WORKON_HOME=$HOME/.virtualenvs
source $HOME/.local/bin/virtualenvwrapper.sh

Create an Virtual Environment

virtualenvwrapper make this easy. If you're using your OS's default Python, just use the following.

mkvirtualenv myenv

If you want to use a different version you've installed elsewhere, use the -p flag to specify it's path.

mkvirtualenv myenv -p /path/to/python

From now on, to activate this environment again just use: workon myenv.

Using virtualenvs

Since you're opting to use virtualenvs for managing other languages, I'm going to assume you're familiar with the day-to-day process of using them. If you're not familiar, now is a good time to check out the virtualenv and virtualenvwrapper docs and Google for more info.

NodeJs

Node is incredibly easy to setup in virtualenvs, thanks largely to the fact that there is a Python package to handle it for you.

pip install nodeenv

This package provides a tool that does everything for you. If you are using node in many projects, you might consider just installing nodeenv with the --user flag as discussed above.

Now install node using something like one of the following.

nodeenv -p # install into active virtualenv
nodeenv .env/ # install into a virtualenv at a specific path
nodeenv -p -n 8.8.1 # install a specific version of node

That's it. It'll download and install the appropriate version, then update the activate script for your environment to provide all the environment variables Node needs to know it.

Keep in mind that npm has the concept of "global" and "local" modules, in this case "local" will continue to be the current working directory where npm is run, and "global" will be in the lib directory of the virtualenv you specified.

Use the node and npm commands for manage Node in this environment further.

Ruby

Much like Node, some kind soul has provided a Python package for managing Ruby versions. This may also be a good candidate for installing with the --user flag if you're gonig to use it often.

pip install rubyenv

Now you simply install the Ruby version of choice.

rubyenv install 2.4.2

Since it's building from source you may need to install some system level packages for the build to succeed, but it should let you know.

Now use the ruby and gem commands to setup and run whatever you need.

Go (Golang)

Go is a little more work to set up, but not much. There's not a virtualenv aware tool, so you'll need to download the Go Tools binaries. Be sure to grab the appropriate tar.gz archive, not an installer, for your OS. Once you've got the archive, you just need to extract it into your virtualenv.

tar -C $VIRTUAL_ENV -xvf download/go1.9.2.linux-amd64.tar.gz --strip 1

The -C $VIRTUAL_ENV and --strip 1 are particularly important for the contents of the go directory inside the archive to be extracted into you virtualenv correctly.

Next, you need to tell Go where it's based. The best way to do this is have virtualenv set the GOROOT environment variable for you when you activate the virtualenv. If you only ever use Go inside a virtualenv, you can add the following to you user's ~/.virtualenv/postactivate, then you only need to do the extraction step above when you create a new virtualenv. If you only want to use virtualenv Go for this one virtualenv, add the following to ~/.virtualenv/<env name>/bin/postactivate.

GOROOT="$VIRTUAL_ENV"

Deactivate and reactivate your vritualenv and you're all set. Manage everything using the go command as usual.

Rust

Rust's primary install tool, rustup, has the ability to install into custom paths. You just need to add a couple environment variables to tell it how, and we'll let virtualenv activation handle that for us.

Like Go, if you only use Rust within virtualenvs, and want to save a step in the future, you can add the following to ~/.virtualenvs/postactivate. If you just want to set up this one virtualenv, add it to ~/.virtualenvs/<env name>/bin/postactivate.

CARGO_HOME=$VIRTUAL_ENV
RUSTUP_HOME=$VIRTUAL_ENV

Now's a good time to deactivate and reactivate your virtualenv to load those variables. And we're ready to run the rustup setup script.

curl https://sh.rustup.rs -sSf | sh -s -- -y --no-modify-path

Done. Now rustc, cargo and all the other Rust commands should be good to go.

Conclusion

That's it. Now you've got a pretty solid set of tools for developing different languages. They should all run side-by-side in any mixed and matched set you want to use and safely quarantined from the rest of your system.

I'm sure there are several other languages that can be managed this way, and if I run across more I'll try to update this post with details. Otherwise I hope this makes your development processes better.

Securing Your Website with Let's Encrypt

2016-07-23T23:04:30+00:00

This past week I gave a short presentation at the monthly TulsaWebDevs meeting about setting up a secure website using Let's Encrypt. I covered a brief (minimal) overview of how SSL/TLS works and a comparison in the processes of the traditional way to acquire an SSL certificate and acquiring a certificate via the ACME protocol. The slides are now up if you'd like to check them out over on my Presentations page. If you'd like to view my speakers notes simply press `s` while viewing the slideshow in Chrome.

Securing Your Website With Let's Encrypt

2016-07-21T05:32:32+00:00

Securing Your Website With Let's Encrypt

Securing Your Website

With Let's Encrypt

(In Under 5 Minutes)

Jeremy Satterfield

http://jsatt.com

https://github.com/jsatt

@jsatt

jsatt@jsatt.com

Primer on Protocols

HTTP

SSL/TLS

HTTPS

HTTP/2

TLS Handshake

Short: negotiate tls version, trade some random numbers, encrypt messages to verify, start talking application protocol
Negotiation Phase
- ClientHello - client sends supported TLS Version, random number, supported ciphers and compression methods
- ServerHello - server sends chosen TLS Version, random number, chosen ciphers and compression methods (based on client's)
- Certificate - server send public key certificate
- ServerHelloDone - server indicates handshake done
- ClientKeyExchange - client sends PreMasterSecret encrypted using server pub key
- Client and server use random numbers and PreMasterSecret to calculate common "master secret"
ChangeCipherSpec
- client sends "I only talk encrpted now"
- client send encrypted Finished
- server decrypts Finished, if decrypt fails handshake fails and connection torn down
- server sencs "I only talk encrypted now"
- client decrypts Finished, if decrypt fails handshake fails and connection torn down
Application Phase - handshake done, application protocol (HTTP) enabled, application messages encrypted using same methods as Finished
Shutdown - after all messages sent, connection torndown

Obtaining a Certificate

The old way

Prepare for Validation
Generate CSR
Order Certificate $$$
Have domain validated

Obtaining a Certificate

The old way

Prepare for Validation
Generate CSR
Order Certificate $$$
Have domain validated
Receive and Install Certificate

Let's Encrypt

Pros

Free

Automatic

Secure

Transparent

Open

Cooperative

Cons

Short Validity Time

No Wildcard Certificates

No Extended Validation or
Organization Validation

ACME

Obtaining a Certificate

In the modern world

Certbot


wget https://dl.eff.org/certbot-auto
chmod a+x certbot-auto
./certbot-auto certonly

Installing Your Certificate

https://mozilla.github.io/server-side-tls/ssl-config-generator/

nginx
Modern
nignx 1.4.6
openssl 1.0.1f

Wikipedia - Transport Layer Security

Let's Encrypt

ACME - Wikipedia GitHub

http://rebecca.meritz.com/ggm15/

Unit Testing Recursion in Python

2014-12-11T17:05:39+00:00

Today I finally figured out the solution to a problem I've been trying to solve for a while. It's kind of hacky and maybe a bad idea, but now I know it's possible. The problem has always been that I'd like to test that a function recurses, but not needing it to actually have the recursion execute within to test. Just a unit test to assert that recursion is happening. After a little thought about how Python stores references I came up with this.

class ExampleTest(TestCase):
def setUp(self):
self.task = Task()
# ...
# other test stuff
# ...
def test_recursion(self):
original_func = self.task.run
self.mock.StubOutWithMock(Task, 'run')
Task.run(id=44)
self.mock.ReplayAll()
original_func(id=22)
self.mock.VerifyAll()

Because variables in Python are just references to objects in memory, if you create a new reference before stubbing out the primary reference, pointing it to a mock object, you can still access it via this new reference. You're now free to check that recusion is happening, without the need for it to actually recurse in the test.

Keeping MRO In Mind When Mocking Inherited Methods

2014-10-16T19:33:11+00:00

So I just spent a couple of hours banging my head on a ridiculous PyMox mocking issue and though I'd share. Here is an example of the existing code.

# views.py
class MyBaseView(BaseView):
def get_stuff(self):
# this is doing things
class SpecificView(MyBaseView):
example = True
def send_stuff(self):
stuff = self.get_stuff()
# do other things
# tests.py
class SpecificViewTest(TestCase):
def setUp(self):
self.view = SpecificView()
self.mox = mox.Mox()
def tearDown(self):
self.mox.UnsetStubs()
def test_send_stuff(self):
self.mox.StubOutWithMock(MyBaseView, 'get_stuff')
MyBaseView.get_stuff().AndReturn(['list', 'of', 'things'])
self.mox.ReplayAll()
self.view.send_stuff()
self.mox.VerifyAll()
# do assertions

This was all fine and working for months, until I added the following.

# views.py
class ExtraSpecificView(SpecificView):
def other_stuff(self):
self.get_stuff()
# do more things again
# test.py
class ExtraSpecificViewTest(TestCase):
def setUp(self):
self.view = ExtraSpecificView()
self.mox = mox.Mox()
def tearDown(self):
self.mox.UnsetStubs()
def test_other_stuff(self):
self.mox.StubOutWithMox(SpecifcView, 'get_stuff')
SpecificView.get_stuff().AndReturn(['more', 'things'])
self.mox.ReplayAll()
self.view.other_stuff()
self.mox.VerifyAll()

So what started happening at this point is SpecificViewTest.test_send_stuff began failing with this.

ExpectedMethodCallsError: Verify: Expected methods never called:
0. get_stuff.__call__() -> ['list', 'of', 'things']

It would only fail when ExtraSpecificViewTest ran before SpecificViewTest which, with nose, it does by default. So I knew it had to have something to do with the mocking in ExtraSpecificViewTest.test_other_stuff. After more playing and banging my head I came up with this working theory which seems correct the more I think on it.

When you mock out a method on a class, the mocking library stores the original method and replaces it with the mock function. When you tell the library to stop mocking the method, mox.UnsetStubs in this case, it grabs that original method it stored before and assigns it back to the appropriate attribute on the class, restoring it's original functionality.

However, in my class I was mocking a method that SpecificView inherited from MyBaseView. When the library grabs the original method for storing Python sees that SpecificView doesn't have a get_stuff method of it's own, so it moves up the MRO and grabs MyBaseView.get_stuff and the library stores that and assigns a mock function to the get_stuff attribute on the class of SpecificView. Then, when the library goes to un-stub the method, it grabs the stored get_stuff Python gave it from MyBaseView and assigns it also to the get_stuff attribute of the class of SpecificView. Finally, when SpecificViewTest.test_set_stuff runs, send_stuff calls SpecificView's get_stuff, instead of moving up the MRO as to used to, Python now sees that the SpecificView class has it's own get_stuff method (which doesn't have a super call), so it doesn't move up the MRO, therefore MyBaseView is never called as expected and raising a failure when verified.

So this became the fix.

#tests.py
class ExtraSpecificViewTest(TestCase):
...
def test_other_stuff(self):
self.mox.StubOutWithMox(MyBaseView, 'get_stuff')
MyBaseView.get_stuff().AndReturn(['more', 'things'])

It's a convoluted bug that may or may not be specific to PyMox or other mocking libraries, but I'm not sure how they should be properly handling something like this. So I guess the moral of the story is to mock methods from the class which the were originally defined, not just the class you inherited from.

Mocking a property in Python

2014-08-08T21:10:45+00:00

Anytime I see someone turning an instance method into a property on a Python object, I always have to step back and rethink whether it's really the right thing to do. While properties certain have valid use cases, I often see them overused and misused. This can result in code that is harder to refactor should you decide you actually do want to accept arguments as well as less straight forward to separate in unit tests.

The way I found to deal with the latter case is pretty simple but unituitive at first. Simply stub out the property on the class, then assign the property value on the instance.

# Use __class__ if you don't have the class imported
self.mox.StubOutWithMock(self.book.__class__, 'reviews')
self.book.editors = ['editor 1', 'editor 2']
# replace with a MockAnything to go deeper
self.mock.StubOutWithMock(Company, 'employees')
Company.employees = self.mock.CreateMockAnything()
company.employees.are_executives().AndReturn(['CEO', 'COO', 'CIO'])
campaign3.reviews.are_developers().AndReturn(['Jim', 'John', 'Joey'])

Class-based Celery Tasks

2014-06-05T23:25:06+00:00

Update 2017-11-02: Celery 4 now adivises against inheriting from Task unless you are extending common functionality. However you can still get similar functionality by creating a new class and calling is from inside a decorated function task.

class MyTask(object):
def run(self, source):
do_stuff()
...
@app.task
def my_task(source):
MyTask().run(source)

Original Post:

Recently, I've had to write several complicated Celery tasks. Unfortunately, when doing a complicated process standard task functions can become unwieldy to write, read and unit test. After looking into how Celery tasks actually work, I was able to find a more manageable way of writing these complex tasks.

It turns out the task decorator that you're used to using is just an object factory for Task objects that turns the decorated function into the run method of on the Task instance it creates. Skipping the decorator and extending the Task class directly makes things a little more flexible.

from celery import Task
class MyTask(Task):
ignore_result = True
def run(self, source, *args, **kwargs):
self.source = source
data = self.collect_data()
self.generate_file(data)
def generate_file(self, data):
# do your file generation here
...
def collect_data(self):
# do your data collection here
...
return data

In this case run is the equivalent of the function task you're used to, but thanks to OOP you're free to break some of the more complex code into logic blocks, collect_data and generate_file, and access to instance attribute, source.

To call the task your just need to instantiate the it and call the desired method to trigger it.

task = MyTask()
# Run on Celery worker now
task.delay(123)
# Run on Celery worker at a sepcfic time
task.apply_async(args=[123], eta=datetime(2015, 6, 5, 12, 30, 22))
# Run task directly, No celery worker
task.run(123)
task(123)

Testing also now becomes easier as well since you can test each unit on it's own.

For most cases, your standard function-based classes are probably going to do the job. But for those extra complex cases, class-based might make things easier work with.

Mocking Python's built-in open function

2014-04-11T17:25:38+00:00

A while back I was working on my podcast client and ran into an issue downloading large files. The root problem was that all of Django's FileField backends store the file (or file-like object) to memory then saves it to disk, and the cubieboard system I was using had limitied memry resources, resulting in "out of memory" errors. After much searching and hacking I finally settled on just storing the file to disk myself using requests streaming argument. This allowed me to download the file in chunks and save directly to disk and then tell the Django field where I placed it, as you can see here.

Then I went to update tests. This is when a new problem presented itself. I wanted to test that proper calls were being made to store the file to disk, but I didn't want to store a file to disk during tests. Mocking never worked the way you'd expect for the built-in open. I placed it on the back burner for a while until I ran into a similar issue an another project. When I finally found the solution, it was incredibly simple and incredibly nonobvious.

Pythons has a bunch of built-in functions that are just preloaded, such as open for opening files, print for outputting to the console, str, bool, dict, list, etc. for casting or creating an object, among others. Most of these just do what they do, with no affect on the overall environment; therefore, when testing I just let them do what they do and mocking them is probably a bad idea anyway. Except for open, it has real world affects on the environment in that it accesses, and allows writing of, files on the disk.

Finally, I stumbled across the __builtin__ module. This is the magic module where all those functions actually reside, and if you can access where a function resides you can mock it out. The following should work on most Python mocking frameworks, but this is how to use PyMox to do it.

class Client(APIClient):
...
def open_file(self):
file = open(self.path, 'r')
contents = file.read()
file.close()
return self.prepare_file(contents)
...
# test.py
import __builtin__
from StringIO import StringIO
class ClientTest(TestCase):
...
def test_open_file(self):
self.mock.StubOutWithMock(__builtin__, 'open')
mock_content = StringIO('test')
self.mock.StubOutWithMock(mock_content, 'close')
open('/tmp/mypath.txt', 'r').AndReturn(mock_content)
mock_content.close()
self.mock.ReplayAll()
file = self.client.open_file()
self.mock.VerifyAll()
...

In this case, returning an instance of StringIO results in a file-like object allowing me to set the contents of the "file" being accessed directly in the test. No fixtures or creating files on the disk. This pattern works for both reads and writes.

But the code I'm testing here doesn't do all the error handling to make sure the file is closed even if there's an error during the read. Frankly, they made it possible to use open as a context manager so that you don't have to worry about it. So combining this new knowledge with my previous post about mocking context managers, we can easily write cleaner code that's still testable.

class Client(APIClient):
...
def write_file(self):
with open(self.path, 'wb') as f:
f.write(self.content)
...
# test.py
class ClientTest(TestCase):
...
def test_write_file(self):
self.mock.StubOutWithMock(__builtin__, 'open')
mock_file1 = self.mock.CreateMockAnything()
open('/tmp1/mypath.txt', 'wb').AndReturn(mock_file1)
mock_file1.__enter__().AndReturn(mock_file1)
mock_file1.write('this is my content')
mock_file1.__exit__(None, None, None)
self.mock.ReplayAll()
self.client.write_file()
self.mock.VerifyAll()
...

Since StringIO can't be used as a context manager, this pattern doesn't allow for the use of StringIO. This means you have to mock out the reads, writes and any other methods you would call on a file-like object, but I think that's a small price for being able to write cleaner, testable code.

Abusing Django Rest Framework Part 4: Object-level field exclusion

2014-03-27T16:55:11+00:00

Similar to the object-level readonly field from my previous post, there are some cases where we want to exclude certain fields based on what object the user is trying to access. You could overwrite the views get_serializer method to use a different serializer based on their access, but if nesting serializers is a possiblility this get messy, somewhere in the neighborhood of O(n²). Another option is to modify a serializers to_native method.

class SampleSerializer(serializers.ModelSerializer):
...
def to_native(self, obj):
restricted = not obj.review_set.filter(
client__plan__account_details=True).exists()
if restricted:
self.fields.pop('billing_address', None)
return super(SampleSerializer, self).to_native(obj)

The to_native method is responsible for turning the Python object into a simple, serializable value. This means it has to be called for every object that it being retrieved, so it will always be able to affect your output, even when nested.

One drawback is that it only works in conjunction with retrieve views. If the field is defined in the serializer at any point as writeable, the users can still update that field, even if they can't see it. This can be a pretty big security risk when used with update views.

Another Approach

Having implemented this method several months ago, based on a spec that was several months old at the time, I would likely approach the root issue from a different angle if it came up again. Maybe creating updating the spec to have a consistant api response and return blank values for the unauthorized fields would be a better approach. SerializerMthodField has the potential to make this approach much cleaner.

Abusing Django Rest Framework Part 3: Object-level read-only fields

2014-03-26T16:25:04+00:00

DRF has tools to control access in a few ways. Serializers make it easy to select what fields can be accessed and whether or not they are read-only. Permissions are great for restricting access to objects at all or even making certain objects read-only. But there are also cases where you might only want to allow access to a field on a specific object but leave that field restricted on other objects, or vice-versa.

In our case, we had an endpoint for modifying users, which has a field to dictate whether the user is an account admin. Our view already uses permissions to verify that the request user is allow to modify these items at all, but we wanted to make sure the user couldn't accidentally remove their own admin access but still be able to change other fields on this endpoint.

Whenever a Serializer is instantiated, the get_fields method is called to look at the attributes and decide what fields to include and instantiate them. As part of the serialzer instantiation the view also passes the request, view and format_kwarg as context to the serializer which is attached to the instance. This means we can use the request or view, or objects attached to them in to change attributes of the fields, in our case making one of them readonly.

class SampleSerializer(serializers.ModelSerializer):
...
def get_fields(self, *args, **kwargs):
fields = super(SampleSerializer, self).get_fields(*args, **kwargs)
request = self.context.get('request', None)
view = self.context.get('view', None)
if (request and view and getattr(view, 'object', None) and
request.user == view.object.user):
fields['is_admin'].read_only = True
return fields

The one issue with this method is that you can not nest this serializer and have this change work. Whenever a Serializer is nested on to another, the parent gets one instance of this child at the time that the server process starts, which gets reused for all future instances of said parent. Due to lots of complicated handling on DRF's part, that I'll leave to you to read up on, everything just works the way you expect most of the time. However, we are trying to access the request and view objects that clearly don't exist at the time the server process starts.

Using Tox with Travis CI to Test Django Apps

2014-03-09T01:18:20+00:00

Being a fan of good testing, I'm always trying to find ways to improve testing on various projects. Travis CI and Coveralls are really nice ways to set up continuous integration for your open-source projects. A couple months ago I finally started hearing grumblings about tox and how everyone was it using for their Python test automation. Every time I'd try to wrap my head around it, something always eluded me, so this week I finally decided to dive in head first and see if I could get to the bottom of it and how it could improve my current integration setup.

Headless Django Environments

Most of my open-source Python projects have been developed from the beginning to be installable modules for use with tools like pip. With Django apps, this tends to be a little more difficult as Django development environments generally assumes you have a project that contains a lot boilerplate (manage.py, project/settings.py, project/urls.py), but some of this stuff is just clutter in an installable package. After trying several patterns, I found the one used by James Socol on several projects, using fabric to wrap my development enviroment and easily provide some of the common commands we use manage.py for, but without all the clutter of a full Django project. I've also created a repository that's intended to be used as a Django project template, to help with setting out a project like this. It even includes some of the setup I'm about to discuss. You can get that template over at Github.

Preparing tox

Start by installing tox with a simple pip install tox. Then create the tox.ini file in the base of your project. The first section you care about is configuring tox itself.


[tox]
envlist = django15, django16

The envlist directive tells tox which environments to run by default when calling it from the CLI. Tox comes with a few built-in environments (py26, py27, py31, py33, etc.) for testing different Python versions, but in my case I care about testing different Django versions, so I'll be building my own environments.

[testenv]
commands = django-admin.py test
setenv =
DJANGO_SETTINGS_MODULE=test_app.settings
PYTHONPATH={toxinidir}

testenv is the set of default environment settings that all named environments will inherit from. Here I'm telling it that the default command I'm going to use is django-admin.py test to run tests. And setup environment variables that are usually handled by manage.py. Tox also allows for substituting variables, in this case {toxinidir} is simply the path of the directory which contains the tox.ini file. It also allows for doing substitutions based on lookups within the ini file. By defining a custom section, we can define settings that can be reused and extended within our environments.

[base]
deps =
mox
nose
django-nose

Here I'm defining some dependencies that I'm going to add to in each environment. In my case, I prefer Nose and Mox to their standard library counterparts.

[testenv:django15]
deps =
django>=1.5, <1.6
{[base]deps}
[testenv:django16]
deps =
django>=1.6, <1.7
{[base]deps}

This creates two new test environments, named django15 and django16, and adds the dependency to install the proper version of Django, 1.5 and 1.6 respectively. It also uses the section lookup to append the deps from the base section we defined above. Now, when you return to the CLI and run tox, you should see it setup virtualenvs for each environment and run your tests under them. That's all it take to get it running. Anyone can now clone your repository and simply run tox to confirm that tests are passing. This is particularly useful if they intend to contribute back to the project.

Adding Coverage

Now, as I mentioned before, I want to also use Coveralls to show coverage stats as well. To do that I just need to define another environment.

[testenv:coverage]
commands =
coverage run --branch --omit={envdir}/*,test_app/*.py,*/migrations/*.py {envbindir}/django-admin.py test
coveralls
deps =
coverage
coveralls
{[testenv:django16]deps}

This time I'm running coverage run with branch coverage, ignoring migrations and my test app and running the django-admin.py in the enviroments bin directory, followed by the coveralls reporting script. We also have a couple of additional dependencies to add to the django16 deps.

Running Tox on Travis

With tox configured to handle the details, Travis just needs to know about the different environments. Here's what the .travis.yaml looks like.

language: python
install:
- pip install tox
script:
- tox
env:
- TOXENV=django15
- TOXENV=django16
- TOXENV=coverage

This tells Travis to install tox and to run the tox command. Adding the TOXENV environment variable makes it easy to specify which testenv to run.Travis runs it's tasks based on the build matrix of environment variables and language versions, allowing us to run each testenv asynchronously and reporting the results for each version separately.

Mocking Context Managers in Python

2014-02-28T22:33:13+00:00

I've often found Python's context managers to be pretty useful. They make a nice interface that can handle starting and ending of temporary things for you, like opening and closing a file.

For example:

f = open('myfile.txt', 'w')
try:
for row in records:
f.write(row)
finally:
f.close()

can be replaced with

with open('myfile.txt', 'w') as f:
for row in records:
f.write(row)

This last week I was working with the ZipFile module and wanted to use it's context manger interface, but I ran into a little confusion when it came to unit testing. After a little better understanding of how context managers work, I figured out that the __enter__ and __exit__ methods are what really makes a context handler. As explained at PyMOTW, when you invoke with on a class, __enter__ is called and should return an object to be used in the context (f in the above example), the code within the block is executed, and __exit__ is called no matter the outcome of the block. So both of these would be roughly equivalent, assuming do_stuff doesn't raise an exception.

with Context(foo) as bar:
do_stuff(bar)
c = Context(foo)
bar = c.__enter__()
try:
do_stuff(bar)
finally:
c.__exit__(None, None, None)

With this understanding, here is the solution to my mocking problem using PyMox.

#module/tasks.py
def zip_it_up(filename):
with ZipFile(filename, 'w') as f:
for file in FILES:
f.write(file.path, file.name)
# tests.py
... # setup and stuff is up here somewhere
def test_building_zipfile(self):
self.mock.StubOutWithMock(module.tasks, 'ZipFile')
    mock_zip = self.mock.CreateMockAnything()
    module.tasks.ZipFile('/tmp/export.zip', 'w').AndReturn(mock_zip)
    mock_zip.__enter__().AndReturn(mock_zip)
    mock_zip.write('/tmp/export-1.xml', 'export-1.xml')
    mock_zip.write('/tmp/export-2.xml', 'export-2.xml')
    mock_zip.__exit__(None, None, None)
self.mock.ReplayAll()
zip_it_up()
self.mock.VerifyAll()

Mocking out ZipFile allows us to return a mock object from it's instantiation. We can then set the expectation that __enter__ will be called on the instance, returning the instance itself, expecting write to be called twice on the instance and finally __exit__ to be called. The three arguments of None here are to indicate that an exception isn't expected. If the code inside the context block were to raise an exception, these arguments would be the type, value and traceback as returned by raise. In the event you are testing for an exception, these arguments should be set accordingly when setting expectations.

Abusing Django Rest Framework Part 2: Non-rate-based Throttling

2014-02-19T16:43:17+00:00

Anyone running an API that can be reached by the outside world should most definitely be concerned that someone might pummel their server by making a massive amount of requests to that one endpoint that requires a bunch of on-the-fly calculations. Enter Django Rest Framework's throttling. It allows you to easily configure the framework to stop allowing requests from a user once they've made so many requests in a period of time. Whether you're concerned about requests over a sustained period of time or in short bursts, rate limiting with throttles will handle it.

However, there are times you might need to limit based on something less consistent than number of calls per minute or day. One example we we ran into was clients submitting requests for a process that requires a lot of human work to complete, and the possibility that a large number of these requests might mean that the client doesn't actually understand the real purpose of the feature, or an indicator of a bigger problem for that client. If you can limit these clients to a certain number of outstanding requests at any given time, this allows for the time for our client contacts to process the appropriate requests or contact them if the requests are inappropriate.

While the throttles provided by Rest Framework are all about rate limiting, writing a custom throttle to do this is pretty simple.

### throttles.py
from django.db.models import Count
from rest_framework.throttling import BaseThrottle
from rest_framework.exceptions import Throttled
from .models import Inspection
class InspectionThrottle(BaseThrottle):
def allow_request(self, request, view):
inspections = Inspection.object.filter(client=view.client)
if inspections < 15:
return True
raise Throttled(detail=(
"You have reached the limit of 15 open requests. "
"Please wait until your existing requests have been "
"evaluated before submitting additional disputes. "))
###views.py
class InspectionCreateView(CreateAPIView):
model = Inspection
throttle_classes = InspectionThrottle

By simply defining an allow_request method you are free to use whatever logic you want to throttle requests, so long as you return True to allow the request and raise the Throttled exception to deny the request. Throttled also accepts a kwarg of wait which will set the X-Throttle-Wait-Second header for notifying the client how long to wait before trying again. This is mostly just useful for rate limiting, but maybe you'll be clever and have a reason to use it.

About

2014-02-10T22:36:53+00:00

I'm a web developer and homebrewer from Tulsa, OK with an affinity toward open-source. I work primarily in Python, Django, Javascript and and related languages such as CoffeeScript, CSS and Stylus. As a member of TulsaWebDevs, CodeforTulsa, and Civic Ninjas, I also work on many civic-minded projects.

Abusing Django Rest Framework Part 1: Non-Model Endpoints

2014-01-29T18:32:43+00:00

When working with Django Rest Framwork a few months back, there were a few road blocks that we ran into. Rest Framwork is awesome with most models for providing a simple CRUD API, in any (or multiple) serializations, with authentication and permissions. Sometimes, however, things aren't so simple. Things get ugly. Framworks get abused.

This is the first of a few posts looking at some of the ways Rest Framwork can be beaten into submission. Some of them may be the way the framework intended them to be implement, but non-obvious, while others were never intended, are probably bad ideas, and should be avoided at all costs (except when the needs of the business require it, of course).

Creating Endpoints to Trigger Non-model Processes

Let's start out easy. As I mentioned Rest Framework is great at working with models, but it can also be easily used for things other than retrieving and saving models. Say for instance you have an email form on your site for sharing a post. Using AJAX posting of those fields makes for a better experience for your user and keeps them browsing the page longer.

While it's not exactly clear, the APIView allows for this. Most of the documentation for APIView is spent doing model retrievals that are made easier when you start using GenericAPIView and those that inherit from it. However, using the dispatch methods (post, get, delete, patch, put, etc.), just like a Django class-based view, our example above becomes pretty easy.

from rest_framework import views
from rest_framework.response import Response
class ShareView(views.APIView):
permission_classes = []
def post(self, request, *args, **kwargs):
email = request.DATA.get('email', None)
url = request.DATA.get('url', None)
if email and url:
share_url(email, url)
return Response({"success": True})
else:
return Response({"success": False})

In these dispatch methods you can easily access the deserialized posted data (via request.DATA), run your custom code (share_url() in this case) and return a serialized response (passing your data to Response). If you're arbitrary code execution applies to a model object, maybe a method call, but doesn't including retrieving or updating that model, simply inherit from the GenericAPIView instead the APIView and use the get_object method of the view.

The one caveat when using APIView is to watch what permissions are applied to the view. Since it is not associated with a model, any permission that expects a model object to be available, like the DjangoModelPermission that is applied by default, will fail, so you should be explicit about permissions.

Finding Un-mocked HTTP requests in Python tests with Nose

2014-01-16T18:40:58+00:00

I'm a big fan of proper unit testing with mocking. So I'm pretty disappointed when we run across a unit that is not only not mocked properly, but results in real-world consequences outside the testing environment. One case we've run into couple of times now is tests that are making actual outbound HTTP requests to remote servers. Since clients don't necessarily like getting fake information posted to their servers every time you run tests, I figured this was a good time to get to the bottom of this.

I caught this particular case while researching another broken test and noticed that one particular test was hanging for several second before passing every time. After digging in, I found that the view being tested called a method, which called a method, which executed a task that included a requests.post() call. It was quick to mock out the first method call that spawned it all, but I was worried about next time we test against that view and forget the mock again.

After a little research I found that Nose provides the ability to run package and module level setup and teardown functions. Simply add setUpPackage, setUpModule, tearDownPackage and tearDownModule functions to the __init__.py of the package or module appropriately and you're free to do any setup and cleanup you need for the package or module as a whole. This can include setting up fixtures, test databases, or, as I discovered, mocking. Placing the following code in the __init__.py of my project, quickly identified another instance where we were making outbound requests that we weren't catching.

pkg_mox = None
def setUpPackage():
global pkg_mox
import urllib
import urllib2
import mox
import requests
pkg_mox = mox.Mox()
pkg_mox.StubOutWithMock(requests.api.sessions.Session, 'send')
pkg_mox.StubOutWithMock(urllib.URLopener, 'open')
pkg_mox.StubOutWithMock(urllib2.OpenerDirector, 'open')
pkg_mox.ReplayAll()
def tearDownPackage():
pkg_mox.VerifyAll()
pkg_mox.UnsetStubs()

Cases where requests were being made, but not properly mocked, raised the "Unexpected method called" exception you'd expect, including the traceback to the offending test.

Through the years, we've gone through a few iterations on how we like making our http calls. While we've settled on the requests library for now you're always going to be dealing with legacy code. This flow catches calls to all three major request libraries, just in case. Mocking the underlying methods of the libraries catches all of the common paths for executing requests for each, while still allowing those common cases to be properly mocked in your test code.

I'm certain there are many other libraries that this could be helpful for. Now let's get to writing better unit tests.

Decorators vs Mixins for Django Class-Based Views

2014-01-13T20:21:58+00:00

I've been huge fan of Django's class-based views (CBVs) since I first tried them out in Django 1.4. While they're much more complicated then the classic function based views, once you understand how they work, they're much more powerful, flexible and allow for DRYer code. I highly recommend anyone who hasn't delved into CBVs take a look at Class Class-based views or GoDjango.

However, one of the early issues I ran into was for views that required Django user permissions. With function based views, Django's auth application provides decorators to check that users are logged in, have a specific permission, or pass other custom checks the developer can provide. You simply add the decorator to the view...

@permission_required('auth.change_user')
def user_list(request):
...

And now any user that doesn't have the required permission is redirected to the login page.

For CBVs though, it's not quite that simple, it's the dispatch method that begins the process that you're used to picking up from with function based views. This means that to apply a decorator to the CBV, you have a couple of options.

You can override the dispatch method to apply the decorator:

class UserListView(ListView):
model = User
@my_custom_decorator
def dispatch(self, request, *args, **kwargs):
return super(UserListView, self).dispatch(request, *args, **kwargs)

You could write a reusable class view decorator:

def class_view_decorator(function_decorator):
def deco(View):
View.dispatch = method_decorator(function_decorator)(View.dispatch)
return View
return deco
@class_view_decorator(my_custom_decorator)
class UserListView(ListView):
model = User

This second option is the one that I've lived with for a long time. It's relatively clean and DRY and not much different than what you're used to seeing with function views.

But recently I was pointed toward Django Braces. It's a library of commonly (and not-so-commonly) needed view decorator-like functionality in the form of mixins. I immediately realized that when thinking about the functionality these decorators provide, I was stuck in the old way of doing things rather than looks at these views from an OOP perspective.

Writing these decorators as mixins instead, makes them more flexible, extensible and DRYer, just like the move from function views to CBVs. You're able to build a base mixin that others can inherit from and extend. What would have been arguments to the decorator can are much cleaner as class attributes for the view that's extending it. You can even have views that apply affect the mixin on a one-off basis. Take for instance the <code>UserCheckMixin</code> below.

class UserCheckMixin(object):
user_check_failure_path = '' # can be path, url name or reverse_lazy
def check_user(self, user):
return True
def user_check_failed(self, request, *args, **kwargs):
return redirect(self.user_check_failure_path)
def dispatch(self, request, *args, **kwargs):
if not self.check_user(request.user):
return self.user_check_failed(request, *args, **kwargs)
return super(UserCheckMixin, self).dispatch(request, *args, **kwargs)

I now simply add the mixin to a view:

class UserListView(UserCheckMixin, ListView):
model = User
user_check_failure_path = 'auth_login'
def check_user(self, user)
# do some check against the user here and return True or False

You can extend it to build a more useful mixin:

class PermissionRequiredMixin(UserCheckMixin):
user_check_failure_path = 'auth_login'
permission_required = None
def check_user(self, user):
return user.has_perm(self.permission_required)
class UserListView(PermissionRequredMixin, ListView):
model = User
permission_required = 'auth.change_user'
user_check_failure_path = 'auth_login'

As you can see, once you've defined the PermissionRequiredMixin, it's much cleaner, more object-oriented and I would argue more pythonic.

If you're relying on built-in decorators, the class_view_decorator is a pretty helpful tool to have in your codebase. But for anything custom you may be writing, it really is hard to beat a mixin.

Fresh start

2014-01-11T22:35:43+00:00

This is going to be a fresh attempt to start blogging again on a regular basis. I plan to share development and brewing experiences, open-sourcing in both cases whenever possible.

Jeremy Satterfield

Resume

Skills

Experience

Peloton

Peerfit

Summit ESP (acquired by Halliburton in 2017)

ConsumerAffairs.com

OpenTundra

AT&T Mobility

Community Involvement

TulsaWebDevs (Volunteer, 2010 - 2020)

Code for Tulsa (Delivery Lead, 2014 - 2019)

Civic Ninjas (Developer, 2013 - 2014)

Education

Links

The Pains of MySQL Perfomance Under Docker

Background

The Issue with Mac

The Issue with Linux

Python Catalog Data Structures

Python Catalogs

The Choice Definition Problem

More Complex Cases

Enter Python Catalog

Conclusion

Manage All the Languages Using Python Virtualenv

Python

Installing virtualenv

Create an Virtual Environment

Using virtualenvs

NodeJs

Ruby

Go (Golang)

Rust

Conclusion

Securing Your Website with Let's Encrypt

Securing Your Website With Let's Encrypt

Securing Your Website

With Let's Encrypt

(In Under 5 Minutes)

Jeremy Satterfield

Primer on Protocols

HTTP

SSL/TLS

HTTPS

HTTP/2

TLS Handshake

Obtaining a Certificate

The old way

Obtaining a Certificate

The old way

Let's Encrypt

Pros

Cons

ACME

Obtaining a Certificate

In the modern world

Certbot

Installing Your Certificate

More

Unit Testing Recursion in Python

Keeping MRO In Mind When Mocking Inherited Methods

Mocking a property in Python

Class-based Celery Tasks

Mocking Python's built-in open function

Abusing Django Rest Framework Part 4: Object-level field exclusion

Another Approach

Abusing Django Rest Framework Part 3: Object-level read-only fields

Using Tox with Travis CI to Test Django Apps

Headless Django Environments

Preparing tox

Adding Coverage

Running Tox on Travis

Mocking Context Managers in Python

Abusing Django Rest Framework Part 2: Non-rate-based Throttling

About

Abusing Django Rest Framework Part 1: Non-Model Endpoints

Creating Endpoints to Trigger Non-model Processes

Finding Un-mocked HTTP requests in Python tests with Nose