Context
This is a continuation of Redis and Its New Features article. Generally, in scalable backend applications, we would set up a DB and use redis as cache. This requires additional and separate processes to load data into cache, and introduces a complex system with multiple components. Redis has evolved from a cache into a primary database. In this article, I will provide a tutorial on how to write a simple scalable video-sharing backend application which leverages python, flask, and RedisJson.
Components
Redis
For this setup, docker and docker-compose are required. Once they have been installed in the system, include below docker-compose.yaml in the project’s root directory.
1
2
3
4
5
6
7
8
9
10
11
12
|
version: "3.9"
services:
redis:
container_name: redis
image: "redis/redis-stack:latest"
ports:
- 6379:6379
- 8001:8001
deploy:
replicas: 1
restart_policy:
condition: on-failure
|
Flask App
App Structure
This is simple minimal app. The structure looks like below.
1
2
3
4
5
6
7
8
|
~/app_directory
├── app.py
├── data
├── dataloader.py
├── docker-compose.yml
├── models.py
├── requirements.txt
|
Data
First, we build below minimal data models and associations. Note that they are of json type. Below is what they look like.
1
2
3
4
5
6
7
8
9
10
|
{
"title": "Video1",
"description": "Test Video 1",
"uri": "www.google.com",
"likes": 0,
"likedBy": [{
"name": "Robert S",
"email": "zz@gmail.com"
}]
}
|
1
2
3
4
|
{
"user": "John Doe",
"email": "john.d@example.com"
}
|
Then, we can define above representations in models.py.
Basically, we just EmbeddedJsonModel and JsonModel from redis_om package as base classes, and define our data models.
Note that the “Field” object can be used to switch on indexing. It looks like a wrapper class on a data object’s attributes.
Here is the code.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
|
from redis_om import (EmbeddedJsonModel, Field, JsonModel)
from pydantic import NonNegativeInt
from typing import Optional, List
# We keep the models simple here
# Index the fields for easy searching
class User(EmbeddedJsonModel):
name: str = Field(index=True)
email: str = Field(index=True)
# A video can have many likes and users
class Video(JsonModel):
# Indexed for exact text matching
title: str = Field(index=True)
description: str = Field(index=True)
uri: str = Field(index=True)
likes: NonNegativeInt = Field()
likedBy: List[User]
|
API End points
Basically, at high leve, we use Flask’s request object to extract json data that comes in from http request and pass it along to respectives RedisJson Data objects.
We also use redis_om Migrator to index existing data. This is invoked at application startup.
Here is the implementation. You can include below block in app.py.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
|
from flask import Flask, request
from pydantic import ValidationError
from models import Video, User
from redis_om import Migrator
app = Flask(__name__)
# CRUD methods
# Create a new video.
@app.route("/video/new", methods=["POST"])
def create_video():
try:
print(request.json)
new_video = Video(**request.json)
new_video.save()
return new_video.pk
except ValidationError as e:
print(e)
return "Bad request.", 400
# Update like's of a video
# We keep the minimum like's at 0
@app.route("/video/<id>/likes/<int:new_likes>", methods=["POST"])
def update_likes(id, new_likes):
try:
video = Video.get(id)
except NotFoundError:
return "Bad request", 400
new_likes = video.likes + new_likes
video.likes = new_likes if new_likes > 0 else 0
video.save()
return "ok"
# Update uri of a video
@app.route("/video/<id>/uri/<int:new_uri>", methods=["POST"])
def update_uri(id, new_uri):
try:
video = Video.get(id)
except NotFoundError:
return "Bad request", 400
video.uri = new_uri
video.save()
return "ok"
# Delete a video by ID.
@app.route("/video/<id>/delete", methods=["POST"])
def delete_video(id):
# Delete returns 1 if the video existed and was
# deleted, or 0 if they didn't exist. For our
# purposes, both outcomes can be considered a success.
video.delete(id)
return "ok"
# Find a video by ID.
@app.route("/video/byid/<id>", methods=["GET"])
def find_by_id(id):
try:
video = video.get(id)
return video.dict()
except NotFoundError:
return {}
|
We also define user-related routes.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
# Create a new user.
@app.route("/user/new", methods=["POST"])
def create_user():
try:
print(request.json)
new_user = User(**request.json)
new_user.save()
return new_user.pk
except ValidationError as e:
print(e)
return "Bad request.", 400
# Update user's name
@app.route("/user/<id>/name/<int:new_name>", methods=["POST"])
def update_name(id, new_name):
try:
user = user.get(id)
except NotFoundError:
return "Bad request", 400
user.name = new_name
user.save()
return "ok"
# Delete a user by ID.
@app.route("/user/<id>/delete", methods=["POST"])
def delete_user(id):
# Delete returns 1 if the user existed and was
# deleted, or 0 if they didn't exist. For our
# purposes, both outcomes can be considered a success.
user.delete(id)
return "ok"
# Create a RediSearch index for instances of the models.
Migrator().run()
|
Dependencies (requirements.txt)
Flask uses a txt file called requirements.txt to track / organize its application dependencies. Technically, you can name it anything as long as you pass in its name when installing.
Below is the file specific for our app.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
|
aioredis==2.0.1
async-timeout==4.0.2
certifi==2021.10.8
charset-normalizer==2.0.12
cleo==1.0.0a4
click==8.0.4
crashtest==0.3.1
Deprecated==1.2.13
Flask==2.0.3
hiredis==2.0.0
idna==3.3
itsdangerous==2.1.0
Jinja2==3.0.3
MarkupSafe==2.1.0
packaging==21.3
pptree==3.1
pydantic==1.9.0
pylev==1.4.0
pyparsing==3.0.7
python-dotenv==0.19.2
python-ulid==1.0.3
redis==4.1.4
redis-om==0.0.20
requests==2.27.1
six==1.16.0
types-redis==4.1.17
types-six==1.16.11
typing-extensions==4.1.1
urllib3==1.26.8
Werkzeug==2.0.3
wrapt==1.13.3
|
Seed Data
Lastly, we include some seed data for testing our app.
In our app directory, you can create below items.
1
2
3
4
|
├── data
│ ├── user.json
│ └── video.json
├── dataloader.py
|
user.json
1
2
3
4
5
6
7
8
9
10
|
[
{
"name": "Robert S",
"email": "zz@gmail.com"
},
{
"name": "Henry Z",
"email": "zzh@gmail.com"
}
]
|
video.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
[
{
"title": "Video1",
"description": "Test Video 1",
"uri": "www.google.com",
"likes": 0,
"likedBy": [{
"name": "Robert S",
"email": "zz@gmail.com"
}]
},
{
"title": "Video2",
"description": "Test Video 2",
"uri": "www.yahoo.com",
"likes": 3,
"likedBy": [{
"name": "Henry Z",
"email": "zzh@gmail.com"
}]
}
]
|
dataloader.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
import json
import requests
with open('data/user.json', encoding='utf-8') as f:
user = json.loads(f.read())
for u in user:
r = requests.post('http://127.0.0.1:5000/user/new', json = u)
print(f"Created u {u['name']} {u['email']} with ID {r.text}")
with open('data/video.json', encoding='utf-8') as f:
video = json.loads(f.read())
for vid in video:
r = requests.post('http://127.0.0.1:5000/video/new', json = vid)
print(f"Created vid {vid['title']} {vid['likes']} with ID {r.text}")
|
Once those files are in place, we can populate data by running below commands.
1
2
3
|
docker-compose up -d
flask run
python3 dataloader.py
|
You can also modify dataloader.py to test the API endpoints for various CRUD operations.
Summary
In this tutorial, I have walked through the code of a minimal video share API app, which uses Python and RedisJson. With a bit of work, we can make it horizontally scalable.
