Redis Library

A collection of convenient functions and redis/lua scripts.

This code was partial inspired by the Bullet-Proofing Lua Scripts in RedisPy article.

searx.redislib.drop_counter(client, name)

Drop counter with redis key SearXNG_counter_<name>

The replacement <name> is a secret hash of the value from argument name (see incr_counter() and incr_sliding_window()).

searx.redislib.incr_counter(client, name: str, limit: int = 0, expire: int = 0)

Increment a counter and return the new value.

If counter with redis key SearXNG_counter_<name> does not exists it is created with initial value 1 returned. The replacement <name> is a secret hash of the value from argument name (see secret_hash()).

The implementation of the redis counter is the lua script from string INCR_COUNTER.

Parameters:

• name (str) – name of the counter

• expire (int / see EXPIRE) – live-time of the counter in seconds (default None means infinite).

• limit (int / limit is 2^64 see INCR) – limit where the counter stops to increment (default None)

Returns:

value of the incremented counter

A simple demo of a counter with expire time and limit:

>>> for i in range(6):
...   i, incr_counter(client, "foo", 3, 5) # max 3, duration 5 sec
...   time.sleep(1) # from the third call on max has been reached
...
(0, 1)
(1, 2)
(2, 3)
(3, 3)
(4, 3)
(5, 1)

searx.redislib.incr_sliding_window(client, name: str, duration: int)

Increment a sliding-window counter and return the new value.

If counter with redis key SearXNG_counter_<name> does not exists it is created with initial value 1 returned. The replacement <name> is a secret hash of the value from argument name (see secret_hash()).

Parameters:

• name (str) – name of the counter

• duration – live-time of the sliding window in seconds

Typeduration:

int

Returns:

value of the incremented counter

The implementation of the redis counter is the lua script from string INCR_SLIDING_WINDOW. The lua script uses sorted sets in Redis to implement a sliding window for the redis key SearXNG_counter_<name> (ZADD). The current TIME is used to score the items in the sorted set and the time window is moved by removing items with a score lower current time minus duration time (ZREMRANGEBYSCORE).

The EXPIRE time (the duration of the sliding window) is refreshed on each call (increment) and if there is no call in this duration, the sorted set expires from the redis DB.

The return value is the amount of items in the sorted set (ZCOUNT), what means the number of calls in the sliding window.

A simple demo of the sliding window:

>>> for i in range(5):
...   incr_sliding_window(client, "foo", 3) # duration 3 sec
...   time.sleep(1) # from the third call (second) on the window is moved
...
1
2
3
3
3
>>> time.sleep(3)  # wait until expire
>>> incr_sliding_window(client, "foo", 3)
1

searx.redislib.lua_script_storage(client, script)

Returns a redis Script instance.

Due to performance reason the Script object is instantiated only once for a client (client.register_script(..)) and is cached in LUA_SCRIPT_STORAGE.

searx.redislib.purge_by_prefix(client, prefix: str = 'SearXNG_')

Purge all keys with prefix from database.

Queries all keys in the database by the given prefix and set expire time to zero. The default prefix will drop all keys which has been set by SearXNG (drops SearXNG schema entirely from database).

The implementation is the lua script from string PURGE_BY_PREFIX. The lua script uses EXPIRE instead of DEL: if there are a lot keys to delete and/or their values are big, DEL could take more time and blocks the command loop while EXPIRE turns back immediate.

Parameters:

prefix – prefix of the key to delete (default: SearXNG_)

searx.redislib.secret_hash(name: str)

Creates a hash of the name.

Combines argument name with the secret_key from server:. This function can be used to get a more anonymized name of a Redis KEY.

Parameters:

name (str) – the name to create a secret hash for

searx.redislib.LUA_SCRIPT_STORAGE = {}

A global dictionary to cache client’s Script objects, used by lua_script_storage