Favicons (source)¶

Implementations for providing the favicons in SearXNG

searx.favicons.favicon_proxy()[source]¶

REST API of SearXNG’s favicon proxy service

/favicon_proxy?authority=<...>&h=<...>

authority:: Domain name RFC 3986 / see favicon_url
h:: HMAC RFC 2104, build up from the server.secret_key setting.

searx.favicons.favicon_url(authority: str) → str[source]¶

Function to generate the image URL used for favicons in SearXNG’s result lists. The authority argument (aka netloc / RFC 3986) is usually a (sub-) domain name. This function is used in the HTML (jinja) templates.

<div class="favicon">
   <img src="{{ favicon_url(result.parsed_url.netloc) }}">
</div>

The returned URL is a route to favicon_proxy REST API.

If the favicon is already in the cache, the returned URL is a data URL (something like data:image/png;base64,...). By generating a data url from the cache.FaviconCache, additional HTTP roundtripps via the favicon_proxy are saved. However, it must also be borne in mind that data urls are not cached in the client (web browser).

Favicons Config ¶

pydantic model searx.favicons.config.FaviconConfig[source]¶

The class aggregates configurations of the favicon tools

Show JSON schema

{
   "title": "FaviconConfig",
   "description": "The class aggregates configurations of the favicon tools",
   "type": "object",
   "properties": {
      "cfg_schema": {
         "title": "Cfg Schema",
         "type": "integer"
      },
      "cache": {
         "$ref": "#/$defs/FaviconCacheConfig",
         "default": {
            "db_type": "sqlite",
            "db_url": "/tmp/faviconcache.db",
            "HOLD_TIME": 2592000,
            "LIMIT_TOTAL_BYTES": 52428800,
            "BLOB_MAX_BYTES": 20480,
            "MAINTENANCE_PERIOD": 3600,
            "MAINTENANCE_MODE": "auto"
         }
      },
      "proxy": {
         "$ref": "#/$defs/FaviconProxyConfig",
         "default": {
            "max_age": 604800,
            "secret_key": "",
            "resolver_timeout": 3.0,
            "resolver_map": {},
            "favicon_path": "/home/runner/work/searxng/searxng/searx/static/themes/{theme}/img/empty_favicon.svg",
            "favicon_mime_type": "image/svg+xml"
         }
      }
   },
   "$defs": {
      "FaviconCacheConfig": {
         "description": "Configuration of the favicon cache.",
         "properties": {
            "db_type": {
               "default": "sqlite",
               "enum": [
                  "sqlite",
                  "mem"
               ],
               "title": "Db Type",
               "type": "string"
            },
            "db_url": {
               "default": "/tmp/faviconcache.db",
               "format": "path",
               "title": "Db Url",
               "type": "string"
            },
            "HOLD_TIME": {
               "default": 2592000,
               "title": "Hold Time",
               "type": "integer"
            },
            "LIMIT_TOTAL_BYTES": {
               "default": 52428800,
               "title": "Limit Total Bytes",
               "type": "integer"
            },
            "BLOB_MAX_BYTES": {
               "default": 20480,
               "title": "Blob Max Bytes",
               "type": "integer"
            },
            "MAINTENANCE_PERIOD": {
               "default": 3600,
               "title": "Maintenance Period",
               "type": "integer"
            },
            "MAINTENANCE_MODE": {
               "default": "auto",
               "enum": [
                  "auto",
                  "off"
               ],
               "title": "Maintenance Mode",
               "type": "string"
            }
         },
         "title": "FaviconCacheConfig",
         "type": "object"
      },
      "FaviconProxyConfig": {
         "description": "Configuration of the favicon proxy.",
         "properties": {
            "max_age": {
               "default": 604800,
               "title": "Max Age",
               "type": "integer"
            },
            "secret_key": {
               "default": "",
               "title": "Secret Key",
               "type": "string"
            },
            "resolver_timeout": {
               "default": 3.0,
               "title": "Resolver Timeout",
               "type": "integer"
            },
            "resolver_map": {
               "additionalProperties": {
                  "type": "string"
               },
               "default": {},
               "title": "Resolver Map",
               "type": "object"
            },
            "favicon_path": {
               "default": "/home/runner/work/searxng/searxng/searx/static/themes/{theme}/img/empty_favicon.svg",
               "title": "Favicon Path",
               "type": "string"
            },
            "favicon_mime_type": {
               "default": "image/svg+xml",
               "title": "Favicon Mime Type",
               "type": "string"
            }
         },
         "title": "FaviconProxyConfig",
         "type": "object"
      }
   },
   "required": [
      "cfg_schema"
   ]
}

Fields:

cache (searx.favicons.cache.FaviconCacheConfig)
cfg_schema (int)
proxy (searx.favicons.proxy.FaviconProxyConfig)

field cache: FaviconCacheConfig = FaviconCacheConfig(db_type='sqlite', db_url=PosixPath('/tmp/faviconcache.db'), HOLD_TIME=2592000, LIMIT_TOTAL_BYTES=52428800, BLOB_MAX_BYTES=20480, MAINTENANCE_PERIOD=3600, MAINTENANCE_MODE='auto')¶: Setup of the cache.FaviconCacheConfig.

field cfg_schema: int [Required]¶: Config’s schema version. The specification of the version of the schema is mandatory, currently only version CONFIG_SCHEMA is supported. By specifying a version, it is possible to ensure downward compatibility in the event of future changes to the configuration schema

field proxy: FaviconProxyConfig = FaviconProxyConfig(max_age=604800, secret_key='', resolver_timeout=3.0, resolver_map={}, favicon_path='/home/runner/work/searxng/searxng/searx/static/themes/{theme}/img/empty_favicon.svg', favicon_mime_type='image/svg+xml')¶: Setup of the proxy.FaviconProxyConfig.

classmethod from_toml_file(cfg_file: Path, use_cache: bool) → FaviconConfig[source]¶: Create a config object from a TOML file, the use_cache argument specifies whether a cache should be used.

searx.favicons.config.CONFIG_SCHEMA: int = 1¶: Version of the configuration schema.

searx.favicons.config.TOML_CACHE_CFG: dict[str, FaviconConfig] = {}¶: Cache config objects by TOML’s filename.

Favicons Proxy ¶

Implementations for a favicon proxy

pydantic model searx.favicons.proxy.FaviconProxyConfig[source]¶

Configuration of the favicon proxy.

Show JSON schema

{
   "title": "FaviconProxyConfig",
   "description": "Configuration of the favicon proxy.",
   "type": "object",
   "properties": {
      "max_age": {
         "default": 604800,
         "title": "Max Age",
         "type": "integer"
      },
      "secret_key": {
         "default": "",
         "title": "Secret Key",
         "type": "string"
      },
      "resolver_timeout": {
         "default": 3.0,
         "title": "Resolver Timeout",
         "type": "integer"
      },
      "resolver_map": {
         "additionalProperties": {
            "type": "string"
         },
         "default": {},
         "title": "Resolver Map",
         "type": "object"
      },
      "favicon_path": {
         "default": "/home/runner/work/searxng/searxng/searx/static/themes/{theme}/img/empty_favicon.svg",
         "title": "Favicon Path",
         "type": "string"
      },
      "favicon_mime_type": {
         "default": "image/svg+xml",
         "title": "Favicon Mime Type",
         "type": "string"
      }
   }
}

Fields:

favicon_mime_type (str)
favicon_path (str)
max_age (int)
resolver_map (dict[str, str])
resolver_timeout (int)
secret_key (str)

field favicon_mime_type: str = 'image/svg+xml'¶

field favicon_path: str = '/home/runner/work/searxng/searxng/searx/static/themes/{theme}/img/empty_favicon.svg'¶

field max_age: int = 604800¶: HTTP header Cache-Control max-age

field resolver_map: dict[str, str] = {}¶: The resolver_map is a key / value dictionary where the key is the name of the resolver and the value is the fully qualifying name (fqn) of resolver’s function (the callable). The resolvers from the python module searx.favicons.resolver are available by default.

field resolver_timeout: int = 3.0¶: Timeout which the resolvers should not exceed, is usually passed to the outgoing request of the resolver. By default, the value from outgoing.request_timeout setting is used.

field secret_key: str = ''¶: By default, the value from server.secret_key setting is used.

favicon(**replacements)[source]¶: Returns pathname and mimetype of the default favicon.

favicon_data_url(**replacements)[source]¶: Returns data image URL of the default favicon.

get_resolver(name: str) → Callable | None[source]¶: Returns the callable object (function) of the resolver with the name. If no resolver is registered for the name, None is returned.

searx.favicons.proxy.favicon_proxy()[source]¶

REST API of SearXNG’s favicon proxy service

/favicon_proxy?authority=<...>&h=<...>

authority:: Domain name RFC 3986 / see favicon_url
h:: HMAC RFC 2104, build up from the server.secret_key setting.

searx.favicons.proxy.favicon_url(authority: str) → str[source]¶

Function to generate the image URL used for favicons in SearXNG’s result lists. The authority argument (aka netloc / RFC 3986) is usually a (sub-) domain name. This function is used in the HTML (jinja) templates.

<div class="favicon">
   <img src="{{ favicon_url(result.parsed_url.netloc) }}">
</div>

The returned URL is a route to favicon_proxy REST API.

If the favicon is already in the cache, the returned URL is a data URL (something like data:image/png;base64,...). By generating a data url from the cache.FaviconCache, additional HTTP roundtripps via the favicon_proxy are saved. However, it must also be borne in mind that data urls are not cached in the client (web browser).

searx.favicons.proxy.search_favicon(resolver: str, authority: str) → tuple[None | bytes, None | str][source]¶

Sends the request to the favicon resolver and returns a tuple for the favicon. The tuple consists of (data, mime), if the resolver has not determined a favicon, both values are None.

data:: Binary data of the favicon.
mime:: Mime type of the favicon.

Favicons Resolver ¶

Implementations of the favicon resolvers that are available in the favicon proxy by default. A resolver is a function that obtains the favicon from an external source. The resolver function receives two arguments (domain, timeout) and returns a tuple (data, mime).

searx.favicons.resolvers.allesedv(domain: str, timeout: int) → tuple[None | bytes, None | str][source]¶: Favicon Resolver from allesedv.com / https://favicon.allesedv.com/

searx.favicons.resolvers.duckduckgo(domain: str, timeout: int) → tuple[None | bytes, None | str][source]¶: Favicon Resolver from duckduckgo.com / https://blog.jim-nielsen.com/2021/displaying-favicons-for-any-domain/

searx.favicons.resolvers.google(domain: str, timeout: int) → tuple[None | bytes, None | str][source]¶: Favicon Resolver from google.com

searx.favicons.resolvers.yandex(domain: str, timeout: int) → tuple[None | bytes, None | str][source]¶: Favicon Resolver from yandex.com

Favicons Cache ¶

Implementations for caching favicons.

FaviconCacheConfig:: Configuration of the favicon cache
FaviconCache:: Abstract base class for the implementation of a favicon cache.
FaviconCacheSQLite:: Favicon cache that manages the favicon BLOBs in a SQLite DB.
FaviconCacheNull:: Fallback solution if the configured cache cannot be used for system reasons.

class searx.favicons.cache.FaviconCache(cfg: FaviconCacheConfig)[source]¶

Abstract base class for the implementation of a favicon cache.

abstract maintenance(force=False)[source]¶: Performs maintenance on the cache

abstract set(resolver: str, authority: str, mime: str | None, data: bytes | None) → bool[source]¶: Set data and mime-type in the cache. If data is None, the FALLBACK_ICON is registered. in the cache.

abstract state() → FaviconCacheStats[source]¶: Returns a FaviconCacheStats (key/values) with information on the state of the cache.

pydantic model searx.favicons.cache.FaviconCacheConfig[source]¶

Configuration of the favicon cache.

Show JSON schema

{
   "title": "FaviconCacheConfig",
   "description": "Configuration of the favicon cache.",
   "type": "object",
   "properties": {
      "db_type": {
         "default": "sqlite",
         "enum": [
            "sqlite",
            "mem"
         ],
         "title": "Db Type",
         "type": "string"
      },
      "db_url": {
         "default": "/tmp/faviconcache.db",
         "format": "path",
         "title": "Db Url",
         "type": "string"
      },
      "HOLD_TIME": {
         "default": 2592000,
         "title": "Hold Time",
         "type": "integer"
      },
      "LIMIT_TOTAL_BYTES": {
         "default": 52428800,
         "title": "Limit Total Bytes",
         "type": "integer"
      },
      "BLOB_MAX_BYTES": {
         "default": 20480,
         "title": "Blob Max Bytes",
         "type": "integer"
      },
      "MAINTENANCE_PERIOD": {
         "default": 3600,
         "title": "Maintenance Period",
         "type": "integer"
      },
      "MAINTENANCE_MODE": {
         "default": "auto",
         "enum": [
            "auto",
            "off"
         ],
         "title": "Maintenance Mode",
         "type": "string"
      }
   }
}

Fields:

BLOB_MAX_BYTES (int)
HOLD_TIME (int)
LIMIT_TOTAL_BYTES (int)
MAINTENANCE_MODE (Literal['auto', 'off'])
MAINTENANCE_PERIOD (int)
db_type (Literal['sqlite', 'mem'])
db_url (pathlib.Path)

field BLOB_MAX_BYTES: int = 20480¶: The maximum BLOB size in bytes that a favicon may have so that it can be saved in the cache. If the favicon is larger, it is not saved in the cache and must be requested by the client via the proxy.

field HOLD_TIME: int = 2592000¶: Hold time (default in sec.), after which a BLOB is removed from the cache.

field LIMIT_TOTAL_BYTES: int = 52428800¶: Maximum of bytes (default) stored in the cache of all blobs. Note: The limit is only reached at each maintenance interval after which the oldest BLOBs are deleted; the limit is exceeded during the maintenance period. If the maintenance period is too long or maintenance is switched off completely, the cache grows uncontrollably.

field MAINTENANCE_MODE: Literal['auto', 'off'] = 'auto'¶

Type of maintenance mode

auto:: Maintenance is carried out automatically as part of the maintenance intervals (MAINTENANCE_PERIOD); no external process is required.
off:: Maintenance is switched off and must be carried out by an external process if required.

field MAINTENANCE_PERIOD: int = 3600¶: Maintenance period in seconds / when MAINTENANCE_MODE is set to auto.

field db_type: Literal['sqlite', 'mem'] = 'sqlite'¶

Type of the database:

sqlite:: cache.FaviconCacheSQLite
mem:: cache.FaviconCacheMEM (not recommended)

field db_url: pathlib.Path = PosixPath('/tmp/faviconcache.db')¶: URL of the SQLite DB, the path to the database file.

class searx.favicons.cache.FaviconCacheMEM(cfg)[source]¶

Favicon cache in process’ memory. Its just a POC that stores the favicons in the memory of the process.

Attention

Don’t use it in production, it will blow up your memory!!

maintenance(force=False)[source]¶: Performs maintenance on the cache

set(resolver: str, authority: str, mime: str | None, data: bytes | None) → bool[source]¶: Set data and mime-type in the cache. If data is None, the FALLBACK_ICON is registered. in the cache.

state()[source]¶: Returns a FaviconCacheStats (key/values) with information on the state of the cache.

class searx.favicons.cache.FaviconCacheNull(cfg: FaviconCacheConfig)[source]¶

A dummy favicon cache that caches nothing / a fallback solution. The NullCache is used when more efficient caches such as the FaviconCacheSQLite cannot be used because, for example, the SQLite library is only available in an old version and does not meet the requirements.

maintenance(force=False)[source]¶: Performs maintenance on the cache

set(resolver: str, authority: str, mime: str | None, data: bytes | None) → bool[source]¶: Set data and mime-type in the cache. If data is None, the FALLBACK_ICON is registered. in the cache.

state()[source]¶: Returns a FaviconCacheStats (key/values) with information on the state of the cache.

class searx.favicons.cache.FaviconCacheSQLite(cfg: FaviconCacheConfig)[source]¶

Favicon cache that manages the favicon BLOBs in a SQLite DB. The DB model in the SQLite DB is implemented using the abstract class sqlitedb.SQLiteAppl.

The following configurations are required / supported:

FaviconCacheConfig.db_url
FaviconCacheConfig.HOLD_TIME
FaviconCacheConfig.LIMIT_TOTAL_BYTES
FaviconCacheConfig.BLOB_MAX_BYTES
MAINTENANCE_PERIOD
MAINTENANCE_MODE

maintenance(force=False)[source]¶: Performs maintenance on the cache

set(resolver: str, authority: str, mime: str | None, data: bytes | None) → bool[source]¶: Set data and mime-type in the cache. If data is None, the FALLBACK_ICON is registered. in the cache.

state() → FaviconCacheStats[source]¶: Returns a FaviconCacheStats (key/values) with information on the state of the cache.

DB_SCHEMA: int = 1¶: As soon as changes are made to the DB schema, the version number must be increased. Changes to the version number require the DB to be recreated (or migrated / if an migration path exists and is implemented).

DDL_BLOBS = 'CREATE TABLE IF NOT EXISTS blobs (\n sha256 TEXT,\n bytes_c INTEGER,\n mime TEXT NOT NULL,\n data BLOB NOT NULL,\n PRIMARY KEY (sha256))'¶: Table to store BLOB objects by their sha256 hash values.

DDL_BLOB_MAP = "CREATE TABLE IF NOT EXISTS blob_map (\n m_time INTEGER DEFAULT (strftime('%s', 'now')), -- last modified (unix epoch) time in sec.\n sha256 TEXT,\n resolver TEXT,\n authority TEXT,\n PRIMARY KEY (resolver, authority))"¶: Table to map from (resolver, authority) to sha256 hash values.

SQL_DROP_LEFTOVER_BLOBS = 'DELETE FROM blobs WHERE sha256 IN ( SELECT b.sha256 FROM blobs b LEFT JOIN blob_map bm ON b.sha256 = bm.sha256 WHERE bm.sha256 IS NULL)'¶: Delete blobs.sha256 (BLOBs) no longer in blob_map.sha256.

property next_maintenance_time: int¶: Returns (unix epoch) time of the next maintenance.

class searx.favicons.cache.FaviconCacheStats(favicons: int | None = None, bytes: int | None = None, domains: int | None = None, resolvers: int | None = None)[source]¶: Dataclass wich provides information on the status of the cache.

searx.favicons.cache.init(cfg: FaviconCacheConfig)[source]¶: Initialization of a global CACHE

searx.favicons.cache.maintenance(force: bool = True, debug: bool = False)[source]¶: perform maintenance of the cache

searx.favicons.cache.state()[source]¶: show state of the cache

Favicons (source)¶

Favicons Config ¶

Favicons Proxy ¶

Favicons Resolver ¶

Favicons Cache ¶

Table of Contents

Project Links

Navigation

This Page

Favicons (source)¶

Favicons Config¶

Favicons Proxy¶

Favicons Resolver¶

Favicons Cache¶

Favicons Config ¶

Favicons Proxy ¶

Favicons Resolver ¶

Favicons Cache ¶