feat: added basic structure for searXNG and netdata
This commit is contained in:
Geir Okkenhaug Jerstad
parent
fa5de8f1bb
commit
aa607747c3
8 changed files with 1357 additions and 57 deletions
303
modules/services/README-SearXNG.md
Normal file
303
modules/services/README-SearXNG.md
Normal file
|
|
@ -0,0 +1,303 @@
|
|||
# SearXNG Service Configuration Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This module provides a secure, privacy-focused SearXNG metasearch engine configuration designed for home lab environments. Key features include:
|
||||
|
||||
- **Privacy-first**: No query logging, private instance settings
|
||||
- **Network Security**: Only accessible from Tailscale VPN network
|
||||
- **Reverse Proxy**: Uses your reverse-proxy machine for outbound traffic
|
||||
- **Security Hardening**: Systemd security restrictions, Content Security Policy headers
|
||||
- **Easy Configuration**: Simple NixOS module with sensible defaults
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
[Client on Tailscale]
|
||||
↓ (HTTP/HTTPS)
|
||||
[SearXNG Service]
|
||||
↓ (Search requests via HTTP proxy)
|
||||
[Reverse Proxy]
|
||||
↓ (Outbound to internet)
|
||||
[Search Engines]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Basic Setup
|
||||
|
||||
Add to your machine's `configuration.nix`:
|
||||
|
||||
```nix
|
||||
{
|
||||
imports = [ ../../modules/services/SearXNG.nix ];
|
||||
|
||||
services.searxng-lab = {
|
||||
enable = true;
|
||||
hostName = "searxng.your-lab.internal";
|
||||
reverseProxyHost = "reverse-proxy"; # Your reverse proxy hostname
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### Advanced Configuration
|
||||
|
||||
```nix
|
||||
{
|
||||
services.searxng-lab = {
|
||||
enable = true;
|
||||
hostName = "search.lab.local";
|
||||
port = 8888; # SearXNG backend port
|
||||
reverseProxyHost = "proxy.lab.local";
|
||||
reverseProxyPort = 3128; # HTTP proxy port
|
||||
openFirewall = true; # Open port 80 for HTTP access
|
||||
tailscaleOnly = true; # Restrict to Tailscale network
|
||||
nginxVhost = true; # Create Nginx reverse proxy
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## Reverse Proxy Configuration
|
||||
|
||||
Your reverse-proxy machine needs to provide HTTP proxy functionality. Add this to your reverse-proxy configuration:
|
||||
|
||||
```nix
|
||||
# On reverse-proxy machine
|
||||
services.squid = {
|
||||
enable = true;
|
||||
configText = ''
|
||||
# Basic proxy configuration
|
||||
http_port 3128
|
||||
|
||||
# Allow connections from your home lab network
|
||||
acl homelab src 192.168.1.0/24
|
||||
acl homelab src 100.0.0.0/8 # Tailscale network
|
||||
|
||||
# Allow CONNECT for HTTPS
|
||||
acl CONNECT method CONNECT
|
||||
acl SSL_ports port 443
|
||||
|
||||
# Access rules
|
||||
http_access allow homelab
|
||||
http_access deny all
|
||||
|
||||
# Don't log for privacy
|
||||
access_log none
|
||||
|
||||
# Hide client IP from destination servers
|
||||
forwarded_for delete
|
||||
'';
|
||||
};
|
||||
|
||||
# Open firewall for proxy
|
||||
networking.firewall.allowedTCPPorts = [ 3128 ];
|
||||
```
|
||||
|
||||
## Network Access
|
||||
|
||||
### Tailscale Setup
|
||||
|
||||
Ensure Tailscale is configured on both the SearXNG host and client machines:
|
||||
|
||||
```nix
|
||||
# On SearXNG host and client machines
|
||||
services.tailscale.enable = true;
|
||||
```
|
||||
|
||||
### DNS Configuration
|
||||
|
||||
Add hostname resolution to your machines:
|
||||
|
||||
```nix
|
||||
networking.hosts = {
|
||||
"100.x.x.x" = [ "searxng.lab.local" ]; # Replace with actual Tailscale IP
|
||||
};
|
||||
```
|
||||
|
||||
## Security Features
|
||||
|
||||
### Network Restrictions
|
||||
|
||||
- Only accepts connections from Tailscale network (100.0.0.0/8)
|
||||
- Nginx access controls deny non-Tailscale traffic
|
||||
- Systemd IPAddressAllow/Deny for service-level restrictions
|
||||
|
||||
### Privacy Protection
|
||||
|
||||
- No query logging enabled
|
||||
- Private instance (not listed publicly)
|
||||
- Rate limiting to prevent abuse
|
||||
- Secure headers (CSP, HSTS, etc.)
|
||||
|
||||
### Systemd Hardening
|
||||
|
||||
- NoNewPrivileges: Prevents privilege escalation
|
||||
- PrivateTmp: Isolated temporary directory
|
||||
- ProtectSystem: Read-only system directories
|
||||
- RestrictAddressFamilies: Only IPv4/IPv6 allowed
|
||||
|
||||
## Usage
|
||||
|
||||
1. **Deploy the Configuration**:
|
||||
|
||||
```bash
|
||||
nixos-rebuild switch
|
||||
```
|
||||
|
||||
2. **Check Service Status**:
|
||||
|
||||
```bash
|
||||
systemctl status searxng
|
||||
systemctl status nginx
|
||||
```
|
||||
|
||||
3. **Access the Interface**:
|
||||
Navigate to `http://searxng.lab.local` from a Tailscale-connected device
|
||||
|
||||
4. **Test Search**:
|
||||
Try searching for something to verify it works and uses the proxy
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Check Service Logs
|
||||
|
||||
```bash
|
||||
journalctl -u searxng -f
|
||||
journalctl -u nginx -f
|
||||
```
|
||||
|
||||
### Test Network Connectivity
|
||||
|
||||
```bash
|
||||
# Test if SearXNG is running
|
||||
curl -s http://localhost:8888 | head
|
||||
|
||||
# Test proxy connectivity from SearXNG host
|
||||
curl -x http://reverse-proxy:3128 http://httpbin.org/ip
|
||||
|
||||
# Test Tailscale connectivity
|
||||
ping 100.x.x.x # Replace with your Tailscale IP
|
||||
```
|
||||
|
||||
### Verify Proxy Usage
|
||||
|
||||
Check that searches go through your reverse proxy by monitoring its logs:
|
||||
|
||||
```bash
|
||||
# On reverse-proxy machine
|
||||
journalctl -u squid -f
|
||||
```
|
||||
|
||||
### Debug Network Access
|
||||
|
||||
```bash
|
||||
# Test access restriction
|
||||
curl -H "Host: searxng.lab.local" http://your-tailscale-ip/
|
||||
|
||||
# Check Nginx access logs
|
||||
tail -f /var/log/nginx/access.log
|
||||
```
|
||||
|
||||
## Customization
|
||||
|
||||
### Search Engines
|
||||
|
||||
Modify the `searxngSettings` in the module to enable/disable specific search engines:
|
||||
|
||||
```nix
|
||||
# In the module configuration
|
||||
engines = [
|
||||
{
|
||||
name = "google";
|
||||
disabled = false;
|
||||
}
|
||||
{
|
||||
name = "bing";
|
||||
disabled = true;
|
||||
}
|
||||
];
|
||||
```
|
||||
|
||||
### Themes and UI
|
||||
|
||||
Change the UI theme and settings:
|
||||
|
||||
```nix
|
||||
ui = {
|
||||
default_theme = "oscar"; # Other options: simple, oscar, pix-art
|
||||
infinite_scroll = true;
|
||||
center_alignment = true;
|
||||
};
|
||||
```
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
Adjust rate limiting settings:
|
||||
|
||||
```nix
|
||||
server = {
|
||||
limiter = true;
|
||||
public_instance = false;
|
||||
method = "POST";
|
||||
};
|
||||
```
|
||||
|
||||
## Integration with Other Services
|
||||
|
||||
### Monitoring
|
||||
|
||||
Add Prometheus monitoring (optional):
|
||||
|
||||
```nix
|
||||
services.prometheus.exporters.nginx.enable = true;
|
||||
services.prometheus.scrapeConfigs = [
|
||||
{
|
||||
job_name = "searxng";
|
||||
static_configs = [
|
||||
{ targets = [ "localhost:8888" ]; }
|
||||
];
|
||||
}
|
||||
];
|
||||
```
|
||||
|
||||
### Backup
|
||||
|
||||
Include SearXNG data in your backup strategy:
|
||||
|
||||
```nix
|
||||
# Add to your backup configuration
|
||||
"/var/lib/searxng"
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Keep Updated**: Regularly update SearXNG package
|
||||
2. **Monitor Access**: Review Nginx access logs periodically
|
||||
3. **Proxy Security**: Ensure reverse proxy is properly secured
|
||||
4. **Tailscale Security**: Use Tailscale ACLs to further restrict access
|
||||
5. **Instance Privacy**: Never expose this instance publicly
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### For High Usage
|
||||
|
||||
```nix
|
||||
# Increase worker processes
|
||||
systemd.services.searxng.serviceConfig.MemoryMax = "1G";
|
||||
|
||||
# Nginx caching
|
||||
services.nginx.virtualHosts."searxng.lab.local".locations."/static/" = {
|
||||
expires = "1d";
|
||||
extraConfig = "add_header Cache-Control public;";
|
||||
};
|
||||
```
|
||||
|
||||
### For Low Resources
|
||||
|
||||
```nix
|
||||
# Reduce memory usage
|
||||
systemd.services.searxng.serviceConfig.MemoryMax = "256M";
|
||||
|
||||
# Disable image proxy if not needed
|
||||
server.image_proxy = false;
|
||||
```
|
||||
47
modules/services/SearXNG-example.nix
Normal file
47
modules/services/SearXNG-example.nix
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
# Example SearXNG Configuration for NixOS Home Lab
|
||||
#
|
||||
# This example shows how to enable SearXNG on one of your machines
|
||||
# with proper reverse proxy integration and Tailscale-only access.
|
||||
#
|
||||
# Add this to your machine's configuration.nix:
|
||||
{
|
||||
imports = [
|
||||
../../modules/services/SearXNG.nix
|
||||
];
|
||||
|
||||
services.searxng-lab = {
|
||||
enable = true;
|
||||
hostName = "searxng.lab.local";
|
||||
|
||||
# Configure reverse proxy for outbound traffic
|
||||
reverseProxyHost = "reverse-proxy";
|
||||
reverseProxyPort = 3128;
|
||||
|
||||
# Security settings
|
||||
tailscaleOnly = true; # Only allow Tailscale network access
|
||||
openFirewall = true; # Open HTTP port
|
||||
nginxVhost = true; # Create Nginx virtual host
|
||||
};
|
||||
|
||||
# Optional: Add hostname to /etc/hosts for easy access
|
||||
networking.hosts = {
|
||||
"127.0.0.1" = ["searxng.lab.local"];
|
||||
};
|
||||
}
|
||||
# Reverse Proxy Configuration (add to reverse-proxy machine)
|
||||
#
|
||||
# services.squid = {
|
||||
# enable = true;
|
||||
# configText = ''
|
||||
# http_port 3128
|
||||
# acl homelab src 100.0.0.0/8
|
||||
# acl homelab src 192.168.1.0/24
|
||||
# http_access allow homelab
|
||||
# http_access deny all
|
||||
# access_log none
|
||||
# forwarded_for delete
|
||||
# '';
|
||||
# };
|
||||
#
|
||||
# networking.firewall.allowedTCPPorts = [ 3128 ];
|
||||
|
||||
|
|
@ -0,0 +1,316 @@
|
|||
# SearXNG Service Configuration for Home Lab
|
||||
#
|
||||
# This module provides SearXNG configuration for private metasearch engine
|
||||
# SearXNG aggregates results from various search engines while preserving privacy
|
||||
#
|
||||
# Features:
|
||||
# - Uses reverse proxy for outbound traffic to search engines
|
||||
# - Web UI only accessible from Tailscale network
|
||||
# - No logging of user queries for privacy
|
||||
# - Nginx reverse proxy with security headers
|
||||
# - Rate limiting and security hardening
|
||||
#
|
||||
# Usage:
|
||||
# In your machine's configuration.nix, add:
|
||||
# imports = [ ../../modules/services/SearXNG.nix ];
|
||||
# services.searxng-lab = {
|
||||
# enable = true;
|
||||
# hostName = "searxng.your-domain.com";
|
||||
# reverseProxyHost = "reverse-proxy";
|
||||
# };
|
||||
{
|
||||
config,
|
||||
lib,
|
||||
pkgs,
|
||||
...
|
||||
}: let
|
||||
cfg = config.services.searxng-lab;
|
||||
|
||||
# Generate a unique secret key for this instance
|
||||
secretKeyFile = pkgs.writeText "searxng-secret" (builtins.hashString "sha256" cfg.hostName);
|
||||
|
||||
# SearXNG settings configuration
|
||||
searxngSettings = {
|
||||
use_default_settings = true;
|
||||
|
||||
server = {
|
||||
port = cfg.port;
|
||||
bind_address = "127.0.0.1";
|
||||
secret_key = "@SECRET_KEY@"; # Will be replaced at runtime
|
||||
base_url = "http://${cfg.hostName}";
|
||||
image_proxy = true;
|
||||
public_instance = false;
|
||||
limiter = true;
|
||||
method = "POST";
|
||||
|
||||
default_http_headers = {
|
||||
X-Content-Type-Options = "nosniff";
|
||||
X-Download-Options = "noopen";
|
||||
X-Robots-Tag = "noindex, nofollow";
|
||||
Referrer-Policy = "no-referrer";
|
||||
X-Frame-Options = "DENY";
|
||||
X-XSS-Protection = "1; mode=block";
|
||||
};
|
||||
};
|
||||
|
||||
general = {
|
||||
debug = false;
|
||||
instance_name = "Private Search - ${cfg.hostName}";
|
||||
privacypolicy_url = false;
|
||||
donation_url = false;
|
||||
contact_url = false;
|
||||
enable_metrics = false;
|
||||
};
|
||||
|
||||
search = {
|
||||
safe_search = 0;
|
||||
autocomplete = "";
|
||||
default_lang = "en";
|
||||
ban_time_on_fail = 5;
|
||||
max_ban_time_on_fail = 120;
|
||||
formats = ["html" "json"];
|
||||
};
|
||||
|
||||
ui = {
|
||||
static_use_hash = true;
|
||||
default_locale = "en";
|
||||
query_in_title = false;
|
||||
infinite_scroll = false;
|
||||
center_alignment = false;
|
||||
default_theme = "simple";
|
||||
hotkeys = "default";
|
||||
search_on_category_select = true;
|
||||
};
|
||||
|
||||
# Configure outbound requests through reverse proxy
|
||||
outgoing = {
|
||||
request_timeout = 3.0;
|
||||
useragent_suffix = "";
|
||||
pool_connections = 100;
|
||||
pool_maxsize = 10;
|
||||
enable_http2 = true;
|
||||
using_tor_proxy = false;
|
||||
|
||||
# Use reverse proxy for all outbound requests
|
||||
proxies = {
|
||||
http = "http://${cfg.reverseProxyHost}:${toString cfg.reverseProxyPort}";
|
||||
https = "http://${cfg.reverseProxyHost}:${toString cfg.reverseProxyPort}";
|
||||
};
|
||||
};
|
||||
};
|
||||
|
||||
# Convert settings to YAML format
|
||||
settingsFile = pkgs.writeText "searxng-settings.yml" (lib.generators.toYAML {} searxngSettings);
|
||||
in {
|
||||
options.services.searxng-lab = {
|
||||
enable = lib.mkEnableOption "SearXNG metasearch engine for home lab";
|
||||
|
||||
hostName = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "searxng.local";
|
||||
example = "search.example.com";
|
||||
description = "Hostname for the SearXNG instance";
|
||||
};
|
||||
|
||||
port = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 8888;
|
||||
description = "Port for SearXNG to listen on";
|
||||
};
|
||||
|
||||
reverseProxyHost = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = "reverse-proxy";
|
||||
example = "proxy.internal.lan";
|
||||
description = "Hostname of the reverse proxy for outbound traffic";
|
||||
};
|
||||
|
||||
reverseProxyPort = lib.mkOption {
|
||||
type = lib.types.port;
|
||||
default = 3128;
|
||||
description = "Port of the reverse proxy for outbound traffic";
|
||||
};
|
||||
|
||||
openFirewall = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = true;
|
||||
description = "Whether to open firewall for HTTP access";
|
||||
};
|
||||
|
||||
tailscaleOnly = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = true;
|
||||
description = "Whether to restrict access to Tailscale network only";
|
||||
};
|
||||
|
||||
nginxVhost = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = true;
|
||||
description = "Whether to create Nginx virtual host";
|
||||
};
|
||||
};
|
||||
|
||||
config = lib.mkIf cfg.enable {
|
||||
# Create SearXNG user and group
|
||||
users.users.searxng = {
|
||||
isSystemUser = true;
|
||||
group = "searxng";
|
||||
home = "/var/lib/searxng";
|
||||
createHome = true;
|
||||
description = "SearXNG metasearch engine user";
|
||||
};
|
||||
|
||||
users.groups.searxng = {};
|
||||
|
||||
# Install SearXNG package
|
||||
environment.systemPackages = with pkgs; [
|
||||
searxng
|
||||
];
|
||||
|
||||
# Create configuration directory and files
|
||||
environment.etc."searxng/settings.yml" = {
|
||||
source = settingsFile;
|
||||
mode = "0644";
|
||||
};
|
||||
|
||||
# SearXNG systemd service
|
||||
systemd.services.searxng = {
|
||||
description = "SearXNG metasearch engine";
|
||||
after = ["network-online.target"];
|
||||
wants = ["network-online.target"];
|
||||
wantedBy = ["multi-user.target"];
|
||||
|
||||
environment = {
|
||||
SEARXNG_SETTINGS_PATH = "/etc/searxng/settings.yml";
|
||||
PYTHONPATH = "${pkgs.searxng.pythonPath}";
|
||||
};
|
||||
|
||||
serviceConfig = {
|
||||
Type = "exec";
|
||||
User = "searxng";
|
||||
Group = "searxng";
|
||||
ExecStart = "${pkgs.searxng}/bin/searxng-run";
|
||||
Restart = "always";
|
||||
RestartSec = "10s";
|
||||
|
||||
# Security hardening
|
||||
NoNewPrivileges = true;
|
||||
PrivateTmp = true;
|
||||
ProtectSystem = "strict";
|
||||
ProtectHome = true;
|
||||
ReadWritePaths = ["/var/lib/searxng" "/tmp"];
|
||||
RestrictAddressFamilies = ["AF_INET" "AF_INET6"];
|
||||
|
||||
# Process limits
|
||||
LimitNOFILE = 4096;
|
||||
MemoryMax = "512M";
|
||||
|
||||
# Network restrictions for Tailscale-only access
|
||||
IPAddressDeny = lib.mkIf cfg.tailscaleOnly "any";
|
||||
IPAddressAllow = lib.mkIf cfg.tailscaleOnly [
|
||||
"100.0.0.0/8" # Tailscale network
|
||||
"127.0.0.0/8" # Localhost
|
||||
"::1/128" # IPv6 localhost
|
||||
];
|
||||
};
|
||||
|
||||
preStart = ''
|
||||
# Generate secret key and update settings
|
||||
SECRET_KEY=$(${pkgs.openssl}/bin/openssl rand -hex 32)
|
||||
${pkgs.gnused}/bin/sed -i "s/@SECRET_KEY@/$SECRET_KEY/g" /etc/searxng/settings.yml
|
||||
|
||||
# Ensure proper permissions
|
||||
mkdir -p /var/lib/searxng/cache
|
||||
chown -R searxng:searxng /var/lib/searxng
|
||||
'';
|
||||
};
|
||||
|
||||
# Nginx reverse proxy configuration
|
||||
services.nginx = lib.mkIf cfg.nginxVhost {
|
||||
enable = true;
|
||||
recommendedGzipSettings = true;
|
||||
recommendedOptimisation = true;
|
||||
recommendedProxySettings = true;
|
||||
recommendedTlsSettings = true;
|
||||
virtualHosts."${cfg.hostName}" = {
|
||||
listen = [
|
||||
{
|
||||
addr = "0.0.0.0";
|
||||
port = 80;
|
||||
}
|
||||
];
|
||||
|
||||
locations."/" = {
|
||||
proxyPass = "http://127.0.0.1:${toString cfg.port}";
|
||||
|
||||
extraConfig = ''
|
||||
# Proxy headers
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# Hide server information
|
||||
proxy_hide_header X-Powered-By;
|
||||
proxy_hide_header Server;
|
||||
|
||||
${lib.optionalString cfg.tailscaleOnly ''
|
||||
# Restrict access to Tailscale network only
|
||||
allow 100.0.0.0/8; # Tailscale network
|
||||
allow 127.0.0.1; # Localhost
|
||||
allow ::1; # IPv6 localhost
|
||||
deny all; # Deny all other access
|
||||
''}
|
||||
'';
|
||||
};
|
||||
|
||||
# Security headers for all responses
|
||||
extraConfig = ''
|
||||
# Security headers
|
||||
add_header X-Frame-Options DENY always;
|
||||
add_header X-Content-Type-Options nosniff always;
|
||||
add_header X-XSS-Protection "1; mode=block" always;
|
||||
add_header Referrer-Policy "no-referrer" always;
|
||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||
|
||||
# Content Security Policy for SearXNG
|
||||
add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self'; connect-src 'self'; frame-ancestors 'none'; form-action 'self';" always;
|
||||
|
||||
# Hide server information
|
||||
server_tokens off;
|
||||
more_clear_headers Server;
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
# Firewall configuration
|
||||
networking.firewall = lib.mkIf cfg.openFirewall {
|
||||
allowedTCPPorts = [80];
|
||||
|
||||
# Tailscale-specific firewall rules
|
||||
extraCommands = lib.optionalString cfg.tailscaleOnly ''
|
||||
# Allow Tailscale network access to HTTP
|
||||
iptables -I nixos-fw -i tailscale0 -p tcp --dport 80 -j ACCEPT
|
||||
iptables -I nixos-fw -s 100.0.0.0/8 -p tcp --dport 80 -j ACCEPT
|
||||
'';
|
||||
|
||||
extraStopCommands = lib.optionalString cfg.tailscaleOnly ''
|
||||
# Clean up Tailscale rules
|
||||
iptables -D nixos-fw -i tailscale0 -p tcp --dport 80 -j ACCEPT 2>/dev/null || true
|
||||
iptables -D nixos-fw -s 100.0.0.0/8 -p tcp --dport 80 -j ACCEPT 2>/dev/null || true
|
||||
'';
|
||||
};
|
||||
|
||||
# Ensure required directories exist
|
||||
systemd.tmpfiles.rules = [
|
||||
"d /var/lib/searxng 0755 searxng searxng -"
|
||||
"d /var/lib/searxng/cache 0755 searxng searxng -"
|
||||
];
|
||||
|
||||
# Add any additional packages needed
|
||||
environment.systemPackages = with pkgs; [
|
||||
curl # For health checks
|
||||
jq # For JSON processing if needed
|
||||
];
|
||||
};
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue