diff --git a/hosts/blacksteel/Caddyfile b/hosts/blacksteel/Caddyfile index 8c921e3..a4a5109 100644 --- a/hosts/blacksteel/Caddyfile +++ b/hosts/blacksteel/Caddyfile @@ -14,6 +14,14 @@ } } +http://pek0.ny4.dev:80 { + import default + basicauth { + prometheus $2a$14$2Phk4tobM04H4XiGegB3TuEXkyORCKMKW8TptYPTPXUWmZgtGBj/. + } + reverse_proxy localhost:9091 +} + http://mastodon.ny4.dev:80 { import default handle_path /system/* { diff --git a/hosts/blacksteel/default.nix b/hosts/blacksteel/default.nix index f8ab7f9..485e376 100644 --- a/hosts/blacksteel/default.nix +++ b/hosts/blacksteel/default.nix @@ -61,6 +61,7 @@ ingress = lib.genAttrs [ "mastodon.ny4.dev" "matrix.ny4.dev" + "pek0.ny4.dev" ] (_: "http://localhost"); }; }; diff --git a/hosts/tyo0/Caddyfile b/hosts/tyo0/Caddyfile index dcbd0b3..0656b88 100644 --- a/hosts/tyo0/Caddyfile +++ b/hosts/tyo0/Caddyfile @@ -15,7 +15,10 @@ www.ny4.dev { # get the certificate for hysteria tyo0.ny4.dev { import default - redir https://blog.ny4.dev + basicauth { + prometheus $2a$14$2Phk4tobM04H4XiGegB3TuEXkyORCKMKW8TptYPTPXUWmZgtGBj/. + } + reverse_proxy localhost:9091 } ny4.dev { @@ -105,3 +108,8 @@ vault.ny4.dev { import default reverse_proxy localhost:9500 } + +prom.ny4.dev { + import default + reverse_proxy localhost:9090 +} diff --git a/hosts/tyo0/default.nix b/hosts/tyo0/default.nix index 47b6ada..9e4d961 100644 --- a/hosts/tyo0/default.nix +++ b/hosts/tyo0/default.nix @@ -1,5 +1,6 @@ { lib, + config, modulesPath, pkgs, ... @@ -14,6 +15,7 @@ ./services/miniflux.nix ./services/murmur.nix ./services/ntfy.nix + ./services/prometheus.nix ./services/redlib.nix ./services/sing-box.nix ./services/vaultwarden.nix @@ -38,6 +40,10 @@ "sing-box/auth" = { restartUnits = [ "sing-box.service" ]; }; + "prometheus/auth" = { + owner = config.systemd.services.prometheus.serviceConfig.User; + restartUnits = [ "prometheus.service" ]; + }; "miniflux/environment" = { restartUnits = [ "miniflux.service" ]; }; diff --git a/hosts/tyo0/secrets.yaml b/hosts/tyo0/secrets.yaml index 820dbda..2243c5d 100644 --- a/hosts/tyo0/secrets.yaml +++ b/hosts/tyo0/secrets.yaml @@ -4,6 +4,8 @@ miniflux: environment: ENC[AES256_GCM,data:eT1rVeXbDANk/+9xmxmTHvMNofyplNGvVFgTj4lFQlJSHTi+br1qfg0tddf5aCtE8cNGt0fNm63qguI2Df/+KWENhb0vCpjRG7zryfBhEwMP5jkVgDnaHYolS1z3OmhlEpE=,iv:tWAUCtlk8wDGWGmn7j00QOVwjPYDkTPDGpyxd1pP6ig=,tag:gLNdzK9GZ/m5mWL5YNrzyQ==,type:str] vaultwarden: environment: ENC[AES256_GCM,data:+pcUVL7yVXKVp57/feHHWmSuH/2B0hLtADxZWCQOOMG+M3UQh+4dHA5debiv,iv:Zy6xn4Z4VwVXfWWjVeCYY/gRnDp//7yUPLbtLuABFPY=,tag:LxEc31YhgyjEhDrqoJxCJw==,type:str] +prometheus: + auth: ENC[AES256_GCM,data:sQ7oEL2gGz2nnn+QGcmmI3IwNEWbZ13s2/3QLj0O0BZp,iv:r7F70DzMNrcuxq2LISwm4tXjiR8m9eyt8GQyiuWxvhM=,tag:LfpxK3wcuMFCmFQn/iPZsw==,type:str] sops: kms: [] gcp_kms: [] @@ -28,8 +30,8 @@ sops: UkYrb3JpZDBzOUgzWXFQbUZnWjNUUjAKKuJmaJ6kV5ITsCMXEOzv9ym3L9VQKoB4 n/SE4eCXeaoE/1UCdw4VlpyuUuouHh2pgLWJF49dHhY/zhv84sURtA== -----END AGE ENCRYPTED FILE----- - lastmodified: "2024-08-29T15:22:29Z" - mac: ENC[AES256_GCM,data:wZzk/3ZdCXpMhMfIKbT0ZVm9k+c50MxWwZ88zZv0s44jYgWarzR92W09bTcOxw+SIfakdKt9y4aQENES1+JkGor3JpzxyVO4SGPaiZRFgNjjwAJJ2mAGTI3E69giirQipVHWOaPChZrpfCD2xa5Xrgm+as4fQpQrkgcv9ebyjrQ=,iv:GYsml4JuZ13OCMYcZiynaIlSU2V5lhsJd1GfSrOK/Oc=,tag:QodmEPuhmKA+/nuhP2Cufg==,type:str] + lastmodified: "2024-08-29T16:26:25Z" + mac: ENC[AES256_GCM,data:jpm+TBCtdFcgfRvzg+mTgWtu20/rm6nF/OdxUGbufkC1Y0Z8+eb8nIBe1TJhodt6kT/NdPRVI0N1JLD5XOwduvqL/QoZGzGkBfEVqFvnTxQYVVXp4sWdqji26XPb1sn+gbmobR4qlZPxdmvKZWEQxO2VJpKA3Bfalwa9fy0ajHE=,iv:XDRDEP/+rs2DLLkrftSxlxDMbdz7W9nHBEs0QWIDK88=,tag:UVmyD5FOev9LPRBvMcmJyw==,type:str] pgp: [] unencrypted_suffix: _unencrypted version: 3.9.0 diff --git a/hosts/tyo0/services/prometheus.nix b/hosts/tyo0/services/prometheus.nix new file mode 100644 index 0000000..d05401b --- /dev/null +++ b/hosts/tyo0/services/prometheus.nix @@ -0,0 +1,140 @@ +{ + lib, + pkgs, + config, + ... +}: +{ + services.prometheus = { + enable = true; + listenAddress = "127.0.0.1"; + port = 9090; + + exporters.blackbox = { + enable = true; + listenAddress = "127.0.0.1"; + port = 9093; + configFile = (pkgs.formats.yaml { }).generate "config.yaml" { + modules = { + http_2xx = { + prober = "http"; + }; + }; + }; + }; + + scrapeConfigs = [ + { + job_name = "metrics"; + scheme = "https"; + basic_auth = { + username = "prometheus"; + password_file = config.sops.secrets."prometheus/auth".path; + }; + static_configs = lib.singleton { + targets = [ + "tyo0.ny4.dev" + "pek0.ny4.dev" + ]; + }; + } + { + job_name = "http"; + scheme = "http"; + metrics_path = "/probe"; + params = { + module = [ "http_2xx" ]; + }; + static_configs = lib.singleton { + targets = [ + "https://blog.ny4.dev" + "https://cinny.ny4.dev" + "https://element.ny4.dev" + "https://git.ny4.dev" + "https://id.ny4.dev" + "https://mastodon.ny4.dev" + "https://matrix.ny4.dev" + "https://ntfy.ny4.dev" + "https://pb.ny4.dev" + "https://reddit.ny4.dev" + "https://rss.ny4.dev" + "https://vault.ny4.dev" + ]; + }; + relabel_configs = [ + { + source_labels = [ "__address__" ]; + target_label = "__param_target"; + } + { + source_labels = [ "__param_target" ]; + target_label = "instance"; + } + { + target_label = "__address__"; + replacement = "127.0.0.1:9093"; + } + ]; + } + + ]; + + rules = lib.singleton ( + builtins.toJSON { + groups = lib.singleton { + name = "metrics"; + rules = [ + { + alert = "NodeDown"; + expr = ''up == 0''; + for = "5m"; + } + { + alert = "HTTPDown"; + expr = ''probe_http_status_code < 200 or probe_http_status_code > 299''; + for = "5m"; + } + { + alert = "MemoryFull"; + expr = ''node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1''; + } + { + alert = "DiskFull"; + expr = ''node_filesystem_avail_bytes{mountpoint=~"/|/mnt"} / node_filesystem_size_bytes < 0.1''; + } + { + alert = "UnitFailed"; + expr = ''node_systemd_unit_state{state="failed"} == 1''; + } + ]; + }; + } + ); + + alertmanagers = lib.singleton { + static_configs = lib.singleton { + targets = [ + "127.0.0.1:9092" + ]; + }; + }; + + alertmanager = { + enable = true; + listenAddress = "127.0.0.1"; + port = 9092; + + configuration = { + receivers = lib.singleton { + name = "ntfy"; + webhook_configs = lib.singleton { + url = "https://ntfy.ny4.dev/alert"; + }; + }; + route = { + receiver = "ntfy"; + }; + }; + }; + }; +} diff --git a/nixos/profiles/prometheus/default.nix b/nixos/profiles/prometheus/default.nix new file mode 100644 index 0000000..27f2b34 --- /dev/null +++ b/nixos/profiles/prometheus/default.nix @@ -0,0 +1,10 @@ +{ config, lib, ... }: + +{ + services.prometheus.exporters.node = { + enable = true; + listenAddress = "127.0.0.1"; + port = 9091; + enabledCollectors = [ "systemd" ]; + }; +} diff --git a/nixos/profiles/server/default.nix b/nixos/profiles/server/default.nix index 2af6c22..621dfe1 100644 --- a/nixos/profiles/server/default.nix +++ b/nixos/profiles/server/default.nix @@ -1,5 +1,9 @@ { pkgs, ... }: { + imports = [ + ../prometheus + ]; + environment.systemPackages = with pkgs; [ foot.terminfo ];