Fix prometheus stats from gluttony

oysteikt commented

2026-01-30 17:27:20 +01:00

Owner

Gluttony is not showing up in the node exporter dashboard on grafana, what's up with that?

oysteikt added the bug nixos networking labels 2026-01-30 17:27:20 +01:00

oysteikt added this to the Kanban project 2026-01-30 17:27:20 +01:00

oysteikt moved this to Medium priority in Kanban on 2026-01-30 17:27:30 +01:00

felixalb commented

2026-02-06 13:54:03 +01:00

Owner

5$ on firewall.

felixalb@ildkule:~/ > curl -I kommode.pvv.ntnu.no:9100/metrics
is OK
felixalb@ildkule:~/ > curl -I gluttony.pvv.ntnu.no:9100/metrics
is left hanging

Probably fixed by creating a security group or modifying an existing one to allow tcp:9100 from ildkule, and adding the SG to gluttony (preferred) or gluttonys port.

I believe we had a PVV_DEFAULT SG in the old stackit region, which had allow rules like

SSH from PVV_v4
SSH from PVV_v6
Ping from everywhere(?)
HTTP(S) on default ports from everywhere
Metrics ports from ildkule

Maybe a willing student can try to (re)implement this security group?

5$ on firewall. ``` felixalb@ildkule:~/ > curl -I kommode.pvv.ntnu.no:9100/metrics is OK felixalb@ildkule:~/ > curl -I gluttony.pvv.ntnu.no:9100/metrics is left hanging ``` Probably fixed by creating a security group or modifying an existing one to allow tcp:9100 from ildkule, and adding the SG to gluttony (preferred) or gluttonys port. I believe we had a PVV_DEFAULT SG in the old stackit region, which had allow rules like - SSH from PVV_v4 - SSH from PVV_v6 - Ping from everywhere(?) - HTTP(S) on default ports from everywhere - [Metrics ports](https://git.pvv.ntnu.no/Drift/pvv-nixos-config/src/branch/main/hosts/ildkule/services/monitoring/prometheus/machines.nix#L9-L11) from ildkule Maybe a willing student can try to (re)implement this security group?

vegardbm commented

2026-02-25 05:27:11 +01:00

Owner

Gluttony seems to struggle with IPv6. The security groups kind of just allow all traffic, so there is no obvious firewall issue.

felixalb commented

2026-02-25 12:29:20 +01:00

Owner

felixalb@gluttony:~/ > ping -c3 2001:700:305:aa07::1
PING 2001:700:305:aa07::1 (2001:700:305:aa07::1) 56 data bytes
64 bytes from 2001:700:305:aa07::1: icmp_seq=1 ttl=64 time=6.06 ms
64 bytes from 2001:700:305:aa07::1: icmp_seq=2 ttl=64 time=2.29 ms
64 bytes from 2001:700:305:aa07::1: icmp_seq=3 ttl=64 time=0.691 ms

--- 2001:700:305:aa07::1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.691/3.012/6.060/2.251 ms
felixalb@gluttony:~/ > ping -c3 -6 login.pvv.ntnu.no
PING login.pvv.ntnu.no (2001:700:300:1900::1:9) 56 data bytes

--- login.pvv.ntnu.no ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2032ms

felixalb@gluttony:~/ > ip -6 r
2001:700:305:aa07::368 dev ens3 proto kernel metric 256 pref medium
fe80::/64 dev ens3 proto kernel metric 256 pref medium
default via fe80::f816:3eff:feb0:543 dev ens3 proto ra metric 1024 expires 217sec hoplimit 64 pref medium

Gluttony seems to be connected to an openstack network with an ipv6 prefix, that does contain a router, that can be pinged from gluttony. However, is this router and/or ipv6 subnet connected to the outside world?

The documentation says

The IPv6 subnet is connected to the world using a router; the same way as IPv4 subnets. When an IPv6 network is connected to a router (which has an external gateway set), the router will make sure to announce the connected IPv6 subnet to the rest of the NTNU Network.

Does the router have an ipv6 external gateway set, or can one be added?

``` felixalb@gluttony:~/ > ping -c3 2001:700:305:aa07::1 PING 2001:700:305:aa07::1 (2001:700:305:aa07::1) 56 data bytes 64 bytes from 2001:700:305:aa07::1: icmp_seq=1 ttl=64 time=6.06 ms 64 bytes from 2001:700:305:aa07::1: icmp_seq=2 ttl=64 time=2.29 ms 64 bytes from 2001:700:305:aa07::1: icmp_seq=3 ttl=64 time=0.691 ms --- 2001:700:305:aa07::1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 0.691/3.012/6.060/2.251 ms felixalb@gluttony:~/ > ping -c3 -6 login.pvv.ntnu.no PING login.pvv.ntnu.no (2001:700:300:1900::1:9) 56 data bytes --- login.pvv.ntnu.no ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2032ms felixalb@gluttony:~/ > ip -6 r 2001:700:305:aa07::368 dev ens3 proto kernel metric 256 pref medium fe80::/64 dev ens3 proto kernel metric 256 pref medium default via fe80::f816:3eff:feb0:543 dev ens3 proto ra metric 1024 expires 217sec hoplimit 64 pref medium ``` Gluttony seems to be connected to an openstack network with an ipv6 prefix, that does contain a router, that can be pinged from gluttony. However, is this router and/or ipv6 subnet connected to the outside world? [The documentation](https://www.ntnu.no/wiki/spaces/skyhigh/pages/112272779/IPv6+tenant+networks#IPv6tenantnetworks-Connectthesubnetworkstotherestoftheworld.) says > The IPv6 subnet is connected to the world using a router; the same way as IPv4 subnets. When an IPv6 network is connected to a router (which has an external gateway set), the router will make sure to announce the connected IPv6 subnet to the rest of the NTNU Network. Does the router have an ipv6 external gateway set, or can one be added?

Fix prometheus stats from gluttony #327