Drift/pvv-nixos-config
17
6
Fork 1
Code Issues Pull Requests 6 Actions Wiki Activity

export and scrape dibbler prometheus stats #143

Open
vegardbm wants to merge 1 commits from dibbler-prometheus into main
pull from: dibbler-prometheus
merge into: Drift:main
Conversation 6 Commits 1 Files Changed 4
+101 -3
vegardbm commented 2026-06-07 03:33:26 +02:00
Owner
Copy Link
Copy Source

As it stands it would be pretty difficult to add another database connection.

As it stands it would be pretty difficult to add another database connection.
vegardbm commented 2026-06-07 03:39:46 +02:00
Author
Owner
Copy Link
Copy Source

closes Drift/issues#192

closes Drift/issues#192
vegardbm reviewed 2026-06-07 09:35:19 +02:00
oysteikt reviewed 2026-06-07 10:55:51 +02:00
hosts/ildkule/services/monitoring/prometheus/dibbler.nix Outdated
@@ -0,0 +15,4 @@
jobs.dibbler = {
interval = "1m";
connections = [
"postgres://pvv_vv:WP97&amDp&*gfhg3TyR8@postgres.pvv.ntnu.no"
oysteikt commented 2026-06-07 10:55:51 +02:00
Owner
Copy Link
Copy Source

bro...

bro...
vegardbm commented 2026-06-07 10:57:42 +02:00
Author
Owner
Copy Link
Copy Source

lol, lmao even

lol, lmao even
vegardbm marked this conversation as resolved
vegardbm force-pushed dibbler-prometheus from 13a9af64bc to 864cdab35e 2026-06-07 17:58:42 +02:00 Compare
vegardbm reviewed 2026-06-07 18:03:52 +02:00
hosts/ildkule/services/monitoring/prometheus/dibbler.nix Outdated
@@ -0,0 +48,4 @@
help = "Sum of purchases for the current day.";
labels = [ "thing" ];
values = [ "sum" ];
query = "SELECT SUM(price) FROM purchases GROUP BY DATE(time) ORDER BY DATE(time) DESC LIMIT 1";
vegardbm commented 2026-06-07 18:03:52 +02:00
Author
Owner
Copy Link
Copy Source

This is likely inefficient.

This is likely inefficient.
oysteikt commented 2026-06-08 02:34:02 +02:00
Owner
Copy Link
Copy Source

If we put a database index on time it's probably gonna be less inefficient. How fast does it run with current data?

If we put a database index on `time` it's probably gonna be less inefficient. How fast does it run with current data?
vegardbm commented 2026-06-08 11:13:19 +02:00
Author
Owner
Copy Link
Copy Source
pvv_vv=*# EXPLAIN ANALYZE SELECT SUM(price) FROM purchases GROUP BY DATE(time) ORDER BY DATE(time) DESC LIMIT 1;
                                                             QUERY PLAN                                                             
------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=2359.35..2359.35 rows=1 width=12) (actual time=30.052..30.054 rows=1.00 loops=1)
   Buffers: shared hit=367
   ->  Sort  (cost=2359.35..2501.88 rows=57010 width=12) (actual time=30.051..30.052 rows=1.00 loops=1)
         Sort Key: (date("time")) DESC
         Sort Method: top-N heapsort  Memory: 25kB
         Buffers: shared hit=367
         ->  HashAggregate  (cost=1361.68..2074.30 rows=57010 width=12) (actual time=28.429..29.387 rows=5194.00 loops=1)
               Group Key: date("time")
               Batches: 1  Memory Usage: 1305kB
               Buffers: shared hit=364
               ->  Seq Scan on purchases  (cost=0.00..1076.62 rows=57010 width=8) (actual time=0.034..14.387 rows=57038.00 loops=1)
                     Buffers: shared hit=364
 Planning:
   Buffers: shared hit=51
 Planning Time: 0.349 ms
 Execution Time: 31.113 ms
(16 rows)
```text pvv_vv=*# EXPLAIN ANALYZE SELECT SUM(price) FROM purchases GROUP BY DATE(time) ORDER BY DATE(time) DESC LIMIT 1; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------ Limit (cost=2359.35..2359.35 rows=1 width=12) (actual time=30.052..30.054 rows=1.00 loops=1) Buffers: shared hit=367 -> Sort (cost=2359.35..2501.88 rows=57010 width=12) (actual time=30.051..30.052 rows=1.00 loops=1) Sort Key: (date("time")) DESC Sort Method: top-N heapsort Memory: 25kB Buffers: shared hit=367 -> HashAggregate (cost=1361.68..2074.30 rows=57010 width=12) (actual time=28.429..29.387 rows=5194.00 loops=1) Group Key: date("time") Batches: 1 Memory Usage: 1305kB Buffers: shared hit=364 -> Seq Scan on purchases (cost=0.00..1076.62 rows=57010 width=8) (actual time=0.034..14.387 rows=57038.00 loops=1) Buffers: shared hit=364 Planning: Buffers: shared hit=51 Planning Time: 0.349 ms Execution Time: 31.113 ms (16 rows) ```
vegardbm commented 2026-06-08 11:16:28 +02:00
Author
Owner
Copy Link
Copy Source

There is another one which is also pretty bad compared to the others:

pvv_vv=*# EXPLAIN ANALYZE SELECT SUM(price) FROM purchases;
                                                      QUERY PLAN                                                      
----------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=1076.62..1076.63 rows=1 width=8) (actual time=16.758..16.759 rows=1.00 loops=1)
   Buffers: shared hit=364
   ->  Seq Scan on purchases  (cost=0.00..934.10 rows=57010 width=4) (actual time=0.030..8.101 rows=57038.00 loops=1)
         Buffers: shared hit=364
 Planning Time: 0.042 ms
 Execution Time: 16.784 ms
(6 rows)
There is another one which is also pretty bad compared to the others: ``` pvv_vv=*# EXPLAIN ANALYZE SELECT SUM(price) FROM purchases; QUERY PLAN ---------------------------------------------------------------------------------------------------------------------- Aggregate (cost=1076.62..1076.63 rows=1 width=8) (actual time=16.758..16.759 rows=1.00 loops=1) Buffers: shared hit=364 -> Seq Scan on purchases (cost=0.00..934.10 rows=57010 width=4) (actual time=0.030..8.101 rows=57038.00 loops=1) Buffers: shared hit=364 Planning Time: 0.042 ms Execution Time: 16.784 ms (6 rows) ```
oysteikt commented 2026-06-08 14:01:41 +02:00
Owner
Copy Link
Copy Source

Yeah, but that one is kinda inherent. There's no way to sum all purchases without actually looping over all the purchases and summing them. You can speed either by creating a index on the expression (effectively moving the calculation to being done everytime someone makes a purchase) or by doing more fancy things to only sum news rows onto a previously cached sum.

I wouldn't really worry though, it's still just 16 milliseconds

Yeah, but that one is kinda inherent. There's no way to sum all purchases without actually looping over all the purchases and summing them. You can speed either by creating a index on the expression (effectively moving the calculation to being done everytime someone makes a purchase) or by doing more fancy things to only sum news rows onto a previously cached sum. I wouldn't really worry though, it's still just 16 milliseconds
oysteikt commented 2026-06-08 14:44:25 +02:00
Owner
Copy Link
Copy Source

Here's the relevant statements ftr:

CREATE INDEX "purchases_by_date" ON "purchases"(DATE("date"));
CREATE INDEX "purchases_price_sum" ON "purchases"(SUM("price"));
Here's the relevant statements ftr: ```sql CREATE INDEX "purchases_by_date" ON "purchases"(DATE("date")); CREATE INDEX "purchases_price_sum" ON "purchases"(SUM("price")); ```
vegardbm reviewed 2026-06-07 18:10:38 +02:00
secrets/ildkule/ildkule.yaml Outdated
@@ -1,5 +1,6 @@
config:
mysqld_exporter_password: ENC[AES256_GCM,data:I9K+QMqaN3FOOVKzeOR9Q6UERStXX0P8WEHyN1jzzbM=,iv:UxvIdlfAyJvNuxPkU4+guKPa0fiD0vVLzHOTYktcmso=,tag:ltnIqEwESYx9HBu8UN0ZLw==,type:str]
postgresql_dibbler_password: ENC[AES256_GCM,data:wP4CVz9qRE3CJrblWWYqOIkcH3LM5H81,iv:j8zr1TRNlPPqIppYlWhDoKlL7m2Ph2wQlU6bvFj8R9A=,tag:Cfp8zbYkoLqI7DTDpOBlJw==,type:str]
vegardbm commented 2026-06-07 18:10:38 +02:00
Author
Owner
Copy Link
Copy Source

This duplicates the dibbler postgresql database password.

This duplicates the dibbler postgresql database password.
oysteikt commented 2026-06-08 02:30:24 +02:00
Owner
Copy Link
Copy Source

Probably fine for now, we don't have very good order in our sops files. At some point we should do some spring cleaning and create shared files for shared passwords

Probably fine for now, we don't have very good order in our sops files. At some point we should do some spring cleaning and create shared files for shared passwords
oysteikt reviewed 2026-06-08 03:23:11 +02:00
hosts/ildkule/services/monitoring/prometheus/dibbler.nix Outdated
@@ -0,0 +80,4 @@
LoadCredential = "postgresql_dibbler_password:${
config.sops.secrets."config/postgresql_dibbler_password".path
}";
ExecStartPre = ''/bin/sh -c '${lib.getExe' pkgs.coreutils "cat"} ${configFile} | ${lib.getExe' pkgs.jq "jq"} -c \".jobs[0].connections[0]=\\\"postgres://pvv_vv:$(${lib.getExe' pkgs.coreutils "cat"} %d/postgresql_dibbler_password)@postgres.pvv.ntnu.no\\\"\" > /run/prometheus-sql-exporter/config.yaml' '';
oysteikt commented 2026-06-08 03:23:11 +02:00
Owner
Copy Link
Copy Source

This one is kinda hard to read.

Modern systemd allows you to prepend a | to get rid of the /bin/sh invocation. jq also has --slurpfile and --rawfile to get rid of most of the file trickery. Also, newlines are allowed with a \ just like in ExecStart (alternatively you can construct the args with lib.cli.toCommandLineShellGNU)

ExecStartPre = ''
  |${lib.getExe pkgs.jq} \
    --null-input \
    --compact-output \
    --slurpfile config '${configFile}' \
    --rawfile pw '%d/postgresql_dibbler_password' \
    --from-file '${pkgs.writeText "prometheus-sql-exec-start-jq-filter" ''
      ("postgres://pvv_vv:\($pw | gsub("\n"; ""))@postgres.pvv.ntnu.no") as $pg_uri
      | $config[0]
      | .jobs[0].connections[0] = $pg_uri
    ''}' > /run/prometheus-sql-exporter/config.yaml
'';
This one is kinda hard to read. Modern systemd allows you to prepend a `|` to get rid of the `/bin/sh` invocation. `jq` also has `--slurpfile` and `--rawfile` to get rid of most of the file trickery. Also, newlines are allowed with a `\` just like in `ExecStart` (alternatively you can construct the args with `lib.cli.toCommandLineShellGNU`) ``` ExecStartPre = '' |${lib.getExe pkgs.jq} \ --null-input \ --compact-output \ --slurpfile config '${configFile}' \ --rawfile pw '%d/postgresql_dibbler_password' \ --from-file '${pkgs.writeText "prometheus-sql-exec-start-jq-filter" '' ("postgres://pvv_vv:\($pw | gsub("\n"; ""))@postgres.pvv.ntnu.no") as $pg_uri | $config[0] | .jobs[0].connections[0] = $pg_uri ''}' > /run/prometheus-sql-exporter/config.yaml ''; ```
vegardbm commented 2026-06-08 10:56:00 +02:00
Author
Owner
Copy Link
Copy Source

What you provided works. Modified to use |, --slurpfile and --rawfile.

What you provided works. Modified to use |, --slurpfile and --rawfile.
vegardbm marked this conversation as resolved
vegardbm added 1 commit 2026-06-08 10:54:27 +02:00
prometheus for dibbler
Eval nix flake / evals (pull_request) Successful in 6m14s
Details
Eval nix flake / evals (push) Successful in 6m33s
Details
be87d98060
vegardbm force-pushed dibbler-prometheus from 864cdab35e to be87d98060 2026-06-08 10:54:27 +02:00 Compare
Some checks are pending
Eval nix flake / evals (pull_request) Successful in 6m14s
Details
Eval nix flake / evals (push) Successful in 6m33s
Details
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin dibbler-prometheus:dibbler-prometheus
git checkout dibbler-prometheus
Sign in to join this conversation.
Reviewers
No Reviewers
vegardbm
oysteikt
Labels
Clear labels
art
The issue needs someone to be creative
 backup
big
No, not bug, **big** - this is gonna take a while
 blocked
This issue/PR depends on one or more other issues/PRs
 bug
Something is not working, and it's not the shield hero
 comprehensive first issue
Do this as your first issue, and learn a lot of things
 crash report
Report an oopsie
 disputed
kranglefanter
 documentation
Documentation changes required
 duplicate
This issue or pull request already exists
 enhancement
New feature
 good first issue
Get your hands dirty with a new project here
 logging
networking
TTM4100
 nixos
Requires changes to our nixos setup
 question
More information is needed
 salt
Requires changes to our salt setup
 security
Skommel
 servers n' hardware
Regards hardware, servers or VMs
 simple first issue
Done in nix no time
 wontfix
This won't be fixed
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
adriangl (Adrian Gunnar Lauterer) albertba (Albert Bayazidi) alfhj (Alf Helge Jakobsen) amalieem (Amalie Erdal Mansaker) bjornoka (Bjornar Orjansen Kaarevik) chrisfjo (Christian Fredrik Johnsen) chrivi (Christoffer Viken) danio (Daniel Lovbrotte Olsen) davidk (David Kaasen) dungby (Dung-Bai Yen) eirikwit (Eirik Witterso) felixalb (Felix Albrigtsen) frero (Fredrik Robertsen) jonmro (Jon Martinus Rodtang) jovre (Jo Vassbotn Remvik) karolds (Karoline Dyve Samuelsen) knuta (Knut Auvor Grythe) krisleri (Kristoffer Longva Eriksen) larshhan (Lars Halvor Hansen) olived (Oliver Didriksen-Kolstad) oysteikt (Oystein Kristoffer Tveit) pederbs (Peder Bergebakken Sundt) root (rot) torsteno (Torstein Nordgård-Hansen) vegardbm (Vegard Bieker Matthey) yorinad (Yorin Anne De Jong)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Drift/pvv-nixos-config#143
Delete Branch "dibbler-prometheus"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?