When you zoom in a bit, do you see short bursts of CPU usage? Typically, things should be a bit hectic for a minute or two and then calm for ten minutes. Given that each validation run includes validating signatures on around 100,000 objects, quite a bit of CPU usage is to be expected.

Similarly, these 100,000 objects – each is its own file –, are read every 10 minutes explaining the disk usage. I’d expect them to be mostly cached if the machine has enough memory, but if it is a VM on a busy host, ten minutes may be too long for them to stay in the cache.

Here's the closest zoom I can get, last 24 hours:

I have an operational issue with the RAM usage of the last release (0.9.0), it jumped from some megs to more than 1G. It’s OOM-killed by the kernel every now and them. As a quick fix, I’m back on 0.8.3.
The upgrade has been done at the end of week 22.

Thanks for the report and the graphs! While we expected higher RAM usage due to the new database in 0.9, it certainly is too much now and consumption also seems to be growing over time. We are investigating both right now and hopefully will have a fix soon.

Please note that RAM usage in 0.10.0 is now significantly lower than in 0.9.0:

$ sudo systemctl status routinator
● routinator.service - Routinator 3000
   Loaded: loaded (/lib/systemd/system/routinator.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2021-08-23 12:22:18 UTC; 1 weeks 3 days ago
     Docs: man:routinator(1)
 Main PID: 7389 (routinator)
    Tasks: 5 (limit: 2330)
   Memory: 1.4G
   CGroup: /system.slice/routinator.service
           └─7389 /usr/bin/routinator --config=/etc/routinator/routinator.conf --syslog server

$ cat /proc/7389/status | grep 'VmHWM\|VmRSS'
VmHWM:	  428368 kB
VmRSS:	  369752 kB

Yeah, I have upgraded last week and I confirm. Thanks a lot for the work done. It even seems to be lower than 0.8.0 ;)

I am running the container version 0.12.0 in Nomad, and the CPU is spiking to unthinkable levels: 1500% (thousand and 5 hundred percent):

or to 30 Ghz (thirty Ghz):

It is a bit odd that this happens on every third validation run (assuming you’ve kept the refresh time at ten minutes), but otherwise not entirely unexpected if you have a lot of cores. Routinator uses a thread pool during validation that is by default configured to be the number of cores. Each thread processes one repository including updating the repository. If there is nothing to update and you have most of the files buffered (if you have enough memory), you can end up with all threads basically just validating signatures and using a lot of CPU. This would just be a short spike since most repositories are quite small and eventually there will be only two or three working threads left.

So, while seeing this every third run is certainly strange, I think this is normal. You can, however, limit the amount of threads (and thus cores used) via the validation-threads configuration option.