The Problem Link to heading

So yeah, during the last sudo zypper dup I saw some weird rpm error (each <num> was actually a different positive integer):

error: rpmdbNextIterator: skipping h#    <num> blob size(-<num>): BAD, <num> + <num> * il(<num>) + dl(-<num>)

As it turned out, this probably means corrupted rpm database (rpm db). Well, no luck this time…

Both sudo zypper ve and sudo rpm -qa did not show anything alarming, though. But the error was logged.

At first, I thought that snapper will help. You know, if root partition has snapshots, then perhaps rpm db has them too, to reflect root partition’s contents. No luck again… rpm database is kept on /var subvolume and thus is not present on BTRFS snapshots.

The “Solution” Link to heading

Searching the Internet pointed me to a simple one: rpm --rebuilddb. This command should fix the database as long as Packages.db is not damaged. As far as I understand, of course.

Alright then, let’s do it! One run for computer:

sudo rpm --rebuilddb

and one for me (with verbose output):

sudo rpm --rebuilddb -vv

Phew, that was easy.

The PEBKAC Link to heading

If only I had read first that rebuilddb itself can break the database too…

rpm -qa will show me all installed packages, at least according to fixed rpm db. But what do I compare it to? Yep, the results of rpm -qa run before rebuilding the database!

Not a problem, since the system stores rpm db backups in /var/adm/backup/rpmdb, right?

Wrong! For some reason, my Tumbleweed stopped making these correctly somewhere around last week of March. Nooooooooooooooo!!!

Well, at least I filed a bug report on OpenSUSE Bugzilla.

The Final Solution Link to heading

My first thought: reinstall the system just to be sure. But it took me so much time to configure it “perfectly”. How can I be sure that the db is still fine?

I searched the Internet again, this time for an answer on how to fix broken rpm db after losing it. The suggestion was to dig package manager logs to recreate list of installed packages. Then I could reinstall those to repopulate the db, but I didn’t actually lose it, so this step was unnecessary for me. Yeah, this might be it!

/var/log/zypp/history contains everything I need, including packages installed during initial system installation!

rpm -qa outputs installed packages in the form of name-version.architecture.

zypp history, on the other hand, has date time|action|name|version|architecture||repository|checksum| as well as other informations.

Back it up for once, please! Link to heading

So this time I decided to first copy the history file not to repeat the mistake of being to hasty, like with running rpm --rebuilddb first and ask more questions later.

Extract installed packages Link to heading

grep '|install|' history

Good, this gave me list of installed packages only.

Take only what’s needed Link to heading

cut -d '|' -f 3-5

Resulted with the same list, but only containing name|version|architecture in each line.

Adjust format to that of RPM Link to heading

sed -r 's;(.+)\|(.+)\|(.+);\1-\2.\3;g'

This changed each line from the aforementioned format to name-version.architecture, like in rpm -qa.

Duplicates are not cool Link to heading

1
2
3
4
5
6
7

grep '|install|' history \
| cut -d '|' -f 3-5 \
| sed -r 's;(.+)\|(.+)\|(.+);\1-\2.\3;g' \
| sort \
> sorted_history

rpm -qa | sort > sorted_qa

Now all I needed was to compare these files and see if everything sorted_history has is present in sorted_qa. But sorted_history has duplicated package names just with different versions! That’s expected, as some packages were installed multiple times because of distribution upgrades (this is how Tumbleweed should be updated).

Grande Finale Link to heading

Ok, so let’s update the last ones slightly:

1
2
3
4
5
6
7
8
9

grep '|install|' history \
| cut -d '|' -f 3-5 \
| tac \
| sort -rut '|' -k 1,1 -k 2,1 \
| sort \
| sed -r 's;(.+)\|(.+)\|(.+);\1-\2.\3;g' \
> sorted_history

rpm -qa | sort > sorted_qa

So, now graphical diff should be pretty manageable. There will be some false positives still:

  • sorted_history may contain packages removed during upgrades or manually, but these should be easy to find using grep '|removed|' history;
  • sorted_history lacks duplicates even for packages that are installed with concurrent versions (eg. latest and previous kernel-default);
  • sorted_qa contains gpg-pubkey-* packages, but these were easy to find on openSUSE-build-key page and in openSUSE-build-key.spec .

The only package that “magically disappeared” was libndr0. I could not find final removed row in sorted_history with it, but:

  • there’s libndr1 present, so perhaps it was upgraded to newer version;

  • there’s no such package in OpenSUSE Factory , so indeed it looked like it was upgraded;

  • the samba.spec contains:

    553
554
555
556
557
558

    %package -n libndr1
Summary:        Network Data Representation library
License:        GPL-3.0-or-later
Group:          System/Libraries
Provides:       libndr0
Obsoletes:      libndr0

    - bingo, Provides says it all!

The rpm database seems to have kept its state! :)

A Lesson Learned Link to heading

Make sure backups are actually there first.
Always back important files up before investigation, even those that appear to be broken.
Read the docs thorougly.
Resist the urge to reinstall the system anyways. ;)