OpenSUSE Tumbleweed and rpmdb error

The Problem

So yeah, during the last sudo zypper dup I saw some weird rpm error (each <num> was actually a different positive integer):

error: rpmdbNextIterator: skipping h#    <num> blob size(-<num>): BAD, <num> + <num> * il(<num>) + dl(-<num>)

As it turned out, this probably means corrupted rpm database (rpm db). Well, no luck this time…

Both sudo zypper ve and sudo rpm -qa did not show anything alarming, though. But the error was logged.

At first I thought that snapper will help. You know, if root partition is snapshotted, then perhaps rpm db is as well to reflect root contents. No luck again… rpm database is kept on /var subvolume and thus is not present on BTRFS snapshots.

The Solution

Searching the Internet pointed me to a simple one: rpm --rebuilddb. This command should fix the database as long as Packages.db is not damaged. As far as I understand, of course.

Alright then, let’s do it! One run for computer:

sudo rpm --rebuilddb

and one for me (with verbose output):

sudo rpm --rebuilddb -vv

Phew, that was easy.

The PEBKAC

If only I had read first that rebuilddb can break the database itself…

rpm -qa will show me all installed packages, at least according to fixed rpm db. But what do I compare it to? Yep, the results of rpm -qa run before rebuilding the database!

Not a problem, since the system stores rpm db backups in /var/adm/backup/rpmdb, right?

Wrong! For some reason, my Tumbleweed stopped making these correctly somewhere around last week of March. Nooooooooooooooo!!!

Well, at least I filed a bug report on OpenSUSE Bugzilla.

The Final Solution

My first thought: reinstall the system just to be sure. But it took me so much time to configure it “perfectly”. How can I be sure that the db is still fine?

I searched the Internet again, this time for an answer on how to fix broken rpm db after losing it. The suggestion was to dig pacmage manager logs to recreate list of installed packages. Then I could reinstall those to repopulate the db, but I didn’t actually lose it, so this step was unnecessary for me. Yeah, this might be it!

/var/log/zypp/history contains everything I need, including packages installed during initial system installation!

rpm -qa outputs installed packages in the form of name-version.architecture.

zypp history, on the other hand, has date time|action|name|version|architecture||repository|checksum| as well as other informations.

Back it up for once, please!

So this time I decided to first copy the history file not to repeat the mistake of being to hasty, like with running rpm --rebuilddb first and ask more questions later.

Extract installed packages

grep '|install|' history

Good, this gave me list of installed packages only.

Take only what’s needed

cut -d '|' -f 3-5

Resulted with the same list, but only containing name|version|architecture in each line.

Adjust format to that of RPM

sed -r 's;(.+)\|(.+)\|(.+);\1-\2.\3;g'

This changed each line from the aforementioned format to name-version.architecture, like in rpm -qa.

Duplicates are not cool

1
2
3
4
5
6
7
grep '|install|' history \
| cut -d '|' -f 3-5 \
| sed -r 's;(.+)\|(.+)\|(.+);\1-\2.\3;g' \
| sort \
> sorted_history

rpm -qa | sort > sorted_qa

Now all I needed was to compare these files and see if everything sorted_history has is present in sorted_qa. But sorted_history has duplicated package names just with different versions! That’s expected, as some packages were installed multiple times because of distribution upgrades (this is how Tumbleweed should be updated).

Grande Finale

Ok, so let’s update the last ones slightly:

1
2
3
4
5
6
7
8
9
grep '|install|' history \
| cut -d '|' -f 3-5 \
| tac \
| sort -rut '|' -k 1,1 -k 2,1 \
| sort \
| sed -r 's;(.+)\|(.+)\|(.+);\1-\2.\3;g' \
> sorted_history

rpm -qa | sort > sorted_qa

So, now graphical diff should be pretty manageable. There will be some false positives still:

  • sorted_history may contain packages removed during upgrades or manually, but these should be easy to find using grep '|removed|' history;
  • sorted_history lacks duplicates even for packages that are installed with concurrent versions (eg. latest and previous kernel-default);
  • sorted_qa contains gpg-pubkey-* packages, but these were easy to find on openSUSE-build-key page and in openSUSE-build-key.spec .

The only package that “magically disappeared” was libndr0. I could not find final removed row in sorted_history with it, but:

  • there’s libndr1 present, so perhaps it was upgraded to newer version;

  • there’s no such package in OpenSUSE Factory , so indeed it looked like it was upgraded;

  • the samba.spec contains:

    553
    554
    555
    556
    557
    558
    
    %package -n libndr1
    Summary:        Network Data Representation library
    License:        GPL-3.0-or-later
    Group:          System/Libraries
    Provides:       libndr0
    Obsoletes:      libndr0
    

    - bingo, Provides says it all!

The rpm database seems to have kept its state! :)

A Lesson Learned

Make sure backups are actually there first.
Always back important files up before investigation, even those that appear to be broken.
Read the docs thorougly.
Resist the urge to reinstall the system anyways. ;)