CFEngine Enterprise tip: showing hosts that have a broken RPM database

RPM database corruption is a common problem on Red Hat Linux systems at scale.

When it happens, you have to rebuild the RPM database:
https://access.redhat.com/solutions/6903
http://www.cyberciti.biz/tips/rebuilding-corrupted-rpm-database.html

I am working on automating this repair with CFEngine. In the meantime, here is a Custom Report to identify these systems:

-- Aleksey Tsalolikhin, 12 July 2016
-- Show hosts that have broken RPM databases

select hosts.hostname,changetimestamp
from promiseexecutions
inner join hosts on promiseexecutions.hostkey = hosts.hostkey
where logmessages::text ilike '%rpm%db3%' 
and changetimestamp >  current_timestamp - '24 hours'::interval
order by changetimestamp;

Dumping RPM metadata

The other day, I wanted to find out which RPM metadata field was used to store the “el6” value in the “rpm -q” output for a package, e.g:

[root@as-ws-pr-la-01 Desktop]# rpm -q kernel-2.6.32-504.el6.x86_64
kernel-2.6.32-504.el6.x86_64
[root@as-ws-pr-la-01 Desktop]# 

This came up because this output looks identical for CFEngine’s RHEL 4, 5 and 6 packages, even though there are two different packages involved.

So I used this one-liner to dump all RPM metadata which should be that RELEASE was most likely the proper field to update:

[root@as-ws-pr-la-01 Desktop]# echo  rpm  -q kernel-2.6.32-504.el6.x86_64 --queryformat " $(for f in `rpm --querytags`; do echo \n $f = %{${f}} ' '  ; done  )  " |sh |grep el6
BASENAMES = .vmlinuz-2.6.32-504.el6.x86_64.hmac
 CHANGELOGNAME = Johnny Hughes <johnny@centos.org> [2.6.32-504.el6.centos]
 EVR = 2.6.32-504.el6
 FILENAMES = /boot/.vmlinuz-2.6.32-504.el6.x86_64.hmac
 NEVR = kernel-2.6.32-504.el6
 NEVRA = kernel-2.6.32-504.el6.x86_64
 NVR = kernel-2.6.32-504.el6
 NVRA = kernel-2.6.32-504.el6.x86_64
/sbin/new-kernel-pkg --package kernel --install 2.6.32-504.el6.x86_64 || exit $?
/sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 2.6.32-504.el6.x86_64 NEWKERNARGS || exit $?
/sbin/new-kernel-pkg --package kernel --rpmposttrans 2.6.32-504.el6.x86_64 || exit $?
    /sbin/weak-modules --add-kernel 2.6.32-504.el6.x86_64 || exit $?
PREUN = /sbin/new-kernel-pkg --rminitrd --rmmoddep --remove 2.6.32-504.el6.x86_64 || exit $?
    /sbin/weak-modules --remove-kernel 2.6.32-504.el6.x86_64 || exit $?
PROVIDEVERSION = 2.6.32-504.el6
 R = 504.el6
 RELEASE = 504.el6
 SOURCERPM = kernel-2.6.32-504.el6.src.rpm
[root@as-ws-pr-la-01 Desktop]#

Does anybody know a better way to find out what RPM fields are used to construct the “rpm -q” output?

Feedback from “Taming the Git Filesystem” talk on June 2nd, 2016

Mike Weilgart is going to repeat his “Taming the Git Filesystem” talk on June 23rd in Burbank. In the meantime, here is what attendees said about the debut of this talk June 2nd at UUASC-LA/LOPSA-LA meetup:

“You definitely filled out some conceptual holes.”
Stephen Franklin
Systems Engineer

“I liked that it was approachable and assumed no prior knowledge.”
Eric White
Senior System Engineer

“Good foundational talk, Michael! Just as the description says, I’ve just learned enough about git to get my development work done. I occasionally find my ignorance of git’s foundation comes back to bite me in the rear. With this talk I can hope that happens less often.”
George Wu
Vice President of Engineering

“I was able to understand the explanations even as a student with limited technical knowledge.”
Eric, Community College Student

“This talk has something to offer for most git users. It’s not a run of the mill ‘how to use’ git presentation.”
Jordan Schwartz
Systems / Storage Engineer

Identifying critical unpatched vulnerabilities on a Red Hat system

These are some working notes for identifying critical unpatched vulnerabilities on a Red Hat Enterprise Linux system (version 6).

If you install yum-security plugin, you can list security updates available and which CVEs they relate to, as well as their severity according to Red Hat ratings system:

yum update-info list cves available

Identifying which unpatched CVEs (as returned by the yum-security plugin) are Critical according to CVSS (Common Vulnerability Scoring System), with score > 7:

Scored CVEs are available from National Vulnerability Database through a set of XML feeds. The NIST web site says:

A common way to use the feeds is to perform a one-time import of all of the
main XML vulnerability feeds and then use the “modified” feeds to keep
up-to-date.

The “xml2” package converts from XML to various formats. I started by converting the 2015 XML to CSV:

$ xml2 < nvdcve-2.0-2015.xml > nvdcve-2.0-2015.flat
$ 2csv entry vuln:cve-id vuln:cvss/cvss:base_metrics/cvss:score vuln:summary < nvdcve-2.0-2015.flat > nvdcve-2.0-2015.csv
$

Next step: lookup the CVSS scores for CVEs returned by yum-security, is left as an exercise for the reader.

Notes:
xml2 home
– “xml2” is available through sudo apt install xml2 on Ubuntu

Mario Obejas on “a replacement for bash?” and on writing production-grade code in any language.

Recently Yves Dorfsman asked on the lopsa-tech mailing list:

A lot of people love to hate bash, and there are good reasons for it, but it
seems that there isn’t an obvious replacement for it.

[…]
What do you use? Do you see any clear winner to replace it on the horizon?

Mario Obejas, a popular occasional Instructor for Vertical Sysadmin, and living proof that there is life after system administration, answered with this gem. This is the voice of 34 years of Software Development, Infrastructure, and Information Security experience, folks. (Thanks for allowing us to re-post it here, Mario!)

bash replaced sh the same way vim replaced vi, for many of the same reasons. They took what was working, kept the solid parts of the base and improved it. The jury is back – bash works well enough for a lot of production work.

My prediction, Yves, is that the only thing that will replace vim and bash to the same degree will similarly be another iteration that improves upon the already existing widely installed vim and bash base, and leverages the same widely held skill sets.

Some things, like a Ted Cruz, no language can deal with. The last production bash scripts I wrote were to provide data traffic management between the VIIRS satellite ground station and NASA’s net, through their itty bitty data pipe. Our requirements were to be able to recover from up to a catastrophic three day data clog or outage, without losing any of the satellite data. We did very well through 2013 until Ted affected us. Nobody expected a 17 day US government shutdown …..

I’m an odd data point with respect to languages. I loved PASCAL and ADA. But here’s some heresy – I hate C. That’s probably due to the way a lot of people write C, which I characterize as, “as obfuscated and comment free as possible”. I think most people write C with speed in mind (filling in the diamond or checkbox on the schedule) versus maintainability (write like you will not see the code for a year, and then you will have to come back and upgrade/extend it).

We wrote Theater High Altitude Air Defense in Ada. When the day came to integrate our code (including that of our partners and contract vendors) in the lab, we were all shocked to find the system cycling, acquiring and tracking targets, not crashing as expected. We had to go back to our offices and retrieve the formal tests, nobody expected that level of robustness the first week, much less the first 15 minutes.

For longevity, my oldest production code is the Jovial and assembly I wrote for three Middle East mountaintop radars. I participated in the customer selloff in 1984, and 29 years later I received a call in December 2013 – no kidding – about helping the team to do an upgrade in 2014. It was still working(!). The only other longer lived code I know about is off world (satellites) or in the “Don’t touch it!” holy COBOL shrines.

Tangential moral of the story: regardless of language, put some damn comments in the code, like you are talking to somebody years later and explaining what the code does. That somebody might be you, years later.