I encourage *nix sysadmins to go to the annual USENIX LISA conference and avail themselves of the training there.

USENIX is the UNIX Users Group. Around since the seventies, it is now a global professional society spanning industry and academia.

The LISA (Large Installation System Administration) conference offers a mix of talks, presentations, expo hall, social activities and formal tutorials (e.g., last year’s program).

I attribute my professional success to being connected with USENIX, to attending every LISA conference I can, getting as much training as I can and to the amazing people I’ve met at LISA, many of whom are now dear friends.

The training is key to broadening my knowledge base, increasing my skill set, and making me more valuable as an individual contributor and executive.

Often the person doing the training literally wrote the book on the subject. I’ve been trained on UNIX system administration by Æleen Frisch, on RRDTool by Tobias Oetiker, on CFEngine by Mark Burgess and so on.

The conference is organized by the community — it is truly a conference by sysadmins, for sysadmins.

I want to thank the staff of USENIX and the community for nurturing this resource and commend them for keeping it relevant. I want to express my heartfelt appreciation to the sysadmin executives of EarthLink in the 1990’s who insisted their staff get trained at LISA to get professional.

See you in Boston, December 4 – 9, 2016!

RPM database corruption is a common problem on Red Hat Linux systems at scale.

When it happens, you have to rebuild the RPM database:
– https://access.redhat.com/solutions/6903
– http://www.cyberciti.biz/tips/rebuilding-corrupted-rpm-database.html

I am working on automating this repair with CFEngine. In the meantime, here is a Custom Report to identify these systems:

-- Aleksey Tsalolikhin, 12 July 2016
-- Show hosts that have broken RPM databases

select hosts.hostname,changetimestamp
from promiseexecutions
inner join hosts on promiseexecutions.hostkey = hosts.hostkey
where logmessages::text ilike '%rpm%db3%' 
and changetimestamp >  current_timestamp - '24 hours'::interval
order by changetimestamp;

The other day, I wanted to find out which RPM metadata field was used to store the “el6” value in the “rpm -q” output for a package, e.g:

[root@as-ws-pr-la-01 Desktop]# rpm -q kernel-2.6.32-504.el6.x86_64
kernel-2.6.32-504.el6.x86_64
[root@as-ws-pr-la-01 Desktop]# 

This came up because this output looks identical for CFEngine’s RHEL 4, 5 and 6 packages, even though there are two different packages involved.

So I used this one-liner to dump all RPM metadata which should be that RELEASE was most likely the proper field to update:

[root@as-ws-pr-la-01 Desktop]# echo  rpm  -q kernel-2.6.32-504.el6.x86_64 --queryformat " $(for f in `rpm --querytags`; do echo \n $f = %{${f}} ' '  ; done  )  " |sh |grep el6
BASENAMES = .vmlinuz-2.6.32-504.el6.x86_64.hmac
 CHANGELOGNAME = Johnny Hughes <johnny@centos.org> [2.6.32-504.el6.centos]
 EVR = 2.6.32-504.el6
 FILENAMES = /boot/.vmlinuz-2.6.32-504.el6.x86_64.hmac
 NEVR = kernel-2.6.32-504.el6
 NEVRA = kernel-2.6.32-504.el6.x86_64
 NVR = kernel-2.6.32-504.el6
 NVRA = kernel-2.6.32-504.el6.x86_64
/sbin/new-kernel-pkg --package kernel --install 2.6.32-504.el6.x86_64 || exit $?
/sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 2.6.32-504.el6.x86_64 NEWKERNARGS || exit $?
/sbin/new-kernel-pkg --package kernel --rpmposttrans 2.6.32-504.el6.x86_64 || exit $?
    /sbin/weak-modules --add-kernel 2.6.32-504.el6.x86_64 || exit $?
PREUN = /sbin/new-kernel-pkg --rminitrd --rmmoddep --remove 2.6.32-504.el6.x86_64 || exit $?
    /sbin/weak-modules --remove-kernel 2.6.32-504.el6.x86_64 || exit $?
PROVIDEVERSION = 2.6.32-504.el6
 R = 504.el6
 RELEASE = 504.el6
 SOURCERPM = kernel-2.6.32-504.el6.src.rpm
[root@as-ws-pr-la-01 Desktop]#

Does anybody know a better way to find out what RPM fields are used to construct the “rpm -q” output?

Mike Weilgart is going to repeat his “Taming the Git Filesystem” talk on June 23rd in Burbank. In the meantime, here is what attendees said about the debut of this talk June 2nd at UUASC-LA/LOPSA-LA meetup:

“You definitely filled out some conceptual holes.”
Stephen Franklin
Systems Engineer

“I liked that it was approachable and assumed no prior knowledge.”
Eric White
Senior System Engineer

“Good foundational talk, Michael! Just as the description says, I’ve just learned enough about git to get my development work done. I occasionally find my ignorance of git’s foundation comes back to bite me in the rear. With this talk I can hope that happens less often.”
George Wu
Vice President of Engineering

“I was able to understand the explanations even as a student with limited technical knowledge.”
Eric, Community College Student

“This talk has something to offer for most git users. It’s not a run of the mill ‘how to use’ git presentation.”
Jordan Schwartz
Systems / Storage Engineer

These are some working notes for identifying critical unpatched vulnerabilities on a Red Hat Enterprise Linux system (version 6).

If you install yum-security plugin, you can list security updates available and which CVEs they relate to, as well as their severity according to Red Hat ratings system:

yum update-info list cves available

Identifying which unpatched CVEs (as returned by the yum-security plugin) are Critical according to CVSS (Common Vulnerability Scoring System), with score > 7:

Scored CVEs are available from National Vulnerability Database through a set of XML feeds. The NIST web site says:

A common way to use the feeds is to perform a one-time import of all of the
main XML vulnerability feeds and then use the “modified” feeds to keep
up-to-date.

The “xml2” package converts from XML to various formats. I started by converting the 2015 XML to CSV:

$ xml2 < nvdcve-2.0-2015.xml > nvdcve-2.0-2015.flat
$ 2csv entry vuln:cve-id vuln:cvss/cvss:base_metrics/cvss:score vuln:summary < nvdcve-2.0-2015.flat > nvdcve-2.0-2015.csv
$

Next step: lookup the CVSS scores for CVEs returned by yum-security, is left as an exercise for the reader.

Notes:
– xml2 home
– “xml2” is available through sudo apt install xml2 on Ubuntu