So we finally wrapped up the big migration of bugzilla.redhat.com yesterday morning. We moved from an internal system (1 web, 1 db w hot spare) to one in the datacenter (multiple web and db) plus some basic code fixes. It was an epic effort with large amounts of data having to be copied over relatively slow links and being converted from Postgresql to MySQL. We made the decision to move to MySQL for a handful of reasons, primary being real replication. Slony is a mostly adequate option for Postgresql, however it’s performance isn’t great and it’s got some pretty large limitations since it’s all triggers and stored procedures. Now we have a system that’s much more scalable and should have a noticeable improvement in performance.
We had a few minor issues during the migration, but data is intact and it was a largely uneventful process. It took a total of ~15 hrs to complete which completely sucked, but was pretty much unavoidable. Apparently rsync got confused with some of our converted dump files and decided they had to be copied whole. Best guess was that it didn’t care for the format of some of the converted dump files, but oh well….
If you notice any issues with the new system, please drop an email to bugzilla-owner.
Hurrah for proper infra!
Been trying to track down some issues with the new Subversion servers at work. They are setup to be accessed via https only, authentication provided by mod_auth_kerb. This works beautifully for doing Negotiate auth, however I’d been seeing some odd errors when it would fall back to Basic. I was seeing a bunch of replay errors logged:
krb5_verify_init_creds() failed: Request is a replay
Along with the more cryptic:
The locks could not be queried for verification against a possible "If:" header. [500, #0]
It was pretty obvious that the errors were related, however the replay message were spewing much more frequently. I enlisted the help of jorton (resident Apache guru) and we managed to fix a few other small annoyances, but got nowhere on the errors. Till yesterday. Joe forwarded me an email from the m_a_k mailing list that said to set kdc_timesync = 0. The thread has a good explanation of why this needs to be set, so go read it. 
Brew/Koji’s db is well over 100G, Bugzilla is ~11G so your puny 8.5G RT database fails to impress.
BTW, for the record, Request Tracker is a slow and confusing beast. I’d _really_ like to put it out of my misery.
Been playing with the new virt layer in RHEL5 and having a blast. I did have to make some changes to get NAT to work with guest domains. Nothing huge, just created a dummy network device and bridged to that instead of the normal eth0 as such -
/etc/sysconfig/network-scripts/ifcfg-dummy0:
# Dummy interface for Xen
DEVICE=dummy0
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
IPV6INIT=no
PEERDNS=yes
TYPE=Ethernet
NETMASK=255.255.255.0
IPADDR=x.x.x.x
ARP=yes
/etc/xen/xend-config.sxp – changed (network-script network-bridge) to (network-script 'network-bridge netdev=dummy0')
I then enabled NAT for the interfaces -
/etc/sysctl.conf – changed net.ipv4.ip_forward = 1 to net.ipv4.ip_forward = 1 and ran sysctl -p to make the change live
Last step was doing the needful iptables setup. Running iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE will setup the basics, you may also have to add a rule to allow the traffic on the dummy interface iptables -I INPUT -i dummy+ -j ACCEPT. The ‘-I’ is important with the default RHEL firewall as it is evaluated before the other rules.
Been a while since I last posted. Tho not from a lack of desire or subject matter, simply a lack of time.
Quite a bit has happened since my last post. The Girls have been growing like crazy and being quite the handful as of late.
Britton’s gaining knowledge (and hopefully some wisdom) at a scary rate. I only wish I still picked things up as easily as she does… She’s also progressing very nicely with her gymnastics and ballet, tho I’m prolly not the most objective judge of such things. She’s also becoming very independent and headstrong. Very independent and headstrong.
Blythe officially hit the ‘terrible twos’ last month. She started with the ‘no!’s and ‘mine!’ about a month before that. But she’s cute and, even at her age, knows full well how to use that to manipulate those around her. And she’s also starting to manifest a pretty hefty temper when she doesn’t get her way.
But they are both good kids, and generally fun to be around.
Work’s been quite hectic as of late, what with the RHEL 5 release and all. As an aside, RHEL 5 is really good. Been playing with it over the past few weeks and have been hard pressed to find faults on the server side.
It’s also been the fiscal year transition so I’ve been bogged down with budgetary and infrastructure planning. Small tasks such as defining policy & procedures for things like disaster recovery, incident response and general system maintenance are only part of my task list for the next year. It’s amazing how much more complex a system gets when it’s global. 
Oh yea, I did manage to get out to my first try at some paintball several weeks ago. We went with a friend of Tiff’s who’s hubby is pretty hardcore. Didn’t manage to get Tiff to try it out, but it’s only a matter of time…
And we got snow here. Twice.
