This post from Modwest Mage (aka Systems Engineer) Tom:
Once upon a time, we wrote a yarn of PHP Versions and I/O woe. Mired in a battle of attrition, we enlisted a tough, gritty, mysterious stranger to bolster the ranks of our demoralized filesystem army. However, even a legendary hero could not rescue us from inevitable performance peril, and our wizards have been furtively developing new majicks to provide salvation for us and our citizens. As one of these wizards, I have my own chronicle to relate, although it lacks the glory of front-line battle.
My most recent task was to freshen our shared system webservers with the newest release of their operating system. By doing so, I would be opening the portal that would, in the near future, allow our webserver software to communicate with our new filesystem architecture. Unexpected foes riddled the veins of the project, and the task became a trial that spanned many days. The challenges tore at my sanity; my therapy is to share the hurdles with you.
The most daunting problem was rooted in Linux's "Lazy Dynamic Linking." While I am not sure I even completely understand -- it is magic, after all -- I will do my best to paraphrase the situation and solution.
First, for perspective, let me describe how our webservers handle CGI scripts. A request for a CGI triggers our modified sbox. Sbox chroots the script inside the shared system environment, and then runs the script as the owner of the script, not as the webserver user. We call this process "jailing".
My upgrade plan included no changes to jailing. The only thing that was to change was the underlying operating system. So, after rebuilding a webserver from scratch, I was surprised to see this error when attempting to run any CGI script:
/usr/lib/apache/suexec: relocation error: /lib/libnss_compat.so.2: symbol _nss_files_parse_grent, version GLIBC_2.0 not defined in file libc.so.6 with link time reference.
While the error was clue enough to hint at C, I am not much of a C programmer. I was beginning to get a bad feeling about this. Like any good apprentice, hoping it was common lore, I asked the arcane compendium seer for knowledge. Alas, information was sparse, and relevant tomes did not exist. I was on my own.
After days entombed in my computing cave, I had gathered shockingly little helpful data in isolating the problem.
But then, a flicker of hope sparkled from a configuration in nscd. With passwd caching off, CGI scripts began to run just fine, without explanation!
However, as any Linux hedge wizard knows, nscd yields massive performance gains on name lookups, and webservers like ours do these lookups constantly. Without solving this problem, the webservers would be too lethargic to work as a foundation for our project. Growing weak from monitor radiation and frustration, the situation looked grim to me.
I was certain I had failed king and country. Desperate, we summoned a magician from another fiefdom. One afternoon later, filled with gdb, strace, and ltrace, his expertise finally revealed the true nature of our error:
warning: .dynamic section for "/lib/libnss_compat.so.2" is not at the expected address
Error while mapping shared library sections:
/lib/i686/mmx/libnsl.so.1: No such file or directory.
Error while reading shared library symbols:
/lib/i686/mmx/libnsl.so.1: No such file or directory.
Error while reading shared library symbols:
/lib/i686/mmx/libnsl.so.1: No such file or directory.
By using a clever technique of adding a sleep statement in the C code to provide time to bind to the process with the debugging programs, he finally received the output above and instantly knew the problem.
Lazy Linking normally maps library functions in the Procedure Linkage Table, but doesn't actually load the functions. The functions only load when called. This is fine for normal environments, but for this instance, ld.so was trying to load the functions while in the jail, but the mapping had been done outside of the jail. In other words, after deciding that we would use the hardware libraries, we then later tried to use the jail libraries instead. Or, as the computer would say, 'dynamic section for "/lib/libnss_compat.so.2" is not at the expected address.'
The key to modifying this behavior lies in a simple environment variable.
Running with "LD_BIND_NOW = 1" solved all of the library problems I had seen. The variable tells ld.so to map and load the functions at runtime. We lose a little bit of efficiency, because the linker may be loading functions the program doesn't actually use, but it guarantees we use the libraries we think we are using.
Another ritual complete, but does it mean we're any closer to smashing the stalemate?
Websites on our shared hosting cluster aren't affected by this change, aside from any miscast spells on my part. The change does mean that we can, within a few weeks, deploy our new Apache stack that talks to the new filesystem. Assuming that deployment goes well, the beta testing that we've hinted about will start to become a reality a few weeks later.
Soon after, I hope we may close the chapter on this epic battle and enter an age of renaissance.
-TRC
Subscribe to this blog's feed