LOPSA-NJ News Aggregator
NYC-area Sysadmins: free beer!
Etsy Now Sponsoring SysAdmin Appreciation Day Event in NYC!
Last night, I got an email out of the blue. It was from Chris Munns, sysadmin at Etsy, the home to a huge online community of people who make and sell things. The email basically asked if there was any way that Etsy could help sponsor the SysAdmin Appreciation Day event! Excellent.
The only question in my mind was, what kind of sponsorship would our event need? In the end, it winds up being a bunch of system administrators sitting around drinking, swapping war stories. I told them as much, and Chris responded:
Hey Matt,
Chad Dickerson who is the CTO here at Etsy was actually the one who wanted to us to help sponsor/participate. We were wondering if maybe we could just throw some money in for drinks on behalf of Etsy?
- Chris
Pick up some of the bar tab? Well, ok!
After some more discussion, we’ve got it settled down, and I am happy to say that Etsy is contributing a very significant amount towards our bar tab tomorrow. I’m not going to say how much just yet, because I haven’t worked out how it’s going to be handled, but I’ll be surprised if anyone ends up paying for a drink themselves.
A huge(!) thank you to Etsy! And if you’re wondering why a site largely dedicated to crafting cares this much about the community of System Administrators, you should read their blog, Code As Craft. They believe strongly in Dev/Ops cooperation, and they spend a lot of time on that blog discussing their infrastructure. If you’re interested in Hadoop installations and continuous deployments, I recommend you check it out.
If you were holding back because you didn’t want to spend the dough on drinks, then don’t be afraid any longer. Check out the event page, then register!
Event registration for SysAdmin Appreciation Day – NYC powered by EventbriteBook Idea: Distributed Computing Economics... EXPLAINED
Lists of length 1, 2, 3 and 4 have special meaning
Certification? Not yet.
If the sysadmin of the year is for good work…
…do we have an appropriate award for doing bad work?
I’m only asking, because today on reddit, I came across an amazing post.
There is a subreddit called IAMA, where you can submit a thread allowing people to ask you questions because you are, in some way, unusual or interesting. The thread I found was called “IAMA Wildly Incompetent Network Security Admin and have no business in my job“.
The job? He’s network security for a Vegas casino.
When you actually click on the thread, it gets way, way worse. There’s a summary at the top, so I’m stealing some and pasting here. This is all copyright of reddit user throwawayscared, I don’t want it.
Since alot of people are asking this question: The reason I dont spend time learning the job is partly due to laziness. I mean it’s awesome spending all day playing battlefieldheroes or transformice.
I refuse to wear my ID badge so people dont stop and ask me questions. I’ve been reprimanded and even warranted the CEO sending out a memo that stated ‘EVERYONE HAS TO WEAR THEIR BADGE’ and I still dont do it. I just changed my schedule to leave earlier than any execs and get in after they do so they never see me without it.
also working at a casino means you get free lunches too. we’re only supposed to eat once, but i go several times throughout the day. I once changed the settings on the turnstyle applicatoin to allow me unlimited cafeteria entries. Everyone else was set at 1. The benefits of admin passwords
To further prove how much I should be fired, I’d like to share a quick story with you. I have stolen every bit of computer shit I can get my hands on. When the security team started cracking down on thieving employees and searching us on the way out, I just started mailing the shit to my house through the mailroom. Then I just started listing shit on ebay and sending it to the buyers right through the same mailroom. I also convinced the mailroom dude that I should’t pay for postage. I’m not proud, but I’m certainly not ashamed.
wow. It’s like a trainwreck.
Please, don’t be this guy.
Yubikeys Have Arrived
In my previous post about the Yubikey I suggested that computer users’ groups should arrange bulk purchases to get the best prices [1]. I ran such a buying club for Linux users in Australia as well as members of SAGE-AU [2].
The keys have arrived and I now have to start posting them out. Above is a picture of two boxes that each contain 100 keys. Presumably if you buy a smaller number of keys then you get more fancy packing.
Thanks to Yubico for giving us a greater discount than the usual discount rate for boxes of 100 keys!
Creating a SE Linux Chroot environment
A large part of the use of chroot environments is for the purpose of security, it used to be the only way of isolating a user from a section of the files on a server. In many of the cases where a chroot used to be used for security it is now common practice to use a virtual server. Also another thing to note is that SE Linux provides greater access restrictions to most daemons than a chroot environment would so in many case using SE Linux with a sensible policy is a better option than using a chroot environment to restrict a daemon. So it seems to me that the security benefits that can be obtained by using a chroot environment have been dramatically decreased over the last 5+ years.
One significant benefit of a chroot environment is that of running multiple different versions of software on one system. If for example you have several daemons that won’t run correctly on the same distribution and if you don’t want to have separate virtual machines (either because you don’t run a virtualisation technology or because the resources/expense of having multiple virtual servers is unacceptable) then running multiple chroot environments is a reasonable option.
The Simplest SolutionThe simplest case is when all the chroot environments are equally trusted, that means among many other things that they all have the latest security patches applied. Then you can run them all with the same labels, so every file in the chroot environment will have the same label as it’s counterpart in the real root – this will mean that for example a user from the real root could run /chroot/bin/passwd and possibly get results you don’t desire. But it’s generally regarded that the correct thing to do is to have a chroot environment on a filesystem that’s mounted nosuid which will deal with most instances of such problems. One thing to note however is that the nosuid mount option also prevents SE Linux domain transitions, so it’s not such a good option when you use SE Linux as domain transitions are often used to reduce the privileges assigned to the process.
There are two programs for labeling files in SE Linux, restorecon is the most commonly used one but there is also setfiles which although being the same executable (restorecon is a symlink to setfiles) has some different command-line options. The following command on a default configuration of a Debian/Lenny system will label a chroot environment under /chroot with the same labels as the main environment:
setfiles -r /chroot /etc/selinux/default/contexts/files/file_contexts /chroot
I am considering adding an option to support chroot environments to restorecon, if I do that then I will probably back-port it to Lenny, but that won’t happen for a while.
For a simple chroot once the filesystem is labelled it’s ready to go, then you can start daemons in the chroot environment in the usual way.
Less trusted Chroot environmentsA reasonably common case is where the chroot environment is not as trusted. One example is when you run an image of an old server in a chroot environment. A good way of dealing with this is to selectively label parts of the filesystem as required. The following shell code instructs semanage to add file contexts entries for a chroot environment that is used for the purpose of running Apache. Note that I have given specific labels to device nodes null and urandom and the socket file log in the /dev directory of the chroot environment (these are the only things that are really required under /dev), and I have also put in a rule to specify that no other files or devices under /dev should be labelled. If /dev is bind mounted to /chroot/dev then it’s important to not relabel all the devices to avoid messing up the real root environment – and it’s impractical to put in a specific rule for every possible device node. Note that the following is for a RHEL4 chroot environment, other distributions will vary a little some of the file names.
semanage -i – << END
fcontext -a -t root_t -f -d /chroot
fcontext -a -t bin_t “/chroot/bin.*”
fcontext -a -t usr_t “/chroot/usr.*”
fcontext -a -t usr_t “/chroot/opt.*”
fcontext -a -f -d /chroot/dev
fcontext -a -f -s -t devlog_t /chroot/dev/log
fcontext -a -f -c -t null_device_t /chroot/dev/null
fcontext -a -f -c -t urandom_device_t /chroot/dev/urandom
fcontext -a -t "<<none>>" "/chroot/dev/.*"
fcontext -a -t "<<none>>" "/chroot/proc.*"
fcontext -a -t lib_t “/chroot/lib.*”
fcontext -a -t lib_t “/chroot/usr/lib.*”
fcontext -a -t bin_t “/chroot/usr/bin.*”
fcontext -a -t httpd_exec_t -d — /chroot/usr/bin/httpd
fcontext -a -t var_t “/chroot/var.*”
fcontext -a -t var_lib_t “/chroot/var/lib.*”
fcontext -a -t httpd_var_lib_t “/chroot/var/lib/php.*”
fcontext -a -t var_log_t “/chroot/var/log.*”
fcontext -a -t var_log_t -f — “/chroot/var/log/horde.log.*”
fcontext -a -t httpd_log_t “/chroot/var/log/httpd.*”
fcontext -a -t var_run_t “/chroot/var/run.*”
fcontext -a -t httpd_var_run_t -f — /chroot/var/run/httpd.pid
fcontext -a -t httpd_sys_content_t “/chroot/var/www.*”
END
You could create a shell script to run the above commands multiple times for multiple separate Apache chroot environments.
If there is a need to isolate the various Apache instances from each other (as opposed to just protecting the rest of the system from a rogue Apache process) then you could start each copy of Apache with a different MCS sensitivity label which will provide adequate isolation for most purposes as long as no sensitivity label dominates the low level of any of the others. If you do that then the semanage commands require the -r option to specify the range. You could have one chroot environment under /chroot-0 with the sensitivity label of s0:c0 for it’s files and another under /chroot-1 with the sensitivity label of s0:c1 for it’s files. To start one environment you would use a command such as the following:
runcon -l s0:c0 setsid chroot /chroot-0 /usr/sbin/httpd
Links July 2010
David Byrne gave an interesting TED talk about how changes to architecture drove changes to musical styles [1]. I think he does stretch the point a little. To a certain extent people develop the most complex instruments and the largest music halls that can be supported by the level of technology in their society – people with a hunter-gatherer civilisation play drums because they can build them and can carry them.
The NY Times has an interesting article about paternity leave in Sweden [2]. The Swedish government pays for a total of 13 months leave that can be split between parents for every child. Of those 13 months 2 months can only be taken by the father – and that is likely to increase to a minimum of 4 months of paternity leave after the next election.
Dan Meyer gave an interesting TEDX talk about how the current math curriculum in the US (as well as Australia and lots of other countries that do the same thing) is totally wrong [3]. His main point is that maths problems should be based on real-world use cases where not all needed data is immediately available and there is also useless data that must be discarded. He believes that the most important thing is developing mathematical problem solving skills – basically the things that I did for fun when I was in primary school are skills that need to be taught to high-school students…
The Atlantic magazine has an amusing article by Daniel Byman and Christine Fair about the incompetent Islamic terrorists [4]. In Afghanistan half the suicide bombers kill only themselves and the US government has a lot of evidence of Taliban soldiers practicing bestiality and collecting porn. Islamic extremist groups are staffed by people who are bad soldiers and bad Muslims.
Jon Masters wrote an interesting post titled “What Would Jesus Buy” about ethical purchasing decisions [5]. Jon references The Church of Stop Shopping which isn’t a real religious organisation but a street theatre activist group.
ZeroHedga has an insightful article comparing corporations and the US government to street gangs [6]. The conclusion is that when gangs take over a neighbourhood everyone has to join a gang for their own protection.
Hillel Cooperman gave an interesting TED talk about being obsessed with Lego [7]. He compares Lego fans to Furries and makes a good case for this comparison.
Marian Bantjes gave an interesting TED talk about her graphic art / graphic design work [8]. I’ve never seen anything quite like this.
Business Insider has an interesting article about oil cleanup, it seems that most people who worked on the Exxon Valdez disaster are now dead [9], s opposed to most people who worked in almost every other occupation at that time who are either still working or enjoying their retirement. The current gulf disaster is bigger, will require more workers for the cleanup, and can be expected to have a higher death toll. Some people claim that measures to reduce oil efficiency will impact the economy, how will millions of people who are chronically ill for the rest of their lives impact the economy?
The NY Times has an interesting article on “circle lenses” [10], contact lenses designed to make the eyes look larger. It’s illegal to sell contact lenses in the US without a prescription, but the latest trend is for women to buy them online in a variety of colors. The FDA should probably approve them, it would be better to have the quality controls you expect from a medical supply company instead of having people rely on Malaysian mail-order companies for the safety of their eyes.
Don Marti has written an interesting article about the economic decline in the US, he suggests making pension funds invest in local jobs [11]. Companies are supposed to act on behalf of their stock-holders, but US companies often have the majority of their stock owned by the pension funds of workers but they act on behalf of a small number of rich people who own a minority of the stock. Don’s article was inspired by Andy Grove’s article in Bloomberg about the stagnation in technological development that has been caused by off-shoring the manufacturing [12].
Neil Brown has completed a test release of a new Linux software RAID feature for arrays with multiple redundancy that have bad sectors [13]. When a disk gets a bad sector the current behavior is to kick it out of the array, if you have two such errors on a 3 disk RAID-1 or a RAID-6 array then you lose all redundancy and are at risk of catastrophic failure even though in most cases both disks will still mostly work. With this patch some regions of the disk may be excluded but it can provide redundancy for other stripes. Thanks Neil for your great work here, and all your previous work over the last 10+ years!
The RSPCA has a new campaign titled “Close the Puppy Factories” [14]. Dogs are kept in very poor conditions and forced to churn out puppies for their entire lives to supply pet stores. The RSPCA recommends that people buy puppies from registered dog breeders (not “registered dog breeding companies”) and ask to see the dog’s parents. They also recommend not buying from classified adverts or pet stores. Animal shelters have to euthenise huge numbers of unwanted animals, you can buy a pet dog or cat from an animal shelter for a small fee that covers the expenses related to housing and spaying it – and save that animal from being euthenised!
Maureen Dowd criticises the Catholic Church properly in an article for the New York Times [15]. The Catholic Church officially regards ordaining a woman and raping a child to be equally bad offenses.
Frank Rich wrote an interesting column for the New York Times about Mel Gibson [16]. He describes the destruction of Mel Gibson’s reputation as a symptom of changes in the culture in the US and also links it to the fall of Ted Haggard (who supported Gibson’s most notorious movie The Passion of the Christ).
- [1] http://www.ted.com/talks/david_byrne_how_architecture_helped_music_evolve.html
- [2] http://www.nytimes.com/2010/06/10/world/europe/10iht-sweden.html
- [3] http://www.ted.com/talks/dan_meyer_math_curriculum_makeover.html
- [4] http://www.theatlantic.com/magazine/archive/2010/05/the-case-for-calling-them-nitwits/8130/
- [5] http://www.jonmasters.org/blog/2010/06/27/what-would-jesus-buy/
- [6] http://www.zerohedge.com/article/guest-post-corporate-entities-modern-day-street-gangs
- [7] http://www.ted.com/talks/hillel_cooperman_legos_for_grownups.html
- [8] http://www.ted.com/talks/marian_bantjes_intricate_beauty_by_design.html
- [9] http://www.businessinsider.com/warning-to-gulf-cleanup-workers-almost-every-crew-member-from-the-1989-exxon-valdez-disaster-is-now-dead-2010-6
- [10] http://www.nytimes.com/2010/07/04/fashion/04lenses.html
- [11] http://zgp.org/~dmarti/business/grove-on-jobs/
- [12] http://www.bloomberg.com/news/2010-07-01/how-to-make-an-american-job-before-it-s-too-late-andy-grove.html
- [13] http://lwn.net/Articles/391208/
- [14] http://www.closepuppyfactories.org/
- [15] http://www.nytimes.com/2010/07/18/opinion/18dowd.html
- [16] http://www.nytimes.com/2010/07/18/opinion/18rich.html
SE Linux status in Debian/Squeeze
I’ve updated my SE Linux repository for Squeeze to include a modified version of the ffmpeg packages without MMX support for the i386 architecture. When MMX support is enabled it uses assembler code which requires text relocations (see Ulrich Drepper’s documentation for the explanation of this [1]). This makes it possible to run programs such as mplayer under SE Linux without granting excessive access – something which we really desire because mplayer will usually be dealing with untrusted data. In my past tests with such changes to ffmpeg on my EeePC701 have resulted in no difference to my ability to watch movies from my collection, the ones that could be played without quality loss on a system with such a slow CPU could still be viewed correctly with the patched ffmpeg.
$ mplayer
mplayer: error while loading shared libraries: /usr/lib/i686/cmov/libswscale.so.0: cannot restore segment prot after reloc: Permission denied
The AMD64 architecture has no need for such patches, presumably due to having plenty of registers. I don’t know whether other architectures need such patches, they might – the symptom is having mplayer abort with an error such as the above when running in Enforcing Mode.
The below apt sources.list line can be used to add my SE Linux repository:
deb http://www.coker.com.au squeeze selinux
dpkgIn my repository for i386 and AMD64 architectures I have included a build of dpkg that fixes bug #587949. This bug causes some sym-links and directories to be given the wrong label by dpkg when a package is installed. Usually this doesn’t impact the operation of the system and I was unable to think of a situation where it could be a security hole, but it can deny access in situations where it should be granted. I would appreciate some help in getting the patch in a form that can be accepted by the main dpkg developers, the patch I sent in the bug report probably isn’t ideal even though it works quite well – someone who knows absolutely nothing about SE Linux but is a good C coder with some knowledge of dpkg could beat it into shape.
In my repository I don’t currently provide any support for architectures other than i386 and AMD64. I could be persuaded to do so if there is a demand. How many people are using Debian SE Linux on other architectures? Of course there’s nothing stopping someone from downloading the source from my AMD64 repository and building it for another architecture, I would be happy to refer people to an APT repository that someone established for the purpose of porting my SE Linux packages to another architecture.
Policyselinux-policy-default version 20100524-2 is now in Testing. It’s got a lot of little fixes and among other things allows sepolgen-ifgen to work without error which allows using the -R option of audit2allow – see my post about audit2allow and creating the policy for milters for defails [2].
I have uploaded selinux-policy-default version 20100524-3 to Unstable. It has a bunch of little fixes that are mostly related to desktop use. You can now run KDE4 on Unstable in enforcing mode, login via kdm and expect that everything will work – probably some things won’t work, but some of my desktop systems work well with it. I have to admit that not all of my desktop systems run my latest SE Linux code, I simply can’t have all my systems run unstable and risk outages.
Let me know if you find any problems with desktop use of the latest SE Linux code, it’s the focus of my current work. But if you find problems with chrome (from Google) or the Debian package chromium-browser then don’t report them to me. They each use their own version of ffmpeg in the shared object /usr/lib/chromium-browser/libffmpegsumo.so which has text relocations and I don’t have time to rebuild chromium-browser without text relocations – I’ll make sure it does the right thing when they get it working with the standard ffmpeg libraries. That said the text relocation problem doesn’t seem to impact the use of Chromium, Youtube doesn’t work even when the browser is run in permissive mode.
GNOME is a lower priority than KDE for me at this time. But the only area where problems are likely to occur is with gdm and everything associated with logging in. Once your X session starts up GNOME and KDE look pretty similar in terms of access control. I would appreciate it if someone could test gdm and let me know how it goes. I’ll do it eventually if no-one else does, but I’ve got some other things to fix first.
SE Linux audit2allow -R and Milter policy
Since the earliest days there has been a command named audit2allow that takes audit messages of operations that SE Linux denied and produces policy that will permit those operations. A lesser known option for this program is the “-R” option to use the interfaces from the Reference Policy (the newer version of the policy that was introduced a few years ago). I have updated my SE Linux repository for Lenny [1] with new packages of policy and python-sepolgen that fix some bugs that stopped this from being usable.
To use the -R option you have to install the selinux-policy-dev package and then run the command sepolgen-ifgen to generate the list of interfaces (for Squeeze I will probably make the postinst script of selinux-policy-dev do this). Doing this on Lenny requires selinux-policy-default version 0.0.20080702-20 or better and doing this on Debian/Unstable now requires selinux-policy-default version 0.2.20100524-2 (which is now in Testing) or better.
Would it be useful if I maintained my own repository of SE Linux packages from Debian/Unstable that can be used with Debian/Testing? You can use preferences to get a few packages from Unstable with the majority from Testing, but that’s inconvenient and anyone who wants to test the latest SE Linux stuff would need to include all SE Linux related packages to avoid missing an important update. If I was to use my own repository I would only include packages that provide a significant difference and let the trivial changes migrate through Testing in the normal way.
The new Lenny policy includes a back-port of the new Milter policy from Unstable, this makes it a lot easier to write policy for milters. Here is an example of the basic policy for two milters, it allows the milters (with domains foo_milter_t and bar_milter_t) to start, to receive connections from mail servers, and to create PID files and Unix domain sockets.
policy_module(localmilter,1.0.0)
milter_template(foo)
files_pid_filetrans(foo_milter_t, foo_milter_data_t, { sock_file file })
milter_template(bar)
files_pid_filetrans(bar_milter_t, bar_milter_data_t, { sock_file file })
allow bar_milter_t self:process signull;
type bar_milter_tmp_t;
files_tmp_file(bar_milter_tmp_t)
files_tmp_filetrans(bar_milter_t, bar_milter_tmp_t, file)
manage_files_pattern(bar_milter_t, tmp_t, bar_milter_tmp_t)
After generating that policy I ran a test system in permissive mode and sent a test message. I ran audit2allow on the resulting AVC messages from /var/log/audit/audit.log and got the following output:
#============= bar_milter_t ==============
allow bar_milter_t bin_t:dir search;
allow bar_milter_t bin_t:file getattr;
allow bar_milter_t home_root_t:dir search;
allow bar_milter_t ld_so_cache_t:file { read getattr };
allow bar_milter_t lib_t:file execute;
allow bar_milter_t mysqld_port_t:tcp_socket name_connect;
allow bar_milter_t net_conf_t:file { read getattr ioctl };
allow bar_milter_t self:process signal;
allow bar_milter_t self:tcp_socket { read write create connect setopt };
allow bar_milter_t unlabeled_t:association { recvfrom sendto };
allow bar_milter_t unlabeled_t:packet { recv send };
allow bar_milter_t urandom_device_t:chr_file read;
allow bar_milter_t usr_t:file { read getattr ioctl };
allow bar_milter_t usr_t:lnk_file read;
#============= foo_milter_t ==============
allow foo_milter_t ld_so_cache_t:file { read getattr };
allow foo_milter_t lib_t:file execute;
allow foo_milter_t mysqld_port_t:tcp_socket name_connect;
allow foo_milter_t net_conf_t:file { read getattr };
allow foo_milter_t self:capability { setuid setgid };
allow foo_milter_t self:tcp_socket { write setopt shutdown read create connect };
allow foo_milter_t unlabeled_t:association { recvfrom sendto };
allow foo_milter_t unlabeled_t:packet { recv send };
Running the audit2allow command with the “-R” option gives the following output, it includes the require section that is needed for generating policy modules:
require {
type sshd_t;
type ld_so_cache_t;
type bar_milter_t;
type foo_milter_t;
class process signal;
class tcp_socket { setopt read create write connect shutdown };
class capability { setuid setgid };
class fd use;
class file { read getattr };
}
#============= bar_milter_t ==============
allow bar_milter_t ld_so_cache_t:file { read getattr };
allow bar_milter_t self:process signal;
allow bar_milter_t self:tcp_socket { read write create connect setopt };
corecmd_getattr_sbin_files(bar_milter_t)
corecmd_search_sbin(bar_milter_t)
corenet_sendrecv_unlabeled_packets(bar_milter_t)
corenet_tcp_connect_mysqld_port(bar_milter_t)
dev_read_urand(bar_milter_t)
files_read_usr_files(bar_milter_t)
files_read_usr_symlinks(bar_milter_t)
files_search_home(bar_milter_t)
kernel_sendrecv_unlabeled_association(bar_milter_t)
libs_exec_lib_files(bar_milter_t)
sysnet_read_config(bar_milter_t)
#============= foo_milter_t ==============
allow foo_milter_t ld_so_cache_t:file { read getattr };
allow foo_milter_t self:capability { setuid setgid };
allow foo_milter_t self:tcp_socket { write setopt shutdown read create connect };
corenet_sendrecv_unlabeled_packets(foo_milter_t)
corenet_tcp_connect_mysqld_port(foo_milter_t)
kernel_sendrecv_unlabeled_association(foo_milter_t)
libs_exec_lib_files(foo_milter_t)
sysnet_read_config(foo_milter_t)
To get this working I removed the require lines for foo_milter_t and bar_milter_t as it’s not permitted to both define a type and require it in the same module. Then I replaced the set of tcp_socket operations { write setopt shutdown read create connect } with create_socket_perms as it’s easiest to allow all the operations in that set and doesn’t give any security risks.
Finally I replaced the mysql lines such as corenet_tcp_connect_mysqld_port(foo_milter_t) with sections such as the following:
mysql_tcp_connect(foo_milter_t)
optional_policy(`
mysql_stream_connect(foo_milter_t)
‘)
This gives it all the access it needs and additionally the optional policy will allow Unix domain socket connections for the case where the mysqld is running on localhost.
1 week to SysAdmin Appreciation Day
Just a quick reminder that if you’re in the New York City area next Friday, then you should come celebrate System Administrator Appreciation Day with us at The Gingerman. It’s on the east side of Manhattan, on 36th street. It’s an easy walk from Grand Central Terminal, and not too far from Penn Station, either.
Come one, come all. Raise a pint to…well…ourselves!
Online event registration for SysAdmin Appreciation Day – NYC powered by EventbriteNote: You don’t have to sign up to show up, but it helps me keep track of how many people will be there. Significant Others Welcome!
Replacement Dell PowerEdge R410 Motherboards Compromised
This is probably not quite the news that Dell wanted to get…
According to an article at The Register, Dell service has shipped replacement motherboards that contained spyware, presumably placed there at the manufacturing site.
The original post on the Dell Community Forums has this quote from a Dell rep:
As part of Dell’s quality process, we have identified a potential issue with our service mother board stock, like the one you received for your PowerEdge R410, and are taking preventative action with our customers accordingly. The potential issue involves a small number of PowerEdge server motherboards sent out through service dispatches that may contain malware. This malware code has been detected on the embedded server management firmware as you indicated.
We take matters of information security very seriously and believe that any impact to a customer’s information security is unlikely. To date we have received no customer reports related to data security. Systems running non-Windows operating systems are not vulnerable to this malware and this issue is not present on motherboards shipped new with PowerEdge systems.
We have assembled a customer list and are directly contacting customers like you through a call campaign. On the call, you should be provided a phone number to call if you have additional questions. Hopefully you received this on your call. If not, let me know and we’ll get it to you as soon as possible so you have all of the follow-up information needed.
Dell’s apparently being proactive about it…but what other option do they have? “Our factory-supplied boards come enhanced with spyware” isn’t exactly the ideal sales pitch.
If you have recently gotten a replacement motherboard for a new gen Dell PowerEdge, you might want to call your rep to make sure you’re not affected.
Digital Video Cameras
I’ve just done some quick research on Digital Video Cameras for some relatives. It seems to me that the main feature that is necessary is Full HD (1920*1080) resolution as everyone seems to be getting 1920*1080 resolution monitors (getting smaller doesn’t save enough money to be worth-while). Resolutions higher than 1920*1080 will probably available in affordable monitors in the next few years, so the ability of programs like mplayer to zoom videos will probably be required even for Full HD video soon. Saving maybe $300 on a video camera while getting a lower resolution doesn’t seem like a good idea.
The next feature is optical zoom, most cameras are advertised with features such as “advanced zoom” to try and trick customers, cameras which advertise 60* or better zoom often turn out to only have 20* zoom. I think that about 20* optical zoom should be considered the minimum, not that there is anything special about 20* zoom, it’s just that there is a good range of cameras with better zoom capacity.
Image stabilisation is a required feature, no-one can keep their hand perfectly steady and the typically a DVC only gets hand-held use – most people who own them don’t even own a tripod! Digital image stabilisation is apparently not nearly as good as optical image stabilisation, and image stabilisation that involves moving the CCD is apparently somewhere in between.
Finally it’s good to have the ability to take quality photos as few people will want to carry a Digital Camera and a Digital Video Camera.
I did a search for DVCs on the web site of Ted’s Camera store (a chain of camera stores in Australia that generally provide good service at a competitive price – but not the cheapest price). The best of the Ted’s options seems to be the Panasonic SD60 HD Video [1] which does 25* optical zoom, 1920*1080i video, 5 megapixel still photography, and optical image stabilisation – it costs $750 from Ted’s.
The next best option seems to be the Sony Handycam HDR-CX110 HD [2] which does 25* optical zoom, 1920*1080i video, 3.1 megapixel 2048*1536 still photography, and digital image stabilisation. The Panasonic seems to be a better option due to having optical image stabilisation and a higher resolution for still photographs. It is also $750 from Ted’s.
Now there’s the issue of how well the cameras work on Linux. A quick Google search indicated that the Sony cameras present themselves as USB card readers and can be mounted on a Linux system, I couldn’t discover anything about the Panasonic. If I was going to buy one I would take my Netbook to the store and do a quick test.
I don’t have enough information to recommend either of those cameras, they may have some awful defects that are only apparent when you use them. But in terms of features they seem pretty good. The Panasonic SD60 HD Video should be a good benchmark when comparing cameras in the store. If nothing else the camera store staff seem to not be very helpful if asked generic questions such as “which camera is best”, but if asked questions such as “how is this other camera better than the one I’m looking at” they can usually give good answers.
If anyone has any other advice for purchasing a DVC then please let me know. Either generic advice or specific examples of Linux-friendly DVCs that have been purchased recently.
Wonky Bunny Issue “Fixed”
For those who don’t know what the headline means:
- Bunny is an open source command line utility written in Python that provides a shell for talking to and testing AMQP brokers (tested on RabbitMQ).
- AMQP is a queuing protocol. It’s defined as a binary wire-level protocol as well as a command set. The spec also defines a good portion of the server semantics, so by that logic Bunny should work against other AMQP brokers besides RabbitMQ
- RabbitMQ is written in Erlang atop OTP, so clustering is ‘free and easy’. My experience with RabbitMQ so far has been fantastic, though I’d like to see client libraries in general mature a bit further.
So, Bunny had this really odd quirk upon its first release. If you did something to cause an error that resulted in a connection being dropped, bunny wouldn’t trap the error. It would patiently wait for you to enter the next command, and fail miserably. The kicker is that I actually defined a ‘check_conn’ method to make sure that the connection was alive before doing anything else, and that really wasn’t working.
The reason is because py-amqplib (or, perhaps, its interpretation of the AMQP spec, which defines a Connection class), implements a high-level Connection class, along with a Channel class (also defined in the spec), which is what seems to actually map to what you and I as users actually care about: some “thing” that lets us communicate with the server, and without which we can’t talk to the server.
With py-amqplib, a Connection is actually defined as a channel 0, and always channel 0. I gather that channel 0 gets some special treatment in other sections of the library code, and the object that lives at index ’0′ in Connection.channels is actually defined as a Connection object, whereas others are Channel objects.
The result of all of this is that creating a channel in my code and then checking my own object’s ‘chan’ attribute is useless because channels can be dropped on the floor in py-amqplib, and the only way I can tell to figure that out is to check the connection object’s ‘channels’ dictionary. So that’s what I do now. It seems to be working well.
Not only does bunny now figure out that your connection is gone, but it’ll also attempt a reconnect using the credentials you gave it in the last ‘connect’ command. You see, bunny extends the Python built-in cmd.Cmd object, which lets me define my whole program as a single class. That means that whatever you type in, like the credentials to the ‘connect’ command, can be kept handy, since the lifetime of the instance of this class is the same as the lifetime of a bunny session.
So, in summary, bunny is more useful now, but it’s still not “done”. I made this fix over the weekend during an hour I unexpectedly found for myself. It’s “a” solution, but it’s not “the” solution. The real solution is to map out all of the errors that actually cause a connection to drop and give the user a bit more feedback about what happened. I also want to add more features (like support for getting some stats back from Alice to replace bunny’s really weak ‘qlist’ command).
addthis_url = 'http%3A%2F%2Fwww.protocolostomy.com%2F2010%2F07%2F19%2Fwonky-bunny-issue-fixed%2F'; addthis_title = 'Wonky+Bunny+Issue+%26%238220%3BFixed%26%238221%3B'; addthis_pub = 'jonesy';Australia sysadmins! I'm coming to you!
I'll be speaking at MacTech conference, Nov 3-5, 2010
Tracking down Write/Execute mmap() calls with LD_PRELOAD
One of the access controls in SE Linux is for execmem – which is used to stop processes from creating memory regions that are writable and executable (as they make it easier to compromise programs and get them to execute supplied code). When the SE Linux audit log tells you that a program is attempting such access it’s sometimes difficult to discover where in the code such an access occurs, for example if you have a large code base and mmap() is called in many places it can be difficult to determine which one is the culprit. Especially if you have a source package that contains multiple binaries that use a common shared library and you don’t know which bits of library code are called by each executable.
To solve this problem in the case of freshclam to provide extra information for Debian bug report #588599 [1] I wrote the following little shared object which can be compiled with “gcc -shared -g -fPIC mmap.c -o mmap.so” and used with “LD_PRELOAD=./mmap.so whatever“. Then when the program in question (or any non-SUID program it executes) calls mmap() with both PROT_EXEC and PROT_WRITE set the program will abort. If you run this through gdb then the program will break and you will get a back-trace of the function calls that led to the undesired mmap().
One thing to note is that this method only catches direct calls to a library function outside libc. When the libc code calls the library function (EG all the fwrite() etc code that calls mmap()) the LD_PRELOAD hack won’t catch it. Thanks to Keith Owens for pointing this out.
#include <dlfcn.h>
#include <stdio.h>
#include <sys/mman.h>
#include <stdlib.h>
#undef NDEBUG
#include <assert.h>
void *libc6 = NULL;
void *(*real_mmap)(void *, size_t, int, int, int, off_t);
void do_init()
{
libc6 = dlopen("libc.so.6", RTLD_LAZY | RTLD_GLOBAL);
if(!libc6)
{
printf("Aieee\n");
exit(1);
}
real_mmap = (void * (*)(void *, size_t, int, int, int, off_t))dlsym(libc6, "mmap");
}
void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset)
{
if(!real_mmap)
do_init();
assert(!(prot & PROT_EXEC) || !(prot & PROT_WRITE));
return real_mmap(addr, length, prot, flags, fd, offset);
}
Introduction to SNMP
Introduction to what? This isn’t going to be a “how to configure SNMP for your server” kind of introduction. I’m no great expert there, but if there’s call for it, I can share my configuration bits to help. This is more of a “what the heck is SNMP” introduction. Hopefully it’ll be more valuable, since there are reams of existing documentation on how to actually configure the services, and not so many on why you should care.
Really, the system administration world is divided into two camps. Those of us who want to monitor our servers, network gear, and get performance metrics so that we can trend future usage, and those of us who don’t yet know that we want those things. The former group uses SNMP. The latter group will probably get something out of this post.
If you’re new to the idea of SNMP, bear with me for a second. Suppose that it would be handy to remotely query all of your network devices and retrieve stats from them. If you’re familiar with the concept of logging into, say, a router, you know that you can get information that way. If you buy intelligent switches, you know that you can telnet, ssh, or web-browse to the switch interface and check out what’s going on that way. Likewise, you can log into your servers and check out the stats there, but overall, there are as many different ways of getting this information as there are devices that you want it from. That’s no good, because no one wants to script that many possible interactive sessions.
This is the problem that SNMP was meant to address. SNMP means Simple Network Management Protocol, and it is just a well-agreed-upon language (protocol) that almost all network devices speak. By using SNMP, you can effectively move beyond the normal administrative interface of your network device and just query it for information. Sounds great, right? It Is!
Well, ok, it can be. From this pristine dream of a one-ness of network devices, we muddy the waters a bit when it comes to the specifics. As of right now, there are three different versions of the SNMP protocol, with the primary differences between v1 and v2 being capabilities, and the primary differences between v2 and v3 being security.
Starting with v1, and continuing with v2, SNMP didn’t actually have “usernames” and “passwords”, so much. They instead had “community strings”, which function as passwords, but without all of those messy account details getting in the way. Typically speaking, there was a community string for reading data (the default was “public”), and a community string for writing settings, when the device supported that (with a default of, yep, you guessed it, “private”). It’s hard to imagine why anyone thought this was an insecure protocol, but apparently some people were uncomfortable with the idea of all of their machines being monitored remotely with no accountability whatsoever. Weird, I know.
That brought about the idea of SNMP v3, which packs as many security features into it as the previous versions lacked. In fact, that’s pretty much all it does. The actual protocol request itself is still v1 or v2, but with extra security layers. By default, not only does SNMP v3 require the use of accounts with passwords, but the transmission itself is encrypted (DES by default, though some vendors support better encryption like 3DES)to protect the account credentials and data. In addition, each of the transmissions is signed (using MD5 or SHA-1) to guarantee that it wasn’t altered in transit. Because yeah, that’s not overkill for me querying the number of bits transmitted since the last time I asked.
Anyway, to use the universal car analogy, you can either have the jeep with no roll cage (v1/2) or the armored tank (v3).
Honestly, I use SNMP v2, and as much as I hate to admit it, I have a nearly universal read-only community string that I use for it. It’s not “public”, and I disable the write-access community string, but I run old hardware. A lot of it doesn’t work with v3. In fact, some of it doesn’t even work with v2, but for everything that does, I use v2. It is noticeably faster, and as far as security is concerned, 99% of my things are internal on a private IP-based switched network. If someone is sniffing my packets, I have bigger issues than my read-only community string being compromised. You, on the other hand, may want to check things over the internet. In that case, use SNMP v3. The encryption will be worth the time you invest.
So that’s an introduction to what the versions are, but that’s not much of an explanation of what SNMP *IS*. SNMP is a logical tree.
Imagine that you’re an snmp server in the mid 1990s. You don’t have a lot of RAM, but you have a lot of data to keep track of. Strange remote machines will be querying you to access this data. What method do you use to keep track of the data that they want?
In the case of SNMP, they used a tree. Every branch of the tree is separated from the parent and child branches by a period. Taken together, this string of numbers is called an OID, or Object IDentifier. The very top of the tree (or very bottom, depending on how you look at it) is the most abstracted…and you’re almost always going to see it start with a 1, which has been assigned to the Internet Standards Organization, or ISO. In fact, a lot of the OIDs that you run into will start with 1.3.6.1, which maps to ISO.identified-organization.dod.internet. You can browse the entire registered OID tree at http://www.oid-info.com, if you’re really bored.
Alright, so imagine that you’ve browsed all the way down to 1.3.6.1.2.1.2.2.1.16. Great. What the heck does that mean, though?
The other great tree of numbers strung together with dots, IP addresses, had the same problem a long time ago, and so DNS was invented, to map IP addresses to names. For a very similar reason, there is a Management Information Base, or MIB, that maps OIDs to useful names. That 1.3.6.1.2.1.2.2.1.16 monstrosity above? Yeah, it actually means ifOutOctets, shorthand for interface output octets. It’s a 32 bit counter that shows the number of octets which have been output by each interface. When I query it (more on that shortly) on a machine with 5 interfaces, I get the following output:
IF-MIB::ifOutOctets.1 = Counter32: 2766014067
IF-MIB::ifOutOctets.2 = Counter32: 3209623655
IF-MIB::ifOutOctets.3 = Counter32: 3606918534
IF-MIB::ifOutOctets.4 = Counter32: 2521574893
IF-MIB::ifOutOctets.5 = Counter32: 0
There are some very standard OIDs that are universal across pretty much all devices. On the other hand, many devices have specialized OIDs that you probably wouldn’t otherwise find (and certainly wouldn’t know what they meant!) unless you had the specific MIB for that device. For this reason, many manufacturers have made their MIBs available for download, but there are also websites that archive MIBs and make them searchable by the public. This can be a huge help if you want to know how many VPN users are currently logged in, or really anything else that is non-standard or hard to find.
Think of the MIB files as a map to the information you want to look for.
Now, how to actually get that information out of the device…
If you want to query by hand (certainly only a temporary measure), in the Unix/Linux world, I recommend net-snmp. It includes a suite of tools to poke and prod SNMP-enabled devices, but the two things that I use the most are snmpwalk and snmpget.
The block of results above were retrieved using snmpwalk. What I did was issue the following command:
snmpwalk -v 2c -c CommunityString servername 1.3.6.1.2.1.2.2.1.16
If you notice, the output from that command returned 5 lines, with the first field of each line ending in “ifOutOctets.#”, where # is the number of the interface. That’s because the actual OID of each of those values was 1.3.6.1.2.1.2.2.1.16.#! If I try to use ’snmpget’ (which, unlike snmpwalk, only returns one result), it fails:
snmpget -v 2c -c CommunityString servername 1.3.6.1.2.1.2.2.1.16
IF-MIB::ifOutOctets = No Such Instance currently exists at this OID
However, specifying the correct OID does the trick:
snmpget -v 2c -c CommunityString servername 1.3.6.1.2.1.2.2.1.16.1
IF-MIB::ifOutOctets.1 = Counter32: 2766027795
What ’snmpwalk’ actually does is walk the tree. I specified ‘1.3.6.1.2.1.2.2.1.16′, so it said “alright, I’m going to get that OID, then I’m going to dive in and get ‘.1′, then ‘.2′, etc etc until it reaches a failure message indicating that there aren’t any more children. By this method, you can actually query a huge part (or even all) of the tree.
In this case, I knew I had 5 interfaces, numbered 1-5 (according to the OID results from snmpwalk), but I didn’t know which interface was registered as which number…I did know, however, that one of the interfaces was called ‘eth0′, so I shaved some numbers off of the OID, and executed this snmpwalk:
snmpwalk -v 2c -c CommunityString servername 1.3.6.1.2.1.2.2 | grep eth0
IF-MIB::ifDescr.2 = STRING: eth0
Excellent. At this point, I know that ifDescr is the name (registered in the MIB) that holds the interface descriptions. So I just execute an snmpwalk against that:
snmpwalk -v 2c -c CommunityString servername ifDescr
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
IF-MIB::ifDescr.4 = STRING: bond0
IF-MIB::ifDescr.5 = STRING: sit0
Easy as pie.
Of course, you don’t always want to query by hand…in fact, it’s probably the exception, rather than the rule. You want monitoring software to do all that stuff for you. Pretty much every monitoring software known to man can query snmp directly (and if it can’t, you know how to query it via the command line now, so you can write a script to do it, if it’s absolutely necessary). Most of the graphing solutions like Cacti, MRTG, and everything else include code to query, and even Nagios has a check_snmp plugin (which I highly recommend using, rather than creatively solving the problem yourself).
This really only leaves one stone unturned. SNMP Traps. Essentially, SNMP traps are a way of letting the SNMP server stop being passively queried and start actively letting someone know that something is wrong. Configuring a trap involves specifying a remote server (or servers) to alert when something goes horribly awry.
The remote server specified needs to be listening for SNMP traps. In Unix/Linux, it’s not too difficult to get net-snmp to listen for them, and on Windows, there is software available to do the same thing. Here’s one I found with a quick search. I’m sure there are more, so if you have a favorite, please let us know what it is in the comments.
The only thing left is to tie your notification system into the trap server, but I’ll leave that as an exercise for the reader.
Thanks for reading, and hopefully you got something out. If you have a favorite SNMP tip or trick (or I screwed something up), let us know in the comments!
Progress report and vacation next week
It appears that long laid plans are finally coming to fruition.
That link was to a post written on June 4, 2009, the first time that I mention that I wanted to try puppet. And over a year later, here I am rolling puppet onto my production servers. It took forever, but there’s a lot of underlying infrastructure, too. The RPM building environment and skillset was the biggest hurdle. Compared to that, the repo and subversion repository were cake!
I do want to thank everyone who has given me a hand with my questions throughout the process. I feel like I’ve bugged R.I.Pienaar, Jordan Sissel, and Ben Cotton the most, but I appreciate everyone’s help.
You’re not going to hear too much out of me next week. I’m taking some vacation time and heading to the Damariscotta River Association’s Archaeological Field School in Maine. I’ll spend a week learning how to dig a hole
By far, the most frequent question that I’ve gotten when I tell people about this is, “How did you find out about that?”. As it turns out, there are some really good places online that list digs that you can attend. You do have to pay for them, typically, and you have to show up for a minimum amount of time, but you can go and learn how archaeology works. I used this database at Archaeological.org to find mine, but there are spots all over the world. If you’re into it, go find yourself one!
Also, in case you’re not sick of hearing about it, the last Friday of this month is the SysAdmin Appreciation Day Meetup in NYC. Remember, also, if you’re in the San Francisco Bay area, OpenDNS is throwing their own. There’s no reason that you can’t throw your own if you can’t come to ours. If you want to organize one, drop me an email and I’ll mention it on the blog.
Everybody have a good week!
