Charlas 6-12-2000 Ummet'2000

Charlas del 6/12/2000

Log de la conferencia. Se han suprimido las líneas correspondientes a entradas y salidas de diferentes personas en el canal durante la conferencia

[23:00] (marius) hi
[23:00] (marius) back
[23:01] (MJesus) plas plas plas plas plas plas plas plas plas plas
[23:01] (MJesus) plas plas plas plas plas plas plas plas plas plas
[23:01] (MJesus) plas plas plas plas plas plas plas plas plas plas
[23:01] (MJesus) marius... welcome
[23:01] (CtrInt) plas*
[23:01] (corrsite) well, company people needs not only a good idea, also support compañies which give support to his problem (like microsoft and his patners do)
[23:01] (corrsite) marius lo
[23:02] (marius) this isn't too late is it?
[23:02] (marius) it's 5 here now
[23:03] (fernand0) well
[23:03] (fernand0) the talk was scheduled an hour ago
[23:03] (fernand0) but we can begin it now :)
[23:03] (marius) ? oh... i'm sorry about that -- i got message it was at 5pm est
[23:03] (marius) (right now)
[23:04] (fernand0) well
[23:05] (marius) oh well -- stupid timezones: should we start now then?
[23:05] (marius) alright
[23:05] (marius) fernand0: is it ok if we start?
[23:05] (fernand0) welll
[23:05] (fernand0) yes
[23:05] (fernand0) i was preparing something to introudce you
[23:06] (marius) ok, go ahead
[23:06] (marius) :)
[23:06] (fernand0) but , in these special circunstances i do not know if my introduction will work
[23:06] (fernand0) just a minute ....
[23:08] (fernand0) welll
[23:08] (fernand0) let's start
[23:08] (fernand0) Hello,
[23:08] (fernand0) we are very please to present here Marius Aamodt Eriksen and Mike Baker.
[23:08] (fernand0) Their talk will be about:
[23:08] (fernand0) snoopy - a libc level exec() logger
[23:08] (fernand0) About Marius Aadmodt: he is working for linux.com as Project Manager for
[23:08] (fernand0) the TuneUp and Multimedia sections. He codes, produces content, ...
[23:08] (fernand0) This also involves doing a lot or programming.
[23:09] (fernand0) He is also a full time student at the University of Michigan, College of Engineering. * Logging for #linux halted
[23:09] (fernand0) About Mike Baker: he is working for VA Linux Systems.
[23:09] (fernand0) He is being interested in linux since February'96.
[23:09] (fernand0) He also is involved in lots of coding
[23:09] (fernand0) He enjoys collecting weird/rare hardware.
[23:09] (fernand0) Thank you for comming here
[23:09] (marius) thanks
[23:09] (MJesus) plas plas plas plas plas plas plas plas plas plas
[23:09] (MJesus) clap clap clap clap clap clap clap clap clap clap
[23:09] (MJesus) plas plas plas plas plas plas plas plas plas plas
[23:09] (MJesus) clap clap clap clap clap clap clap clap clap clap
[23:09] (MJesus) plas plas plas plas plas plas plas plas plas plas
[23:09] (_ZaFiO_) nas
[23:10] (marius) an addition to that - i also work at citi - the cente rof information technology integration
[23:10] (marius) on the linux scalability project
[23:10] (marius) who is sponsoring this work, as well
[23:10] (marius) (amongst many other things, like /dev/poll)
[23:10] (marius) ok, now about snoopy
[23:10] (marius) what is snoopy?
[23:11] (marius) it's an exec() logger in userspace
[23:11] (marius) we host it on sourceforge, http://sourceforge.net/projects/snoopy logger/
[23:11] (marius) if you're familiar with c programming, it would help to take a look at http://cvs.sourceforge.net/cgi-bin/cvsweb.cgi/snoopy/snoopy.c?rev=1.22&c ontent-type=text/x-cvsweb-markup&cvsroot=snoopylogger
[23:11] (marius) which is the latest cvs version of it
[23:12] (MbM) nasty long url
[23:12] (marius) it's quite short - so it shouldn't be too hard to follow
[23:12] (marius) heh yeah
[23:12] (marius) ok,
[23:12] (marius) so how does snoopy do this?
[23:12] (marius) it's a shared library
[23:12] (marius) it is able to overload execve()
[23:12] (marius) (we call it 'hijacking')
[23:12] (marius) execve from libc, that is
[23:13] (marius) it does this through the LD_PRELOAD mechanism
[23:13] (marius) so, what is LD_PRELOAD?
[23:13] (marius) i'll let mbm start off on that
[23:13] (MbM) ok, i'm certain you're all ateast somewhat familar with the concepts of shared libraries
[23:14] (MbM) LD_PRELOAD is the environmental variable that lets us hook in a library before any of the others
[23:14] (MbM) so that it's symbols override any that are on the other libraries
[23:14] (MbM) it's meant as a mechnanism for development
[23:14] (marius) ... particular to the gnu version of ld/libc, is the /etc/ld.so.preload file
[23:15] (marius) if you list a shared library in that file, it will automatically be preloaded, without setting LD_PRELOAD
[23:15] (MbM) so if if you wanted to test out a new a new function to your library you might just write up only that routine and hook it in va preload
[23:16] (MbM) snoopy takes the concept abit further and uses preload to form a wrapper around the orgiginal function
[23:16] (MbM) it's really one of those concepts we ran into hacking late one night and though, gee what would this be good for
[23:17] (MbM) we take no credit for coming up with the idea of wrappers like that, if you've ever used esddsp that's nearly the same thing
[23:18] (marius) yeah, esd wraps open()
[23:18] (marius) so when you try to usr /dev/dsp, it will redirect it to the esd mechanism
[23:18] (MbM) (esddsp emulates open() and watches for open(/dev/dep) then redirects to esound)
[23:18] (marius) :)
[23:18] (marius) ok, so how does snoopy use function overloading ("hijacking")?
[23:19] (marius) the particular libc function we're interested in is execve(), the ultiamte endpoint of exec*() statements
[23:19] (marius) snoopy is very simple, it does this in a few steps:
[23:20] (marius) 1) when it's loaded, it gets the address ofhte _real_ execve() and stores that in a function pointer
[23:20] (MbM) (via dlsym)
[23:20] (marius) 2) it simply makes a function called execve(..) with identical parameters
[23:20] (marius) this overloads the real execve()
[23:21] (marius) 3) when execve() is called [in a dynamically linked executable: more on this later], it calls snoopy's version
[23:21] (MbM) don't forget the logging part ;)
[23:21] (marius) 4) snoopy's version of execve() logs the request and passes it on to the real exeve() via our pointer
[23:21] (marius) :)
[23:21] (marius) the logging is done via syslog
[23:22] (marius) this means that we have the security of syslog as well - il.e. you can do offsite logging and such
[23:22] (marius) so, this is pretty simple
[23:22] (marius) the code is pretty clean, and should be easy to interpret
[23:23] (marius) (if you're reading the cvs version, ignore the #ifdef TOYS part, it's exactly taht :)
[23:23] (MbM) 6 lines of c code to overload the excv() functions
[23:23] (marius) now
[23:24] (marius) what happens if an executable isn't dynamically linked?
[23:24] (MbM) well simple fact of the matter is that you can only pull off the LD_PRELOAD trick on dynamically linked executables
[23:25] (MbM) but since snoopy logs the execve() call and not the actual program
[23:25] (MbM) you'll likely be catching bash's executuion attempt
[23:25] (MbM) and so the format of the called program doesn't come into play
[23:26] (MbM) one of the ideas we came up with later was to try alternate forms of logging
[23:26] (marius) as long as you're restricting the type of shells a user can run... (/etc/shells, etc)
[23:27] (MbM) with things like __libc_start_main() which is an internal glibc function
[23:27] (MbM) that's actually the portion of glibc that loads main() and takes pretty much the same arguments
[23:27] (MbM) the difference between that an execve() style logging is which pid is calling snoopy
[23:29] (MbM) while to the end user the results may be identical it does make a difference if you try to trace or log a static linked executable
[23:29] (marius) which brings us to the next point - transparency
[23:30] (MbM) bottom line, snoopy is detectable
[23:30] (marius) if you run an strace or an ltrace on any (dynamic) executable while snoopy is preloaded, it will show
[23:30] (marius) upon examining how those two traces worked, we employed a couple of 'anti tracing' ideas
[23:31] (MbM) let's first pause and exaplain what a trace really is
[23:31] (MbM) when you device you want to strace or ltrace a program it's done via the ptrace() mechanism
[23:31] (marius) (process trace)
[23:32] (MbM) in plain english the program is run, but with orders to stop at any given point
[23:32] (ByTePTr) nas
[23:32] (marius) it's a system call which provides the parent process means of observing / controlling how the program is run
[23:32] (MbM) meanwhile, the tracing program uses wait()
[23:32] (MbM) and watches for a signal from the child process
[23:33] (MbM) when the child signals the tracing program, the trace pokes the executiuon pointer and starts reading off the variables
[23:33] (marius) wait() is sort of a blocking function taht waits for a signal
[23:33] (MbM) those you you playing along athome probably have an idea how we bypassed this one
[23:34] (MbM) if we provide our own wait() function
[23:34] (MbM) we can receive the signal from the child process being traced
[23:34] (MbM) and we can do exactly what trace does
[23:34] (MbM) that is, we can see where it stopped and if we don't like that stop point
[23:34] (MbM) we continue on as if nothign had ever happened
[23:35] (MbM) and the strace is totally in the dark that anything had ever happened
[23:35] (MbM) if we were really cunning we could even send back false information
[23:35] (MbM) (note to self, compile strace statically)
[23:36] (marius) another obvious traceable is dlopen() which the ld mecahnism calls to load the preloads
[23:36] (ROCO) Hola a todos
[23:37] (MbM) well, not preloads as such as runtime libraries
[23:37] (marius) yeah
[23:38] (marius) so we wrote our own overload for that :)
[23:38] (MbM) let's take a step back and explain the problem with dlopen()
[23:39] (MbM) dlopen() is used to load a library at runtime, such as a plugin
[23:39] (MbM) it's used in combination with dlsym()
[23:39] (MbM) which will look up a function and return a pointer
[23:40] (MbM) the problem (although nobody but us would think to look for it)
[23:40] (MbM) is that i can dlopen libc, look up execve()
[23:41] (MbM) and notice that it doesn't quite match the address of execve() in memory
[23:41] (MbM) hence tipping me off to the fact that someone had loaded an extra execve()
[23:42] (marius) so what we did was to ovelroad dlopen()
[23:42] (marius) and seeing if they were tring to load libc
[23:42] (marius) in that case, we just passed snoopy's address back :)
[23:42] (MbM) (yes folks we're paranoid about transparancey)
[23:43] (MbM) what we found amazing was the fact that we could pull off this vanishing act completely in userspace
[23:43] (marius) those are the measures of transparency that's currently in snoopy
[23:44] (marius) we have also done some experiments, messing with elf headers and such to redirect calls to execve() directly to us
[23:44] (MbM) now, you're probably wondering, why does snoopy really need trasnparency, why do we try to hide it
[23:45] * MbM lets marius step in
[23:45] (marius) well, what i was saying irc.uia.mx.uninet.edu)
[23:46] (marius) is that in attempt to sort of bypass all tracing mechanisms, and trying to "catch" execve at its root, we've been messing around with elf headers (of the real libc) and such
[23:46] (marius) but that hasn't produced much tangibles yet
[23:46] (marius) it gets tricky to point back and forth between the two, for example if snoopy were to unexpectedly unload, we'd be stuck with an invalid function pointer
[23:46] (marius) and system tends to have a short lifetime withoiut a valid execve() :)
[23:47] (marius) so, that is still work in progress
[23:47] (marius) now
[23:47] (MbM) (by overloading dlopen() and dlsym() we leave marks in the trace from our calsl to the real function, we're looking into reading the function names directly from elf headers and avoid any detectable calls)
[23:47] (marius) yeah, by simple overloading, you run into a loop effect
[23:47] (marius) like, tracing the trace
[23:48] (marius) ok, now back to the other question: why does snoopy need to be transparent?
[23:48] (marius) our first and original goal of snoopy
[23:48] (marius) was to provide system monitoring functions
[23:48] (marius) that's why we chose to have all exec() calls logged to syslog etc
[23:49] (marius) a system adminstrator would be able to search in their logs for any commands, etc. that were executed (that shouldn't have been)
[23:49] (marius) and because it's realtime, you oculd even watch an attack
[23:49] (marius) someone trying to break into teh system, get root access, etc
[23:49] (MbM) (lotta execve() calls from cron showing up in my syslog.. oh wait that's an intruder)
[23:49] (marius) another use of snoopy is 'behind the scenes' action
[23:49] (marius) like apache executing cgis and such
[23:49] (marius) sometimes, it's nice to have a general "feel" of what's going on in the systme
[23:50] (marius) and snoopy provides
[23:50] (marius) that
[23:50] (marius) the transparency is there, so the intruder isn't able to think that he/she is being watched
[23:50] (marius) this is in actual use
[23:50] (marius) in at least one very large networking company
[23:51] (marius) to provide "light monitoring" of users
[23:51] (marius) so - in many cases, the transparency is good enough.
[23:51] (marius) but, again, the transparency isn't perfect
[23:52] (marius) and also the fact that statically linked executables (that execve()s something) isn't logged can also be a downfall.. but more on that later)
[23:52] (marius) so, many will ask
[23:52] (marius) "isn't that what bsd accounting does?"
[23:52] (MbM) each time you overload a function in attempts to hide it you leave more marks in the trace with the extra code added
[23:52] (marius) -) yes, and that is why we're trying to "catch it at the root," where you'd avoid that
[23:52] (marius) back to teh bsd accounting
[23:52] (marius) bsd accounting is designed for just that. accoutning
[23:53] (marius) it's pure command logging with cpu and memory statistics
[23:53] (marius) it doens't provide any 'realtime loggin'g as such
[23:53] (marius) and doesn't log command arguments (which may be essential)
[23:53] (marius) so, many may have heard of 'exec.o'
[23:53] (marius) which is a kernel module which does the same thing
[23:54] (marius) so, "why not in kernel space"?
[23:54] (marius) a couple of reasons
[23:54] (marius) 1) it's realy cool to do it in userspace
[23:54] (marius) 2) portability - snoopy will run on almost any UNIX
[23:54] (marius) (i've ran it on solaris, openbsd, and more)
[23:54] (marius) 3) convenience
[23:54] (marius) it's a shared library, and it's very portable, generally
[23:54] (marius) 4) exploration
[23:55] (marius) the last point is probably the most important one
[23:55] (marius) because that is what has added some goals to snoopy
[23:55] (MbM) if it were a module you'd have separater versions for the freebsd kernel, the linux kernel, etc, and you'd have to recompile it each kernel
[23:55] (marius) as is now, snoopy realizes a huge potential for security holes, the fact that yo'ure able to change the behavior of any function, and thus executable quite easily
[23:55] (marius) ought to alarm people
[23:56] (marius) essentially
[23:56] (marius) snoopy can be thought of as a worm
[23:57] (marius) or trojan
[23:57] (marius) because it hides, and modifies tihngs in the background quite easily
[23:57] (marius) for example
[23:57] (marius) a novice user dwonloads an rpm
[23:57] (marius) the rpm sticks a malicious ;library in /etc/ld.so.preload
[23:57] (marius) (rpms are usually installed as root, iirc)
[23:57] (marius) the author of the malicious software now has full control over hte system
[23:57] (marius) this is _real_ easy to do
[23:57] (MbM) (if we can override execve() that easily, what would stop someone from overloading say the open() function in ls and making ls into an rm.. there's some security issues to think about there..)
[23:58] (marius) there's just tons of malicious stuff htat can be done
[23:58] (marius) so, what we're investigating now is how much we could possibly try to hide snoopy
[23:58] (marius) how transparent we can make it
[23:58] (marius) when we think that we have reached a really god level of transparency
[23:58] (marius) we want to see how we could possibly detect it
[23:58] (marius) which leads us to the next goal
[23:58] (marius) to create a computer immune system
[23:59] (marius) (for ld/libc related matters)
[23:59] (marius) if we were able to hide snoopy so well, we should be able to detect it (hopefully)
[23:59] (MbM) that's in answer to the "what stops someone from using this transparency idea in their own trojan?"
[23:59] (marius) and thus we would be able to create scanners, etc. to check for snoopy-like "worms" and behavior
[00:00] (marius) additionally, it would be portable, across all UNIXes that have similar ld and libcs
[00:00] (marius) and that's basically it.
[00:00] (marius) :)
[00:00] (marius) now, we're open for questions
[00:01] (fernand0) plas plas plas plas plas plas plas plas plas
[00:01] (fernand0) plas plas plas plas plas plas plas plas plas
[00:01] (fernand0) plas plas plas plas plas plas plas plas plas
[00:01] (fernand0) thank you for this nice talk
[00:01] (CtrInt) applauses.. :)
[00:01] (fernand0) are there any questions ?
[00:02] (MbM) i have one: is anyone still awake?
[00:02] (marius) hehe
[00:02] (fernand0) hehe
[00:02] (marius) it's quite late in europe :)
[00:02] (CtrInt) mm.. where did you find those preload stuff? libc src only?
[00:02] (fernand0) yes it is
[00:03] (marius) CtrInt: man ld.so
[00:03] (marius) i hope tihs served as an eye-opener for how vulnerable our systems can be
[00:03] (CtrInt) heh.. i've been lookin' to the info files :-P
[00:03] (MbM) CtrInt: it's a well known trick, it's also useful for if you get undefined symbols loading programs.. if you know which lib has those symbols you just preload it
[00:03] (marius) it's really amazing how easy it is
[00:03] (fernand0) i was preparing a question about security issues of this ld_preload thing
[00:03] (CtrInt) yeah.. really it is
[00:03] (fernand0) but you commented on this
[00:04] (fernand0) well, anyway
[00:04] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:04] (MJesus) clap clap clap clap clap clap clap clap clap clap
[00:04] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:04] (MJesus) clap clap clap clap clap clap clap clap clap clap
[00:04] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:04] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:04] (MJesus) clap clap clap clap clap clap clap clap clap clap
[00:04] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:04] (fernand0) nobody should install a n rpm without being really confident about the source
[00:04] (MbM) one of the ideas that was bounced back and forth is that with execve() logging you could write a virtual menu system
[00:04] (marius) yeah, or deb for that matter
[00:04] (marius) even apt
[00:05] (marius) if someone hijacks a dns server
[00:05] (MbM) that is, you could watch for someone running say "foo x y z" and treat that as an internal command
[00:05] (CtrInt) yeah.. digitaly signed I think
[00:05] (marius) and redirecets you to a malicious repository..
[00:05] (CtrInt) and deb's doens't support signatures.. isn't it?
[00:05] (fernand0) nice idea mike
[00:05] (MbM) suid programs are somewhat safe, you're not allowed LD_PRELOADS there for obvious reasons
[00:05] (marius) well, you can always compute the checksum of anything
[00:05] (fernand0) so configure lib behaviour or something similar ??
[00:06] (MbM) we're still looking at the suid/preload block to see if there's any holes in it
[00:06] (marius) yeah - we've been doing some intensive drilling in that area
[00:06] (CtrInt) a checksum can be overrided to match those original
[00:07] (marius) yeah - even getting a checksum from a relaiable source, there are all sorts of mathematical tricks you can play
[00:07] (CtrInt) i'm paranoic too :-(
[00:07] (MbM) CtrInt: now there's a thought, what stops me from messing with md5sum, watching for a magic md5 then substitue
[00:07] (marius) MbM: think the idea is to check _before_ you get infected
[00:08] (MbM) like was hinted to before, the best way to catch our transparency tricks is to understand exactly how they're done.. most of them are useless against a static linked debugger
[00:08] (marius-) yup
[00:08] (fernand0) MbM) that is, you could watch for someone running say "foo x y z" and treat
[00:08] (fernand0) that as an internal command
[00:09] (fernand0) this could be used to reconfiure lib behavior ?
[00:09] (MbM) fernand0: well, to some degree
[00:09] (MbM) fernand0: remeber how snoopy hooks in, it's a lib, not a program
[00:09] (MbM) each process will see it's own copy of snoopy
[00:10] (fernand0) ok
[00:10] (fernand0) you are true
[00:10] (MbM) so it's impossible for me to create a virtual command that dumps out all program run (keystroke monitos have been known to do this)
[00:11] (MbM) which is one of the reasons why we log to syslog in the first place
[00:11] (MbM) but you could do reconfiguration on a pid basis
[00:11] (fernand0) but inside a program, maybe you could reach different levels of logging in different sections of it, for example
[00:11] (fernand0) and similar things
[00:12] (MbM) yes
[00:12] (marius) yes
[00:12] (marius) that's possible
[00:12] (CtrInt) mm, maybe some weird and dangerous but this could be used not only for trojan or login but running overloading of current lib's features
[00:12] (CtrInt) doin' that on a per process basis too.. heh
[00:12] (marius) yeah
[00:12] (marius) we could even check who's running it
[00:12] (CtrInt) funny
[00:13] (marius) and then decide on that
[00:13] (marius) you can easily implement a fake root by doing something similar
[00:13] (marius) by modifying the behavior of getuid()
[00:13] (MbM) CtrInt: yes, that's one of the practical uses for preloads, I'ev seen librarys for detecting leaks in programs that will overload malloc() and free() and keep tabs
[00:13] (marius) your shell will think you have root, but you don't..
[00:13] (marius) (getuid() { return 0; })
[00:13] (marius) MbM: i belibe that's how electric fence works
[00:13] (marius) (?)
[00:16] (fernand0) well
[00:17] (MbM) marius: might be
[00:17] (MbM) there's a whole world of possibilities when you realize youc an redefine any function like that
[00:17] (MbM) i've used those same concepts to mess with the uname -a replies
[00:17] (marius) yup
[00:17] (marius) hhe, you can make your systel look like it's solaris :)
[00:17] (MbM) thinking i'll just have it run fortune in place of the kernel timestamp
[00:17] (marius) basically, the whole mechanism is a very hot spot for security exploits
[00:17] (marius) and eventually we hope to be able to create a generic scanner that works against that
[00:17] (MbM) there's an obvious potential for someone to follow in our footsteps and misue our code, we've been aware of that and as such we also include ways of detecting
[00:18] (marius) yes, we've always kept our detect utility up with snoopy
[00:18] (MbM) and that's where the immune system idea comes from
[00:18] (fernand0) you can continue commenting on the talk
[00:18] (fernand0) but i think it is a good moment to thank our speakers of today
[00:18] (fernand0) and all of you for comming here
[00:18] (MbM) actually that was all i had to say ;)
[00:19] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:19] (MJesus) clap clap clap clap clap clap clap clap clap clap
[00:19] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:19] (MJesus) clap clap clap clap clap clap clap clap clap clap
[00:19] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:19] (MJesus) clap clap clap clap clap clap clap clap clap clap
[00:19] (MJesus) clap clap clap clap clap clap clap clap clap clap
[00:19] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:19] (MJesus) clap clap clap clap clap clap clap clap clap clap
[00:20] (fernand0) It is time also to announce
[00:20] (fernand0) that we'll put the log of this talk
[00:20] (fernand0) and pointers to the sites the speakers referenced
[00:21] (fernand0) as soon as possible in our web
[00:21] (fernand0) umeet.uninet.edu
[00:21] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:21] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:21] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:21] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:21] (MJesus) plas plas plas plas plas plas plas plas plas plas
[00:22] (fernand0) tomorrow, we will have at 18 cet
[00:22] (fernand0) an intersting conference about
[00:22] (fernand0) Etica na rede
[00:22] (fernand0) Ethics on the net
Y seguimos un buen rato más charlando...

Contact: