| felix | jaime: startlog #redes |
| felix | jaime: startlog #qc |
| felix | jaime: listlogs |
| jose_n | is it time? |
| jose_n | felix: in what channel do i present? |
| roel | inhere, but normally someone of umeet introduces the speaker, they seem to be afk |
| jose_n | heheh .. |
| jose_n | anyone want to make it up? |
| jose_n | anyone? |
| EMPEROR | jose_n: :) i think to need introduce yourself ? O:) |
| jose_n | ok :) |
| jose_n | hi all ... how is everyone today. |
| jose_n | i guess i'll be presenting here in #linux, with translations in #redes and questtions in #qc |
| jose_n | let's begin! |
| unizar.es.uninet.edu MODE #linux +o EMPEROR | |
| jose_n | talk materials are here: http://monkey.org/~jose/presentations/pysniff04.d/ |
| jose_n | ok |
| jose_n | [slide 0] |
| jose_n | talk materials are here: http://monkey.org/~jose/presentations/pysniff04.d/ |
| jose_n | hello and welcome to my presentation. i'll be explaining how to use |
| jose_n | the python language to sniff network traffic. |
| jose_n | we'll be using some |
| jose_n | python libraries and building a few small applications that give us |
| jose_n | great views into network traffic. i'm also using this as a chance to |
| jose_n | more widely introduce a new tool i wrote, "flowgrep". so, let's get |
| jose_n | going! |
| jose_n | on slide 25 i have a list of URLs and resources. |
| jose_n | [slide 1] |
| jose_n | you might be asking yourself "why use python?" when you think about |
| jose_n | what network traffic sniffing means, you're often facing situations where |
| jose_n | you have to use very fast programs. and since python is an interpreted |
| jose_n | language, you expect it to be slower than C. that's ok for our |
| jose_n | purposes. |
| jose_n | python gives us many other advantages over C. first, we have an |
| jose_n | easy to use language so we can rapidly develop prototypes and new code. |
| jose_n | secondly, we have easy access to basic data structures, like lists and |
| jose_n | arrays, which we can easily build upon. |
| jose_n | a list of lists, for example, |
| jose_n | would look like this in python: |
| jose_n | |
| jose_n | >>> mylist1 = ['item 1', 'item 2'] |
| jose_n | >>> mylist2 = ['cat', 'dog'] |
| jose_n | >>> superlist = [mylist1, mylist2] |
| jose_n | >>> print superlist |
| jose_n | [['item 1', 'item 2'], ['cat', 'dog']] |
| jose_n | |
| jose_n | you can do that in C, but it's harder and requires more lines of code |
| jose_n | ("LoC") to do so. my biggest reason for using a scripting language over |
| jose_n | C is the easy string manipulation, either making tokens or using regular |
| jose_n | expressions. |
| jose_n | |
| jose_n | |
| jose_n | you can learn more about python at http://www.python.org/, including |
| jose_n | tutorials, distributins (UN*X, Windows, and Mac OS X included), and |
| jose_n | modules. |
| jose_n | |
| jose_n | [slide 2] |
| jose_n | |
| jose_n | we are going to marry python and sniffing by using C libraries which |
| jose_n | have been exported to python using the "SWIG" tool. "SWIG" is the |
| jose_n | scripting wrapper interface generator, which lets us make new modules |
| jose_n | for scripting languages from C and C++ libraries. |
| jose_n | it can export to |
| jose_n | Tcl, Python, Perl, C#, and so much more. libraries exist for "pcap", |
| jose_n | which is the canonical sniffing library, "dnet", dugsong's packet crafting |
| jose_n | library, and "libnids", which is what we're using today. |
| jose_n | |
| jose_n | these are exported as Python modules like pcapy or pypcap (for libpcap), |
| jose_n | "dpkt" for a packet inspection tool, and libdnet even includes Python |
| jose_n | bindings. we're going to use "pynids", a Python interface to libnids. |
| jose_n | this lets us reassemble connections and easily see their contents. |
| jose_n | |
| jose_n | i didn't write "pynids", but i built a few tools on top of it. i've been |
| jose_n | sniffing packets for years and like doing it in Python when i get |
| jose_n | a chance |
| jose_n | |
| jose_n | |
| jose_n | @jose_n> |
| jose_n | 13:26 -! |
| jose_n | err .. |
| jose_n | [slide 3] |
| jose_n | |
| jose_n | talk materials are here: http://monkey.org/~jose/presentations/pysniff04.d/ |
| jose_n | |
| jose_n | libnids is a simple library designd to facilitate the IP stream reassembly |
| jose_n | fnuctions. simply put this means that we can see connections as connections |
| jose_n | and not just packets, and the library also reassembles fragments into |
| jose_n | their original packets. |
| jose_n | t's based on the Linux 2.0.36 TCP/IP stack. |
| jose_n | internally it uses libpcap to sniff and libnet to handle the packets. |
| jose_n | |
| jose_n | on slide 25 i have a list of URLs and resources. |
| jose_n | |
| jose_n | [slide 4] |
| jose_n | |
| jose_n | the kernel is what normally houses the TCP/IP stack. functions for networking |
| jose_n | are handled within the kernel, and you call into it using sockets and |
| jose_n | the like. |
| jose_n | |
| jose_n | [slide 5] |
| jose_n | |
| jose_n | libnids exports this IP stack to the userland, where you can manipulate it |
| jose_n | more easily to examine packets. you don't replace your kernel's stack, |
| jose_n | but you do make a copy of it. very handy! |
| jose_n | |
| jose_n | [slide 6] |
| jose_n | |
| jose_n | using libnids is much easier than most people realize. you want to initialize |
| jose_n | the library (nids_init()) first. then you want to register callback |
| jose_n | functions which handle packets into their respective protocols. |
| jose_n | TCP |
| jose_n | packets are handled by TCP functions which you tell the system about by |
| jose_n | using "nids_register_tcp()". here packets will be placed into their |
| jose_n | connection or a new connection state will be made if needed. you do the |
| jose_n | same for UDP and IP packets, as well. |
| jose_n | |
| jose_n | you then launch the whole thing by calling "nids_run()", which starts |
| jose_n | to listen for pakets and send them to the appropriate function that you |
| jose_n | told it about. if you don't register an IP packet handler, it wont |
| jose_n | process them. |
| jose_n | you can even react and kill TCP connections by using |
| jose_n | "nids_kill_tcp()", which sends TCP reset packets to each party in the |
| jose_n | connection. these are the C fuctions, which look a lot like out Python |
| jose_n | functions. |
| jose_n | |
| jose_n | [slide 7] |
| jose_n | |
| jose_n | this shows you what "nids_run()" does. for every type of packet you |
| jose_n | get the library will process it using the callback functions you |
| jose_n | specified earlier. |
| jose_n | |
| jose_n | TCP connections are tracked exactly as you would track them in the kernel, |
| jose_n | using port numbers, IP addresses, sequence numbers and acknowledgement |
| jose_n | numbers. |
| jose_n | |
| jose_n | [slide 8] |
| jose_n | |
| jose_n | libnids exposes the basic TCP states in three ways. the first is for newly |
| jose_n | created TCP sessions, it sets the state "NIDS_JUST_ESTABLISHED". this |
| jose_n | occurs once the TCP 3 way handshake (3WHS) has taken place. |
| jose_n | |
| jose_n | NIDS_DATA is set for new packets in an exsting stream. you can append them |
| jose_n | to the existing data from the connection or discard the data if you want. |
| jose_n | |
| jose_n | for TCP connection closure 3 states are available. NIDS_RESET is when |
| jose_n | the connection was reset by one of the peers. NIDS_CLOSE is for a |
| jose_n | graceful TCP close, and NIDS_TIMED_OUT is for a connection that appears |
| jose_n | to have timed out (ie no closure sent or seen). |
| jose_n | |
| jose_n | libnids doesn't expose the intermediate stages of the TCP connection. |
| jose_n | |
| jose_n | [slide 9] |
| jose_n | |
| jose_n | pynids wraps this in much the same way that you would do in C. you have |
| jose_n | nids_run() which drives it (or you can loop over nids_next() to go |
| jose_n | one packet at a time). it dos the reassembly for you without any work |
| jose_n | on your part. |
| jose_n | |
| unizar.es.uninet.edu MODE #redes +o cygar | |
| unizar.es.uninet.edu MODE #redes +o felix | |
| unizar.es.uninet.edu MODE #redes +o EMPEROR | |
| jose_n | [slide 10] |
| jose_n | |
| jose_n | here's a basic mapping of the functions from slide 6 to how they are done |
| jose_n | in pynids. most of the functions have the same name and make sense. to |
| jose_n | change internal libnids settings you can just use the "nids.param()" |
| jose_n | function. and then you launch the sniffing process like you did in |
| jose_n | C. |
| jose_n | |
| jose_n | [slide 11] |
| jose_n | |
| jose_n | the order of operations in pynids are exactly like they are in C: packets |
| jose_n | arrive, libnids reassembles the fragments as needed, and the right callback |
| jose_n | function is evaluated. |
| jose_n | within that callback function you can parse the |
| jose_n | data stream, look at the packet attributes (ie sequence numbers) and you |
| jose_n | can do whatever you want. you get all of these features exposed to you |
| jose_n | without any trouble. |
| jose_n | |
| jose_n | |
| jose_n | [slide 12] |
| jose_n | |
| jose_n | here's a very simple python example. we will show the TCP handler on |
| jose_n | the nxt slide. we import the nids library, then in main() we set |
| jose_n | a parameter to disbale portscan detection, the library is initialied, |
| jose_n | then we register a TCP handler (handleTcp()), and then we launch the |
| jose_n | setup via nids.run(). very simple! |
| jose_n | |
| jose_n | [slide 13] |
| jose_n | |
| jose_n | here's what the TCP handler looks like. it gets passed a tcp object, which |
| jose_n | holds the data, the addresses, and the attributes of the session. we |
| jose_n | can spit out the data when we're done. |
| jose_n | notice that we have to set the |
| jose_n | tcp.client.collect variable to 1 to collect the client's data (data |
| jose_n | sent to the client), and the same for the server data. without this |
| jose_n | the library wont keep track of the session data, but it will keep track |
| jose_n | of the session itself. |
| jose_n | |
| jose_n | notice that we have to set the |
| jose_n | tcp.client.collect variable to 1 to collect the client's data (data |
| jose_n | sent to the client), and the same for the server data. without this |
| jose_n | the library wont keep track of the session data, but it will keep track |
| jose_n | of the session itself. |
| jose_n | oops ... |
| jose_n | |
| jose_n | [slide 14] |
| jose_n | |
| jose_n | here's the same example in C. here's the main loop showing the library |
| jose_n | setup and parameter setting, and then we launch libnids using nids_run(). |
| jose_n | |
| jose_n | |
| jose_n | [slide 15] |
| jose_n | |
| jose_n | and here is the TCP clalback function. the TCP object here is a pointer |
| jose_n | to the TCP state, and we can peek inside the structure at the destination |
| jose_n | ports and the like. this simple example just collects data from webservers |
| jose_n | and web clients and spits it out when the connection closes. |
| jose_n | |
| jose_n | now, this has about the same number of lines (minus some of the C |
| jose_n | brackets and such). but once you start doing string or data structure |
| jose_n | manipulation, the operations are much easier Python than in C. that's |
| jose_n | why we chose a scripting language. |
| jose_n | |
| jose_n | [slide 16] |
| jose_n | |
| jose_n | so, what can you do with this? well, here's three programs i wrote to |
| jose_n | show you how to play with the library. |
| jose_n | |
| jose_n | the first is VersionDetect, a tool i wrote to use the headers sent |
| jose_n | by various servers and clients to interrogate their OS versions and |
| jose_n | software. it's very simple, and supports SSH client and server strings, |
| jose_n | mail client strings, and web server and client strings. |
| jose_n | it tells you the |
| jose_n | string that was reported by the computer, which often reveals the OS |
| jose_n | or the client architecture. |
| unizar.es.uninet.edu MODE #redes +o krocz | |
| jose_n | |
| jose_n | i wrote this because i had previously written a passive TCP stack fingerprinting |
| jose_n | tool which sometimes gives bad results. i wanted to know for sure what |
| jose_n | kind of system i was talking to, or if it was behind a security device that |
| jose_n | proxied the connection. |
| jose_n | |
| jose_n | some of the things i've found using this include the fact that some of the |
| jose_n | MSN servers i talk to use OpenBSD firewalls and proxies but are |
| jose_n | actually IIS servers. |
| jose_n | |
| jose_n | some of the things i've found using this include the fact that some of the |
| jose_n | MSN servers i talk to use OpenBSD firewalls and proxies but are |
| jose_n | actually IIS servers. |
| jose_n | OOPS .. |
| jose_n | |
| jose_n | what i should do with the tool is have it write to a database so i can |
| jose_n | log it all and query it when i need the information. :) |
| jose_n | |
| jose_n | VersionDetect is available here: http://monkey.org/~jose/software/version_detect/ |
| jose_n | |
| jose_n | [slide 17] |
| jose_n | |
| jose_n | here's some sample output of the tool when i run it on my work |
| jose_n | laptop. you can see, for example, that i use OpenBSD on i386 to |
| jose_n | browse the web with Firefox 0.6.1 (i need to update!). you an also |
| jose_n | see i talk to an OpenSSH 3.5 server and a bunch of web servers. |
| jose_n | some |
| jose_n | of them reveal that they're on a particular type of platform, and |
| jose_n | others don't reveal too much. after a while you get to see some interesting |
| jose_n | stuff, like nice Apache modules and the like. |
| jose_n | |
| jose_n | [slide 18] |
| jose_n | |
| jose_n | http-graph is another tool i wrote to investigate something. ineeded a quick |
| jose_n | proof of concept tool to show people what i meant when i was describing |
| jose_n | how i wanted to think about new representations of web browsing histories. |
| jose_n | |
| jose_n | http-graph sniffs the requests and replies from web servers to build a |
| jose_n | directed graph of how you get from one website to the next. all of the |
| jose_n | information we need is in the header of the request, including the |
| jose_n | URL we want, the server it's on, and the referring website. |
| jose_n | |
| jose_n | http-graph is available here; http://monkey.org/~jose/software/http-graph/ |
| jose_n | |
| jose_n | [slide 19] |
| jose_n | |
| jose_n | what http-graph does is reconstruct this information into a pair of |
| jose_n | strings, one for the referrer and one for the request itself. this |
| jose_n | gives us a view of natural "hubs" of information in our browsing. |
| jose_n | |
| jose_n | other people have looked at this sort of thing, too. see |
| jose_n | http://www.uiweb.com.nyud.net:8090/issues/issue37.htm |
| jose_n | |
| jose_n | [slide 20] |
| jose_n | |
| jose_n | http-graph captures this information and creates a "dot" output file. |
| jose_n | "dot" is part of the "graphviz" toolkit, which lets us graph directed |
| jose_n | graphs very easily. i use the tool "neato" to make the graphs and |
| jose_n | display in various formats, like SVG (scalable vector graphics), |
| jose_n | postscript or PDF formats. |
| jose_n | |
| jose_n | [slide 21] |
| jose_n | |
| jose_n | here is an example graph and a detail of that graph. this particular |
| jose_n | spiderweb came from me lanching from my homepage to other pages i have. |
| jose_n | from there i downloaded more pages, so you can see a natural progression |
| jose_n | from my homepage to, for example, a new york times article. |
| jose_n | |
| jose_n | sometimes these graphs get very large, so i need to come up with a |
| jose_n | way to make them smaller and easier to view. |
| jose_n | |
| jose_n | |
| jose_n | [slide 22] |
| jose_n | |
| jose_n | you can, of course, play with the data from the TCP session. it's just |
| jose_n | a string object. what you can do includes searching, applying regular |
| jose_n | expressions, or even rewriting the strings. |
| jose_n | |
| jose_n | this is how i use the data |
| jose_n | to reconstruct the version strings from servers and clients or look |
| jose_n | at the web browsing graph. just simple string operations. a scripting |
| jose_n | language makes this worlds easier than it would be in C. |
| jose_n | |
| jose_n | [slide 23] |
| jose_n | |
| jose_n | so, knowing that we can sniff traffic reliably and look at the data, |
| jose_n | of course you can have a lot of fun! you can invade peoples' privacy |
| jose_n | and sniff their mail, log their conversations over IM tools or IRC |
| jose_n | ... |
| jose_n | you can also steal files that someone else is downloaded, or just disrupt their |
| jose_n | sessions. dugsong's tool "dsniff" includes a lot of functionality, |
| jose_n | and we can make it up on the fly, too, using Python and pynids. |
| jose_n | |
| jose_n | krocz has asked what is the performance of this library in C (libnids) |
| jose_n | in C you can scale, with the right memory management, easily to fast ethernet. |
| jose_n | as you get faster it gets harder, but with decent hardware you can get over 100 Mbps and handle it. |
| jose_n | (thank you ducky for the translation) |
| jose_n | |
| jose_n | [slide 24] |
| jose_n | |
| jose_n | flowgrep is a new tool i wrote recently to investigate worm activity |
| jose_n | for new worms. i needed a tool that married regular expressions and |
| jose_n | network sniffing. |
| jose_n | "ngrep" (or network grep) is a lot like this, but |
| jose_n | it only logged a single packet. i needed the whole connection. |
| jose_n | tcpkill couldn't look inside the data, and dsniff wasn't flexible |
| jose_n | enough for my needs. |
| jose_n | |
| jose_n | krocz was asking about python performance in big networks ... |
| jose_n | in my experience for fast networks with many simultaneous TCP sessoins, python may not be the best choice ofr analyzing the network |
| jose_n | stick to C |
| jose_n | but for smaller networks or very focused tools (such as in a testbed network) you can use pynids and python to examine the data |
| jose_n | |
| jose_n | (continuing the talk) |
| jose_n | |
| jose_n | what flowgrep does is evaluate the data in the TCP connection using |
| jose_n | these regular expressions and logs it or even kills the connection. |
| jose_n | by marrying network pcap expressions and regular expressions, i have |
| jose_n | a flexible tool to look at network malware like worms or even spam. |
| jose_n | |
| jose_n | flowgrep makes a very cheap and simple IDS or IPS (intrusion prevention |
| jose_n | system) and is written in under 400 lines of code, using Python. |
| jose_n | |
| jose_n | [slide 25] |
| jose_n | |
| jose_n | you can download these libraries and tools in this talk at these links. |
| jose_n | tcpdump.org hosts the libpcap library (which is included in most UN*X |
| jose_n | distributions). my code is on my website at these links, and you'll |
| jose_n | need "pynids" (the last link) for running these tools. a bunch of |
| jose_n | fun network tools are being written in Python lately. |
| jose_n | |
| jose_n | cygar was asking about pythons limitations in congested networks using this library. the main limitatons are the same in C, such as data examination speeds and memory management. you can get more efficnet memory management in C than you can in python because you can chose how to do it yourself. |
| jose_n | python sniffing will be slower than C sniffing, yes. |
| jose_n | |
| jose_n | [slide 26] |
| jose_n | |
| jose_n | finally, you'll want to read TCP/IP Illustrated if you plan to do any |
| jose_n | serious amounts of sniffing or network evaluation. Mike Schiffman wrote |
| jose_n | a great book introducing many of these libraries in C, like libpcap and |
| jose_n | libnids. and finally you will often have to refer to IETF RFC documents |
| jose_n | to look at how a protocol operates. |
| jose_n | |
| jose_n | (fini!) thank you al for your attention. |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| cygar | clap clap clap clap clap !! |
| cygar | clap clap clap clap clap !! |
| roel | thanks |
| cygar | thank you jose_n ! |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| ducky | clap clap clap clap clap |
| ducky | clap clap clap clap clap |
| ducky | clap clap clap clap clap |
| ducky | clap clap clap clap clap |
| MJesus | thank you very much !! |
| ducky | very nice talk jose_n |
| jose_n | thank you to the organizers, too, for ptting on these conferences. thanks! |
| MJesus | nice talk |
| ducky | thanks in advance |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| jose_n | the slides and text of my talk are here: http://monkey.org/~jose/presentations/pysniff04.d/ |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| krocz | CLAP CLPA LCPA LCPA CLAP |
| krocz | CLAP CLAP CLAP CLAP CLAP |
| krocz | CLAP CLAP CLAP CLAP CLAP |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| jose_n | :) |
| krocz | personalemten no conocia, las ventajas de python en ese sentido, se hace mucho mas facil el analisis del trafico |
| jose_n | another great place to get python help is aspn.net .. |
| jose_n | or is it .com .. |
| jose_n | err http://aspn.activestate.com/ASPN/Python/Cookbook/ |
| jose_n | by the way, for everyone who likes perl, you can do this all in perl, too |
| jose_n | http://search.cpan.org/~abergman/Net-LibNIDS-0.01/LibNIDS.pm |
| ducky | Thanks to everyone, for coming here, in a couple of hours |
| unizar.es.uninet.edu MODE #linux +o ducky | |
| ducky | we will have the second day's talk |
| ducky | at 21 GMT |
| ducky | the talk is about |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| ducky | Freedesktop: integration, between desktops |
| ducky | by Alejandro Sanchez Acosta from spain |
| ducky | Alejandro is a gnu member |
| MJesus | thanks you Jose Nazario |
| ducky | he have done, lots of contributions to the free software |
| ducky | thanks a lot to jose |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| MJesus | clap clap clap clap clap clap clap clap clap clap |
| krocz | thanks ducky |
| krocz | jaime: stoplog |
| krocz | jaime: stoplog #linux |
| krocz | jaime: stoplog #redes |
| krocz | no me quiere de nuevo :( |
| cygout | jaime: !endlog |
| krocz | jaime endlog #linux |
The Organizing Comittee
