| IRC log started Mon Dec 3 14:51 |
| fernand0 | 5 minutes and we will start |
|---|
| MJesus | si alquien quiere ayudar a traducir a español, por favor, que entre en #redes |
|---|
| JALH | MJesus, and in englitch? |
|---|
| riel | fernand0: ok, lets wait a few minutes for people who had problems connecting |
|---|
| sarnold | #linux +m ? |
|---|
| riel | sarnold: soon ;) |
|---|
| mail | holas |
|---|
| MJesus | jalh, here we speak english, in #redes spanis, in #media maori, in #seti arabic, in #jornada portunhol, in #media gaelic, etc ¶:Þ |
|---|
| HoraPe | :-) |
|---|
| JALH | :-) |
|---|
| velco | MJesus: #media ? |
|---|
| JALH | I can't speak spanish tho ;( |
|---|
| MJesus | indeed, we can smile in all idiomatic version! |
|---|
| JALH | hehe |
|---|
| HoraPe | velco, it was a joke... |
|---|
| velco | HoraPe: got that much ;) |
|---|
| MJesus | moari, from New Zealand (kia ora=hi) |
|---|
| krUNIX | hola |
|---|
| fernand0 | kr ? |
|---|
| riel | ok, the channel is now moderated |
|---|
| riel | if I think it's an interesting thing to talk about, I'll put it into my lecture right here |
|---|
| * riel likes improvising ;) |
| fernand0 | Dear friends, |
|---|
| fernand0 | we are going to start |
|---|
| fernand0 | we should start this meeting giving thanks. |
|---|
| fernand0 | First of all, giving thanks to all the people who registered for the |
|---|
| fernand0 | conference, and the people who are going to give lectures here. |
|---|
| fernand0 | Of course, an event like this, would not be possible without the |
|---|
| fernand0 | collaboration of many people, that worked hard in order to organize all the |
|---|
| fernand0 | things. |
|---|
| fernand0 | Thanks also to all of them |
|---|
| fernand0 | Last year UMeet begun as an adventure: to try to join people here to talk |
|---|
| fernand0 | about different aspects of Unix |
|---|
| fernand0 | We got impressed: many people collaborated |
|---|
| fernand0 | and helped to construct this, and asked nothing for this collaboration. |
|---|
| fernand0 | Our hope is that this year meeting will be, at least, as good as the last |
|---|
| fernand0 | year one. |
|---|
| fernand0 | One of the persons who offered his collaboration last year was Mr. Riel. |
|---|
| fernand0 | It is our pleasure to present you today Rik van Riel, one year (and some |
|---|
| fernand0 | days) later. |
|---|
| fernand0 | All of you undoubtely have heared about him. Just in case... |
|---|
| fernand0 | et us remember that he is a kernel hacker working for Conectiva, in Brazil. |
|---|
| fernand0 | He is going to give here today a Kernel Surprise Lecture. |
|---|
| fernand0 | Mr. Riel ... |
|---|
| JALH | *clap* *clap* |
|---|
| riel | well, since this is the opening lecture of Umeet 2001, I'll be giving a lecture in the tradition of all keynote lectures |
|---|
| riel | a somewhat high-level overview of a particular topic, with some short technical excursions into the matter |
|---|
| riel | my "slides" are available on http://surriel.com/lectures/umeet2001.html |
|---|
| riel | you don't really need them with this lectur, but it could be nice to look at |
|---|
| riel | ok ;) |
|---|
| riel | I will start this lecture with an overview of what happened with the 2.4 kernel in the last year |
|---|
| riel | about one and a half year ago, kernel 2.4.0-test1 was released |
|---|
| riel | the idea behind this release was that many people would test the kernel and report bugs to the kernel hackers |
|---|
| riel | so the bugs could be fixed before kernel 2.4.0 was released |
|---|
| riel | so much for theory |
|---|
| riel | in practice ... |
|---|
| riel | ... almost nobody tested kernel 2.4, before 2.4.1 |
|---|
| riel | so most major bugs got discovered after 2.4.0 was released |
|---|
| riel | in fact, the first few months of the 2.4 kernel was a period of mainly bug fixing, on a kernel not much more stable than a development kernel ;) |
|---|
| riel | but it was lots of fun, people discovering bugs and getting them fixed within a few weeks |
|---|
| riel | only around kernel 2.4.3 or 2.4.4 were we starting to run into the harder-to-fix bugs, where it took considerable effort and thinking to analyse and fix bugs |
|---|
| riel | luckily Usenix and OSDN were organising a nice event in San Jose this march |
|---|
| riel | the 2.5 kernel summit |
|---|
| riel | at this event many kernel hackers got together, not just to talk about the 2.5 kernel (as was planned), but also about problems in 2.4 and how to fix them |
|---|
| riel | and, in a stroke of luck, Red Hat decided to fly its kernel hackers to the US a week earlier |
|---|
| riel | .. and lock them into the office in North Carolina, with enough coffee and food for a week |
|---|
| riel | this resulted in a lot of harder to find bugs in the core kernel getting found |
|---|
| riel | some of them even got fixed in that week, while others got discussed at the 2.5 kernel summit |
|---|
| riel | this means that the event which was supposed to be about 2.5 has also helped 2.4 a lot ;) |
|---|
| riel | ok, a quick question: <FloodeR> Question to riel: Don't you think that it be a little dangerous to "play" with stable kernel series? |
|---|
| riel | well, FloodeR has posed a good question here |
|---|
| riel | and he is right, once a stable kernel really is stable, we probably should not play with it |
|---|
| riel | but one of the big problems was that 2.4 did not really get tested until about 9 months _after_ 2.4.0-test1 was released |
|---|
| riel | so many of the bugs, which happen only on special workloads on huge servers, simply not found, not even by the stress tests some kernel hackers run |
|---|
| riel | in order to fix these bugs, some playing around had to be done |
|---|
| riel | ok ........ |
|---|
| riel | july 2001 was another strange point for kernel development |
|---|
| riel | for some reason, Linus seems to have gotten very busy around that time |
|---|
| riel | rumours are it has something to do with the fact that he has 3 children, a wife, a full-time job doing hardware things at Transmeta _and_ is trying to run Linux kernel development at the same time |
|---|
| riel | just imagine doing the linux kernel as a hobby ;))) |
|---|
| riel | anyway, because of this, a lot of patches simply didn't get into Linus his kernel |
|---|
| riel | Alan Cox, on the other hand, had integrated enough bug fixes by now that his kernel was getting relatively stable |
|---|
| riel | stable enough to integrate things like big architecture upgrades (S390, PPC, ..) and integrate new code such as UML and ext3 |
|---|
| riel | the extra stability and new features meant that many kernel developers started switching to Alan's kernel and making patches for Alan's kernel |
|---|
| riel | ouch, I'm getting a few --- #linux :Cannot send to channel |
|---|
| MJesus | [22:26] <hensema> riel: just for reference, what kernel was current in juli 2001? 2.4.9? |
|---|
| riel | ok, it seems most people get everything allright |
|---|
| riel | hensema: in juli, we were around 2.4.7, IIRC |
|---|
| riel | well, since many developers were running Alan's kernel around that time and the difference between the kernels by Linus and Alan got larger, some patches being sent to the mailing lists did not even apply to Linus his kernel |
|---|
| riel | and both kernels started forking away from each other |
|---|
| riel | this got worse in august |
|---|
| riel | around this time, Alan's 2.4.9-ac kernel got stable enough to survive pretty heavy stress tests on big servers |
|---|
| riel | while Linus started integrating use-once and other experimental features into his VM |
|---|
| riel | around this time, almost all distributions were using Alan's kernel as the basis for their kernel RPM |
|---|
| riel | also, even more patches only worked on Alan his kernel, making it even harder to get both kernel trees in sync |
|---|
| riel | in september it got to the point where Linus gave up on the VM in his tree and switched to a new VM by Andrea Arcangeli |
|---|
| riel | after which it took until around 2.4.14 to get that new VM stable (which is quite fast for a VM subsystem, btw) |
|---|
| riel | well, a question <onki> riel, we have seen rumours about two vm's (on the kernel list it was suggested a.f.a.i.k) would that be a good idea in your opinion? |
|---|
| riel | after last weekend's thread about natural selection vs. design in kernel development, and also after having my VM and Andrea's VM compete for a month or so, I think it's good to have competition ;) |
|---|
| riel | but having 2 VMs in the kernel at the same time is just too much overhead to be worth it, in my opinion |
|---|
| riel | if somebody can write a VM, that person should have no problem generating a patch people can apply ;) |
|---|
| riel | well, moving on with last year's kernel news ... Linus got his kernel really stable around 2.4.14 |
|---|
| riel | and fixed the last bugs in 2.4.15 (urhmmm, hummm, well, almost) |
|---|
| riel | so 2.4.15 got forked to 2.5.0 and the 2.4 tree was handed over to Marcelo Tosatti |
|---|
| riel | note that a LOT of bug fixes and updates have not yet been merged in 2.4, so I guess marcelo will be buried alive in patches for several weeks to come |
|---|
| riel | and he will have the difficult task of trying to integrate critical patches and updates, while at the same time keeping the kernel stable |
|---|
| riel | I'm sure marcelo will be able to handle this, though ... marcelo is a pretty careful and conservative maintainer by nature ;) |
|---|
| riel | ok, if there are no questions on 2.4, I'll go on with the second half of my lecture |
|---|
| riel | ok, it seems we have somebody with a question ... lets wait a bit for the 2.4 questions to be asked ;) |
|---|
| riel | <wol> riel: I've followed the 2.4 development and I'm nervous that critical bugfixes seem never to get integrated. Do you think this will change now Marcelo is in charge? |
|---|
| riel | ok, I sit next to marcelo at the office, so this is an easy question ;) |
|---|
| riel | then again, maybe marcelo could answer this one ? |
|---|
| riel | marcelo, are you around ? ;) |
|---|
| riel | ok, no marcelo ;( I'll try to answer this one ;) |
|---|
| riel | well, I've heard from marcelo that he has a large list of patches waiting to be integrated |
|---|
| riel | updates frome the PARISC, HPPA and PPC people |
|---|
| riel | and a lot of other bugfixes and updates coming directly from the maintainers of various subsystems |
|---|
| riel | <hensema> riel: lots of people tend to be unhappy about the critical bugs in released kernels, like 2.4.11 and 2.4.15. Do you think anything can be done in order to prevent such bad releases? |
|---|
| riel | hensema: ok, I think this is a very hard problem |
|---|
| riel | the main problem here is that bugs need to be discovered before they can be fixed |
|---|
| riel | but in order to have bugs discovered, people need to be able to download and run the kernel |
|---|
| riel | so this is some sort of chicken-and-egg problem ;/ |
|---|
| riel | on the other hand, now that 2.5 is forked and 2.4 development can slow down a bit, I think the number of critical bugs will get a lot less |
|---|
| riel | <dabeej> the vm you had in earlier releases of 2.4 <dabeej> which release would you call your best ? |
|---|
| riel | dabeej: that would be 2.4.13-ac7 ;)) |
|---|
| riel | dabeej: once the VM got stable in 2.4.9-ac, I still had a lot of work to do making it quick ... and even 2.4.13-ac7 is still missing some of my patches ;) |
|---|
| riel | ok, two related questions |
|---|
| riel | <Folken> riel , and what about the -rc proposed in LKML? |
|---|
| riel | <onki> riel, additional question to hensema's question, is there some sort of list kernal hackers are using to test a new release? |
|---|
| riel | well, if marcelo will use the -rc scheme I don't know, but I think it could be a nice way to attract more testers to the kernel |
|---|
| riel | which would be a good way to find bugs _before_ the "final" release is out |
|---|
| riel | onki: the mailing list would be linux-kernel@vger.kernel.org, which is _the_ place to send any kernel bug reports |
|---|
| riel | bug reports are always welcome, especially if they prevent other people from running into a bug |
|---|
| riel | ok, a clarification of -rc |
|---|
| riel | between eg. 2.4.16 and 2.4.17, there will be various kernels |
|---|
| riel | the first test kernel will be 2.4.17-pre1, which includes new things |
|---|
| riel | 2.4.17-pre2 would include more new things |
|---|
| riel | after some time, there are enough new things for one release, and it would be time to release 2.4.17 final |
|---|
| riel | but, how do we know the things in 2.4.17-pre<last> work ? |
|---|
| riel | the -rc idea is to release a 2.4.17-rc1 kernel, the first Release Candidate |
|---|
| riel | at this point, the quality should be the same as normal final releases |
|---|
| riel | and the idea is that people will test them, and inform the kernel hackers if some bug is left |
|---|
| riel | if a bug gets found, it is fixed and a 2.4.17-rc2 is released, with the bugfix as _only_ change |
|---|
| riel | if that kernel stays stable and no new complaints come in for some days, a final 2.4.17 kernel gets released |
|---|
| riel | this should prevent the embarassing bugs we've seen in earlier 2.4 kernels ... |
|---|
| riel | ... but the problem is, this only works if people test the -rc kernels ;) |
|---|
| riel | if people do not test the -rc releases, the bugs still will not be found until the final version ;) |
|---|
| riel | ================== end 2.4 ... on to 2.5 ===== |
|---|
| riel | time to go on with the second part of my lecture |
|---|
| riel | a lot of people want to know what is going to happen in the 2.5 kernel |
|---|
| riel | well, I have some bad news for those people |
|---|
| riel | I cannot look into the future ;) |
|---|
| riel | also, no real plans are made for the long term future of Linux, so nobody really knows what will go into the 2.5 kernel |
|---|
| riel | but what I can do is give an overview of some interesting projects which are going on, some of which will maybe get into the 2.5 kernel |
|---|
| riel | the first two projects I'm discussing have already been in the 2.4 kernel for a while, or are still in ;) |
|---|
| riel | but they're still pretty new |
|---|
| riel | the first is UML, or User Mode Linux |
|---|
| riel | this is an architecture, just like Linux for PPC, Linux for S390, Linux for m68k, etc... |
|---|
| riel | but the difference is that this is a port of Linux ... to Linux |
|---|
| riel | now you may wonder to yourself: "if I have Linux, why would I need Linux? I'm already running it" |
|---|
| riel | well, good question indeed |
|---|
| riel | the answer is that User Mode Linux was mainly done because it could be done and because Jeff wanted to do it ;) |
|---|
| riel | but interestingly, since the early implementations people have found some legitimate applications for this beast |
|---|
| riel | for one, you can run multiple "linux virtual machines" on one piece of hardware |
|---|
| riel | this allows you to completely separate the people using a machine from the web server program |
|---|
| riel | or to separate individual users from each other |
|---|
| riel | this can increase security and fairness on a box |
|---|
| riel | another implementation is testing new kernel versions, so when you have only 1 machine, you can run UML to test a kernel before corrupting your real root filesystem when rebooting 2.4.15 ;) |
|---|
| riel | <Bugblue> riel: what are the advantages instead of using 'chroot' or in HP-linux so called: 'compartments' ? |
|---|
| riel | Bugblue: when you do chroot(), it is still possible to do things like sending signals to other processes .. |
|---|
| riel | .. or forking 100 infinite loops to starve the CPU and slow down other processes |
|---|
| riel | with UML your virtual machine will not get more resources than allowed, and you will not be able to slow down other virtual machines too much |
|---|
| riel | <Jae> riel: will UML be added to 2.5 ? |
|---|
| riel | Jae: I think it will ;) |
|---|
| riel | <onki> riel, if you would provide a 'kernel instance' with UML would the admin still have control or would the user have to much freedom? |
|---|
| riel | onki: at the moment, UML is not completely secure yet, but I think that in the future it will be possible to give even hostile users root in their own virtual machine without any danger |
|---|
| riel | <kroks> but you have ulimit for preventing that kind of starving resources |
|---|
| riel | kroks: there are two problems with ulimit |
|---|
| riel | first, most ulimits are per PROCESS, so one user can simply start 100 processes and still eat all resources |
|---|
| riel | secondly, ulimit doesn't fix the security problem that a user can see other processes or try to influence them |
|---|
| riel | =====> time for the next project ;) |
|---|
| riel | another new project, which is in 2.4 already, is Inter-Mezzo |
|---|
| riel | Intermezzo is a fun project by Peter Braam |
|---|
| riel | it is a distributed, replicated filesystem |
|---|
| riel | we are all familiar with NFS, where you can see files over the network and use them like normal files |
|---|
| riel | but this has a number of problems |
|---|
| riel | if the server crashes, or you move your laptop to a conference (away from the server) you can no longer see your files |
|---|
| riel | secondly, NFS over the internet is very slow and insecure ;) |
|---|
| riel | inter-mezzo takes a different approach |
|---|
| riel | files get -replicated-, each machine gets a copy |
|---|
| riel | and when an update is made, it is stored on the local disk and sent to the server |
|---|
| riel | this means that when you travel, you can still access the files on your laptop |
|---|
| riel | and when you get back to the network, it will send the changes you made to the server |
|---|
| riel | <Sorvin> riel : meaning, Inter-Mezzo is just .. well.. Offline Files for linux .. ? :) |
|---|
| riel | Sorvin: well, that depends on which meaning for "offline files" you take, it has been (ab)used for at least 5 different things over the years ;))) |
|---|
| riel | another fun use of inter-mezzo would be in web serving |
|---|
| riel | if your web site is so busy that one server can not handle the load, you could mount the files over NFS and serve them from 2 servers |
|---|
| riel | but this means you'll need 3 servers to do the work of 2 ;) |
|---|
| riel | also, if the nfs server goes down, you're bust |
|---|
| riel | another option would be a set of scary rsync scripts, with all the problems of bugs and out-of-sync pages |
|---|
| riel | inter-mezzo would make things easy, just upload your new web content to one of the servers and the other gets it automatically |
|---|
| riel | and if one server crashes, you're still online |
|---|
| riel | <MCArkan> riel: won't intermezzo takes too much network resources if it duplicates files ? |
|---|
| riel | MCArkan: I'm not sure what algorithm inter-mezzo uses, so I really don't know ... but it does have the advantage of knowing when a file is changed, so it is easy to send over just that one change |
|---|
| riel | <HoraPe> what happens when both the laptop and the server have modified the same file? |
|---|
| riel | HoraPe: in that case, inter-mezzo will detect the problem and tell the user to fix it |
|---|
| riel | this is probably just as well, since you don't want the computer to mess up your files ;) |
|---|
| * riel moves on to the next topic =======> LSM |
| riel | LSM, or Linux Security Modules is a security plugin mechanism first proposed at the 2.5 kernel summit |
|---|
| riel | currently, in the open() system call, the kernel does a number of checks |
|---|
| riel | checking things like user id, file permissions, etc... |
|---|
| riel | the LSM project still has these checks, but adds one extra thing .. |
|---|
| riel | .. if a security module is loaded, it calls the security check function from this security module, optionally even the functions from multiple modules |
|---|
| riel | this means you can change the kernel's security policy without having to patch and recompile the kernel |
|---|
| riel | it also means you can use a number of different security modules together, without having to work out conflicting patches |
|---|
| riel | sarnold: I forgot which projects are already using LSM, could you give us a short list if you're around ? ;) |
|---|
| sarnold | SELinux is built entirely on top of LSM |
|---|
| sarnold | serge's DTE project is built using LSM |
|---|
| sarnold | several people are working on porting Solar Designer's Openwall patches |
|---|
| sarnold | and chris wright is working on porting the vserver project to use LSM |
|---|
| riel | ok, lets go for some questions on LSM, maybe sarnold could even answer them (since he is involved with the LSM project) |
|---|
| riel | <onki> riel: could you explain the difference between iptables and LSM? is LSM arranging security on the user part? |
|---|
| zuez | word |
|---|
| sarnold | ah, yes, POSIX.1e capabilties is also implemented as an LSM module |
|---|
| sarnold | and, we have implemented traditional euid==0 checks in the dummy module :) |
|---|
| gregkh | IPTables changes the way the network packets gets routed through the kernel. LSM provides hooks into the network stack to change policy based on the data, which is much |
|---|
| gregkh | more flexible than ibptables itself. |
|---|
| sarnold | NSA has supplied funding, and probably staff members (though I can't say specifically) to the SELinux project, and much of the LSM framework has been designed to meet the needs of LSM; in short, 'create', no. 'contribute to', yes. :) |
|---|
| gregkh | For instance, LSM should be able to reject packets from a specific program on the machine, or only allow specific programs access to specific network ports. |
|---|
| sarnold | sorvin, mulix -- the open() call is not reduced to simply callouts to module functions |
|---|
| sarnold | for instance -- is calling open() with write permission on a file a security check? sometimes write permission is denied because the file is a read-only medium, such as cdrom |
|---|
| sarnold | mulix, it is implemented mostly as callbacks in security structures |
|---|
| sarnold | you can find the patch at http://lsm.immunix.org/ with all the fun gory details :) (look for security.h. :) |
|---|
| sarnold | mulix: no, LSM will not assist hijacking syscalls. |
|---|
| * riel would like to move onto the next subject |
| riel | well, actually I'm skipping some things because of the time |
|---|
| zuez | you have the floor :) |
|---|
| riel | =====> ext2/3/4 developments and other filesystems |
|---|
| riel | in 2.4 we saw the introduction of 2 journaling filesystems for block devices |
|---|
| riel | reiserfs and ext3 |
|---|
| riel | these filesystems still have some limitations, though |
|---|
| riel | for one, 32 bit block numbers |
|---|
| riel | which means a filesystem size limit of 4 billion blocks, or some 16 TB maximum |
|---|
| riel | while I know I won't be hitting that limit any time soon, other people have gathered enough mp3s (or other data) already |
|---|
| riel | so we will need to fix this |
|---|
| riel | of course there are XFS and JFS, which have 64 bit data sizes everywhere and don't have this problem |
|---|
| riel | but since ext2 and reiserfs exist, people are working on them too to get those filesystems fixed |
|---|
| riel | oh, lots of filesystem questions ... I guess I'll go answer those ;) |
|---|
| riel | <setepo> riel: XFS will be added in 2.5? |
|---|
| riel | setepo: I don't know what Linus will do, but there is a chance XFS will get integrated |
|---|
| riel | <zuez> speaking of filesystems, are you folks planing something like growfs for ext2 partitions? |
|---|
| riel | zuez: it already exists, both resize2fs and parted are able to resize ext2 filesystems off-line |
|---|
| riel | Christoph Hellwig even has a patch to be able to grow ext2 filesystems online, that is, while the filesystem is mounted |
|---|
| riel | <HoraPe> riel, bsd people use something called softupdates, supposed to be a more rational way of ordering writes that has lot of the journaled fs without being so complex, will linux get some fs like that? |
|---|
| zuez | nod. |
|---|
| riel | HoraPe: ok, lets get a few things straight here |
|---|
| riel | first, softupdates and journaling are both pretty complex systems |
|---|
| riel | secondly, while softupdates guarantees that the filesystem comes back in a consistent state, you still need to run fsck once in a while |
|---|
| riel | journaling provides transactions, meaning that only a journal replay is enough to get the filesystem fully correct again |
|---|
| riel | having said that, neither softupdates or journaling is "perfect", you still need to do recovery after a crash |
|---|
| riel | luckily there is a third possibility |
|---|
| riel | netapp's WAFL uses a primitive form of this trick and Daniel Phillips seems to have perfected the theory |
|---|
| riel | it is called "phase trees" |
|---|
| riel | with a phase tree, you have 2 copies of the metadata |
|---|
| riel | one copy which is the currently referenced one, which is fully consistent |
|---|
| riel | and when changes to the filesystem are made, they are done in unused space |
|---|
| riel | once the changes are complete, or at least in some consistant state, all the new metadata is written to disk |
|---|
| riel | and as the last change, the superblock is changed to point to the new metadata tree |
|---|
| riel | this way you always have a consistent filesystem |
|---|
| riel | without the need for any recovery after reboot |
|---|
| riel | changing from the current metadata tree to the new one is called a "phase change" (IIRC) |
|---|
| riel | this also fixes a big problem with mail servers, where you have 200 incoming emails and fsync() is called for each of them |
|---|
| riel | the filesystem can simply let 10 fsync()s complete on the same phase change |
|---|
| riel | with only 2 write ordering periods, which is more efficient than either softupdates or journaling can provide |
|---|
| riel | <ninjalj> what happened with the patent claim against Phillips? |
|---|
| riel | ninjalj: ok, first netapp has never filed a patent claim |
|---|
| riel | secondly, daniel phillips says he already implemented the things described in netapp's patent in 1986, so if netapp filed a claim they would just loose their patent ;) |
|---|
| riel | thirdly, phase trees are a new version of the idea in netapp's patent ... most likely not even covered by netapp's patent |
|---|
| riel | ok, one last filesystem question |
|---|
| riel | <wol> related, there's the buffer cache and the page cache, what is the difference? it seems like the buffer cache is slowly going away? |
|---|
| riel | wol: the buffer cache died in 2.4.10 ;) |
|---|
| riel | wol: block devices are now cached in the page cache, too |
|---|
| riel | wol: and buffer heads are now nothing more than IO descriptors, used to tell the device drivers what to do with pages from the cache |
|---|
| riel | ok ==============> OpenGFS |
|---|
| riel | (I'm skipping drbd due to lack of time) |
|---|
| riel | GFS is a very cool project by Sistina Inc. |
|---|
| riel | it allows multiple machines to share the same set of disks and the same data |
|---|
| riel | this means you can share data without having to use a file server |
|---|
| riel | not only is this good for performance, it also means the system can be more reliable |
|---|
| riel | because there is no file server which can crash ;) |
|---|
| riel | this is a great tool for clustering |
|---|
| riel | you can, again, have a number of web servers get their data directly from the same disks |
|---|
| riel | and if any of the servers goes down, the others can just continue their work |
|---|
| riel | <MCArkan> riel: what's the difference with intermezzo ? |
|---|
| riel | ok, the difference with intermezzo is quite large |
|---|
| riel | with intermezzo, each computer has its own copy of the files and synchronisation is done over the network |
|---|
| riel | with GFS, you have _one_ shared pool of disks |
|---|
| riel | for example, two raid cabinets on fibrechannel |
|---|
| riel | and 5 computers on the same fibrechannel bus |
|---|
| riel | all 5 computers have direct access to the disks |
|---|
| riel | without any of them playing file server |
|---|
| riel | in the case of fibrechannel, GFS does the locking on the disk itself |
|---|
| riel | so it asks the disk to lock disk block #34523 for node A |
|---|
| riel | and node B can have disk block #54534 locked |
|---|
| riel | also, if one node needs a lock held by another node, the disk will ask the node to release the lock, with a certain timeout |
|---|
| riel | the problem is, you really want special hardware for the best performance |
|---|
| riel | though there are some tricks to do GFS with normal hardware, you don't get the full benefit when you do that |
|---|
| riel | it is really a much more high-end, but also more flexible solution than eg. intermezzo |
|---|
| riel | lastly, recently Sistina made GFS a closed source commercial product |
|---|
| riel | this is unfortunate, but just like happened with SSH, immediately an open source project started from the last GPL version |
|---|
| riel | you can find OpenGFS on http://www.opengfs.org/ |
|---|
| riel | ===================> memory management |
|---|
| riel | over the last few weeks, many people have asked me about memory management |
|---|
| riel | and people have asked me to talk about it now |
|---|
| riel | so well ... here it goes ;) |
|---|
| riel | over the last years, Linux always had a pretty simple memory management subsystem |
|---|
| riel | and every stable kernel had unstable memory management for the first year or so |
|---|
| riel | I think this is because there is little or no coordination on VM work |
|---|
| riel | and during a development kernel, all good patches get integrated, even if they conflict with other good patches already in the kernel ;) |
|---|
| riel | because of this, I have decided to work on a memory management subsystem outside of the linux kernel |
|---|
| riel | and I will make patches available regularly |
|---|
| riel | other people have already indicated an interest in working together with me trying to make a consistent VM subsystem |
|---|
| riel | one of the main plans is trying to learn from the past, learning from what other OSes do good and where other OSes fail |
|---|
| riel | one of the things we have noticed is that FreeBSD's memory management has reached a point where people are happy with it |
|---|
| riel | and no changes are planned for the future |
|---|
| riel | to me, this is an indication something must be right |
|---|
| riel | on the other hand, there are a number of shortcomings, |
|---|
| riel | for example, FreeBSD's data structures are so large that the kernel doesn't boot on a machine with 4GB of RAM or more ;))) |
|---|
| riel | so just porting over code from any other OS is not really an option |
|---|
| riel | one thing which is obvious is that we will want to have mappings from physical pages to the page tables mapping them, so-called "reverse mappings" |
|---|
| riel | because this allows us to select pages to evict on a physical basis |
|---|
| riel | this is good if you need to free up pages in certain physical address ranges |
|---|
| riel | which is needed when you want to free up ISA DMA memory |
|---|
| riel | or when you want to assemble large pages so processes can run faster (eg. 4MB pages on Intel) |
|---|
| riel | and also when you need to free memory in a particular node on NUMA |
|---|
| riel | as another side effect, it allows us to simplify the pageout selection code, the point where we choose which page to swap out |
|---|
| riel | and simplifying that should get rid of a lot of the balancing issues we've seen over the last 10 years ;) |
|---|
| riel | other things we want to include are support for large page sizes |
|---|
| riel | better IO clustering |
|---|
| riel | and a big code cleanup |
|---|
| riel | for one, we would like to have process memory fit into the page cache, like normal files |
|---|
| riel | so we can remove all the special code needed to handle swapped pages |
|---|
| riel | and use the same filesystem code for everything |
|---|
| riel | cleaning up the code and making SMP locking simpler is another item on the TODO list |
|---|
| riel | ok, it's almost midnight in Europe now |
|---|
| MJesus | [00:01] <riel> ok, it's almost midnight in Europe now |
|---|
| riel | MJesus: oops, this lecture really took too much time ;) |
|---|
| riel | <wol> you say this will happen outside the main kernel, but will it be intended to go back at some point, or will it remain a separate patch forever? |
|---|
| riel | wol: the idea is to submit it to the main kernel when we (the people working on it) think it is ready |
|---|
| riel | ok, any more questions ? |
|---|
| * riel waits a little bit |
| riel | ok, I guess not |
|---|
| MJesus | perhaps, it will be good a second session ? |
|---|
| riel | thanks for your time, I hope I haven't bored you too much |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| sarnold | thank you riel :) |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| riel | please come back for the other lectures of this year's Umeet |
|---|
| fernand0 | plas plas plas plsa plas plas plas plas plas plas |
|---|
| fernand0 | plas plas plas plsa plas plas plas plas plas plas |
|---|
| MCArkan | :) |
|---|
| fernand0 | plas plas plas plsa plas plas plas plas plas plas |
|---|
| FloodeR | plas plas plas |
|---|
| fernand0 | plas plas plas plsa plas plas plas plas plas plas |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| FloodeR | plas plas plas |
|---|
| FloodeR | plas plas plas |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| onki | :)) |
|---|
| fernand0 | plas plas plas plsa plas plas plas plas plas plas |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| Endymion | :) |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| dreim0n | thx riel |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| MJesus | 4plas 5plas 6plas 7plas 8plas 9plas 10plas 11plas 12plas 13plas 1plas 2plas 3plas |
|---|
| MJesus | 4plas 5plas 6plas 7plas 8plas 9plas 10plas 11plas 12plas 13plas 1plas 2plas 3plas |
|---|
| * anders cheers |
| FloodeR | plas plas plas |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| lucre | ddd |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| paranouei | clap clap clap clap |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| jcopenha | of course.. the day to give a lecture I actually have work to do at my job... |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| dardhal | Good luck with yuor new VM efforts !!! |
|---|
| cox | torero torero torero torero torero torero torero |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| cox | torero torero torero torero torero torero torero |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| * jcopenha pouts because he missed the lecture |
| dreim0n | MJesus: this colors.... |
|---|
| lucre | AL FIN |
|---|
| MJesus | clap clap clap clap clap clap clap clap clap clap |
|---|
| cox | torero torero torero torero torero torero torero |
|---|
| MJesus | 4plas 5plas 6plas 7plas 8plas 9plas 10plas 11plas 12plas 13plas 1plas 2plas 3plas |
|---|
| lucre | HOLA |
|---|
| diego | plas plas plas plas plas plas plas plas plas plas plas plas |
|---|
| diego | plas plas plas plas plas plas plas plas plas plas plas plas |
|---|
| cox | torero torero torero torero torero torero torero |
|---|
| Rawsock | plas plas plas plas plas |
|---|
| * hensema_ throws his underware |
| IRC log ended Mon Dec 3 16:53 |
