What CS skills are needed to do research in Operating Systems?

Pure math guy here. Interested in applied CS. What skills are needed to do research in computer systems / operating systems area at the PhD level?

OS design is the epitome of autism.

How come?

Queuing theory maybe

electrical/computer engineering type stuff

you'll need to understand circuits, digital logic design techniques, computer architecture

computational theory, compiler constuction, etc. will also be useful

excellent- thanks! any recommended textbooks/resources/moocs?

There's a sticky with excellent book tips

great thanks

There's nothing to be gained from studying OSes at the PhD level. Get hired to Google's OS team and work on the Fuschia OS or the Magenta kernel

I'm not that guy but it's extremely detailed work.

An OS needs a kernel, which is essentially a program that runs other programs. In order to pull this off the kernel needs to solve all sorts of scheduling problems simultaneously in a clever way. You want several processes to be able to share processor time, you want IO and memory to be accessed efficiently (they're way slower than CPU caches), you want processes to be able to share memory, you want processes to be forkable, you want to be able to handle multiple CPUs/cores at the same time, etc...

There are no perfect solutions to any of those scheduling problems so you've just got a whole bunch of similar algorithms with different tradeoffs that you've got to familiarize yourself with.

That's still just introductory OS stuff (not even an entire semester of an intro undergrad course on the subject).

Interesting... why is OS usually recommended before doing research in a field like security? What about OS / low level details do security people focus on?

because if you don't know what you're working on, you're never going to make progress, you will only ever at best be able to imitate other people who truly know what they're working on, and that is not a skill, it's a redundancy

Assembly, and CPU design.

Because understanding how a kernel and an OS in general work is important in understanding how to exploit it.

Security is another job that the kernel is in charge of handling. In particular the kernel needs to be able to separate system processes from user processes. Moreover it needs to be able to separate processes (and their memory) from each other. If a user process is able to exploit a flaw in the kernel and gain access to other processes then it can cause devastating results. Moreover if it can access system shit then it basically owns the system.

That's still just basic stuff. Other aspects of OS design/implementation are things like filesystems, files, devices, etc.. (all the other shit that comprises an OS). These things also have to have security and permissions implemented and each one of those is hard as fuck to do right.

None of this requires advanced math but it does require very careful thinking and very detailed work with a lot of moving parts that are at times difficult to keep track of.

Excellent answers. For some reason I find this very interesting. Think I will continue taking CS courses pre-reqs up to OS and seriously consider moving into this area / security for graduate studies. I know it isn't math like I do in my pure studies, but I can always study math on the side.

This is church.

OP, you don't do a PHD in Operating Systems. Maybe something like distributed systems? Good OS design is more of a software engineering thing. There are a lot of interesting TCS problems that someone with a good grip on combinatorics can do.

No no no, I want none of that. I want a kernel to run IO only.

Everything else runs in state machines, one state machine per core.

I fucking hate all the bloat in Linux on the Xeon Phi. So much fucking overhead compared with running on bare metal.

I can craft one nice tightly nit bit of code and have it crunch out results on my two phi's, 144 cores in all. So much nicer than cuda.

>computational theory, compiler constuction, etc. will also be useful
Really? I work in these fields and don't really see the relevance, except perhaps in the gruelling implementation of compilers (bootstrapping and so on).

>research in operating systems
It isn't 1995 anymore, OS aren't relevant in academia or research, it's all down to engineering in that area. It's only taught to undergrads so they understand how their system work.

What relevant areas of research use OS? Security?

Nice meme.

there are still problems to be solved in operating systems, like formally proving the correctness of one :)

It's not impossible though. You don't have to make one that's perfect for it to be compelling research. you just have to make one that does one thing well.

Same thing with compilers. You can make a minimalist one that works in a couple weeks.

go to OSdev forums. Everybody there has their own hobby OS.

Terry Davis's philosophy:
> my compiler only has 30k lines
> those who brag about how many lines there OS has are idiots
> it's a CIA plot to intimidate you

Because every exploiter who is worth their salt is going to understand the ins and outs of OS design. If you know how something completely works, then you can generally understand how to manipulate it for your benefit. It's also much easier to make more money by exploiting the system than it is to figuring out who the exploiter is and patching the exploit.

Chances are highly likely that a good-sized body of blackhats know how a kernel works at, or maybe over, that of a PhD level.

If you are interested in math and security try cryptography