A.1 Intro

We show commands starting with $: this is the prompt. Do not type it.
To look at this text, connect to aisa.fi.muni.cz and type:
```
$ pb152 update
$ cd ~/pb152/00
$ cat intro.txt
```

If you see this in your terminal, you got it right.
One of the most useful UNIX commands ever is man.
To find more about man, type
```
$ man man
```

You will get a pager, which allows you to read longer text in a fixed-size terminal. You can use arrow keys on your keyboard, or pgup and pgdown to move about. To learn more about the pager, type
```
$ man less
```

Manual pages are divided into sections. First comes a very short description of the command (or C function), and then a Synopsis: how is the command (function, etc.) written when you want to use it. Usually, all options are listed (there can be quite a few!).
After Synopsis, you usually get a ‘Description’ which explains what the command does in more detail.
You are not forbidden from using Google, but man is often more efficient.

If you want to learn more about cat, you can type:
```
$ man cat
```

Otherwise, you can continue on to:
```
$ cat grading.txt
```

A.2 Rules of the Game

First of all, I am very sorry in case you lost.
The seminar is worth 2 credits and there is no exam. Instead, you will collect points as we go and when you have 30 of those, you pass. There will be various ways to collect those points. There is only one additional requirement: 4 points must be from teacher review (see below).
Each week, there will be a simple homework (only slightly more complicated than the exercises in the seminar). Working it out and handing it in will get you 2-4 points. More about this later.
In 8 out of the 12 weeks, your homework will be graded by your seminar tutor, and you may get additional points, up to 1/2 of the value of the homework itself. If you are in the ‘self-study’ group, you do not have a seminar tutor – instead, you will gain these points by reading and accepting at least 3 peer reviews. You will get one point for each such assignment.
You can collect 6 points for writing feedback (peer reviews) on your classmates' exercises. But only if you passed those exercises yourself! Each instance is worth 0.5-1 points.

The summary of obtainable points:

$ cat tricks.txt

1.1 A Few Useful Tricks

Before we continue, I will try to teach you a few tricks for working with the command line a little more efficiently. The first one is using the command history.
For instance, if you just typed a long command and now want to type a rather similar long command, it might be useful to hit 'arrow up' on your keyboard. Then you can tweak the command before running it again.
If a while passed since you typed a command but want to run it again (or edit it slightly), you can use ^R (that is ctrl+r) to find it in your history. Hit ^R and type a few letters or a word from the command. If you typed the right fragment but you want an older command than you got, just hit ^R again, until you find the right one.
Using ^R is often faster than hitting 'arrow up' 20+ times.
Sometimes it's faster to type the command again instead of looking it up in your history. Might be also better for your memory.
The other trick is using the TAB key, especially when working with files. We will learn more about files shortly. If you want to read this file again, you could do it this way:
```
$ cat tr<TAB><ENTER>
```
Saves a few keystrokes.

The last two items have to do with the terminal emulator itself. First is about copying and pasting: using ^C and ^V (ctrl+c and ctrl+v) won't work. You could use shift+insert to paste. Or you can use the mouse to both copy and paste. You copy by selecting the text with a mouse and nothing else.
You can also paste by clicking the middle button (the scroll wheel). The same approach (select+middle click) also works with the browser and other programs.
You can look back to what you typed and the output of programs that scrolled off the screen by scrolling in the terminal (e.g. by using the scroll wheel). However, it is sometimes more convenient to use a pager, e.g. less. You can read this file in a pager like this:
```
$ cat tricks.txt | less
```
In this case, this would also work:
```
$ less tricks.txt
```

The last thing that is often useful is to have more than one terminal open. For instance, you can follow these instructions in one and try out the commands or solve the exercises in another. I find it most convenient to put them side by side, so I can see both at once.

That are all the tricks for now. You may want to practice them for a bit -- the next section will be a good chance to do that, because we will work with files. Continue to:

$ cat files.txt

1.2 Files

As you know from the lecture, files are pretty fundamental in most operating systems, especially in those from the POSIX family.
There is a rich set of commands for working with files. You already met cat which prints files. It is called cat because it can con-cat-enate multiple files together. Like this:
```
$ cat intro.txt tricks.txt
```
I will remind you that you can learn more about cat by typing:
```
$ man cat
```

Another very useful command is ls. It is short for ‘list files’. Many commands are shorthands and they are easier to remember if you know what the shorthands mean. You can try ls like this:
```
$ ls ..
```
Of course, you can also consult the man page. I'm sure you already know the command to do that?

Then there is cp for copy and rm for remove. You can use them like this:

$ cp intro.txt ~/pb152-intro.txt
$ cat ~/pb152-intro.txt
$ rm ~/pb152-intro.txt

It would be a good time to create a directory in which you could play without messing up your home directory. Let's use mkdir (from ‘make directory’) for that, and let's also enter it right away using cd (for ‘change directory’):

    $ mkdir scratch
    $ cd scratch

You can create an empty file using touch. This command will also update the modification timestamp on a file if it already exists. This is not very important to us for now. Try:
```
$ touch file.txt
```

If you forget where you are, you can always ask the system to pwd (print working directory):
```
$ pwd
/home/yourlogin/pb152/01/scratch
```
(when a line in an example like above does not start with $, it is not a command -- it is what you should see in the terminal after typing the command that's marked with $)

You can refer to the working directory using . or often nothing at all. Try the following commands:

$ ls .
$ ls
$ cp intro.txt /tmp/yourlogin.txt
$ ls
$ cat ./intro.txt
$ cat intro.txt
$ rm /tmp/yourlogin.txt

What follows is a few philosophical remarks about using the command line. We often shorten ‘command line interface’ to CLI or cli. Hence, continue to:

For now, we will demonstrate some of the things that shell scripts can do. First, we will run some of the commands we already know:

mkdir scratch
cd scratch
cp ../intro.txt .
cp ../../00/grading.txt .

The above two commands copied two of the text files you have seen earlier into our working directory. Let's also copy this shell script:

cp ../files.sh .

You may remember echo from earlier, too. Let's tell the user that we did something:

echo "we are done copying the files"
echo

Let's look at something new. You remember ls which lists files. We can list individual files too, instead of entire directories:

echo listing just files.sh
ls files.sh
echo

Now we will use something called ‘globbing’ -- we will refer to multiple files with a shorthand, using the character *:

echo listing all text files
ls *.txt
echo

One of the useful commands that we didn't mention yet is head, which prints a few lines from a given file. We can tell it how many we want using -n. You can learn more, as usual, in

man
head

echo first line of intro.txt
head -n 1 intro.txt
echo first line of grading.txt
head -n 1 grading.txt
echo

Sometimes, we want to run some commands for multiple files. We could type them out, like we did above. But often, we can use a loop instead. In Python, you would write a for loop. We can do something similar in a shell script:

for f in *.txt; do
    echo "first line of $f (in a loop)"
    head -n 1 $f
done
echo

Notice that we used double quotes in the echo this time. That is because parentheses are special to the shell, but we just want to print them. When you want to print something and you are not sure whether it contains special characters, use double quotes. They won't hurt if they are not required.

echo "no double quotes were needed here"
echo

Since this script is getting long, we will stop and continue in another one. Let's first clean up the files though:

rm intro.txt grading.txt files.sh
cd ..
rmdir scratch

Now you can finally do an exercise for yourself. Let me give you the instructions right away:

echo that is all, instructions for an exercise follow
echo
cat 1.txt

1.5 Exercise 1

You can volunteer to do this exercise ‘at the blackboard’. You may want to review the instructions here: Sadly not this semester, due to COVID-related restrictions. Instead, you can ask questions in the discussion forum in IS or in the online chat. ## Text Editors
```
$ cat ../00/activity.txt
```

Perform the following tasks (using commands you already know):
- change into the directory pb152 in your home: ~/pb152
- list the contents of the directory
- create a directory called scratch
- copy all the text files (but only the text files) from ~/pb152/01
- print first two lines of each of them
- list the directory content again
- remove all the exercise description files, that is {number}.txt
- list the directory once more
- remove all the remaining files
- remove the now-empty directory

It may be helpful to know that you can type a for loop on a single line and use it as a command. Like this:
```
$ for f in *.sh; do ls $f; done
```

Another useful tidbit: ? (question mark) matches a single character when globbing.

That was all for the exercise. The next thing to learn will be text editors, because we will want to write some simple scripts. Go on to:

$ cat editors.txt

Editing text files is a very essential activity, especially when interacting with UNIX. We have two basic options.

The traditional UNIX editor is vi (its modernised, extended variant is known as vim). This editor is hard to master, but very efficient. I encourage you to try learning it, especially if you see yourself as a UNIX person in the future. However, it takes a lot of practice and this seminar is not about text editors.

If you want to use vim, you need to know these: use i to enter text, ESC to stop entering text and when you are not entering text, :wq to save your file and exit the editor and :q! to exit the editor and not save the file. You can open a text file like this:
```
$ vim file.txt
```
Unlike traditional vi, you can use keyboard arrows to move around the file while in insert mode (i.e. when entering text). This makes it almost possible to use vim even if you are a novice. It is up to you.

Your other terminal-based option is micro. This is a very simple text editor, and it is not very efficient. It will be sufficient for this course though. Micro is invoked the same way as vim:
```
$ micro file.txt
```

If you are not at the ‘blackboard’ at the moment, feel free to use graphical text editors like gedit. Those are quite simple and intuitive. Sometimes, combining graphical and command line programs is the most efficient way to do things.

Try editing some files in an editor. When satisfied, you can move on to the second exercise for the day. Like before, you can do it at the blackboard if you wish:

$ cat 2.txt

1.6 Exercise 2

You may want to start by reviewing files.sh. In this exercise, we will write a simple script.

The script should do the following:
- create a new directory called scratch
- enter this directory
- copy all exercise files from ~/pb152/01 (1.txt, 2.txt, ...)
- count how many lines there are in each (try man wc)
- change directory back to where you started (hint: try cd ..)
- erase everything in scratch along with the directory (man rm)

Be careful with rm! The option you are looking for is called ‘recursive’. You should also use option -i if you are not 100% sure! This will ask about each file to erase before doing so.

Run the script. If you do not remember how, this was mentioned near the top of files.sh.

We will need to review pipes before we go any further. Hence the way forward is:

$ cat pipes.txt

1.7 Pipes

It will be a few more weeks before we get to describe pipes in the lectures. For now, we can consider them to be magical.
The way we will use pipes is to connect programs together in a simple way.
If you remember, it was mentioned earlier that you can run a file through a pager like this:
```
$ cat file.txt | less
```
This is a pipe. The entire command is sometimes called a ‘pipeline’.

Pipes take output of one command and present them as input to another. Most UNIX utilities are designed to work well with pipes.
If you want to count all the lines in all the text files in your current directory, you can say:
```
$ cat *.txt | wc -l
```

One useful command to use in pipelines is cut (try man cut). You can also try the following command:
```
$ echo hello:world | cut -d: -f2
```
The switch -d gives the delimiter, -f asks for a particular delimited field. You can use ranges too, like -f 2-4.

Pipelines are often used for working with text. A very powerful tool is grep. It allows you to use regular expressions to select a subset of lines that match a pattern. We will only use it in a simpler version, without regular expressions for now. This variant is called fgrep (where f stands for ‘fixed’). To print all lines from file.txt that contain the word frob, you could run:
```
$ fgrep frob file.txt
```
Or with pipes:
```
$ cat file.txt | fgrep frob
```

I will show you one last command for today, and that is sort. This is also quite useful in pipelines. It rearranges input lines into alphabetical order.

To practice working with pipes, check out 3.txt.

1.8 Exercise 3

This is a simple exercise for pipes. It is also interactive: you do not need to write a script.
Perform the following tasks:
- make a copy of oslist.csv
- look at the content using cat
- print names (and only names) of all operating systems made by Apple
- print vendors of all microkernel operating systems
- print names and vendors of operating systems sorted by year
- erase your copy of oslist.csv

Now let's move on to the final exercise for today. What to do is described in 4.txt.

1.9 Exercise 4

The final exercise for today is to chain a few commands into a single pipeline that takes all the text files in today's tutorial and lists all the command names (programs) that appear in the examples, one per line. Each command should appear exactly once. For instance, fgrep, echo or less should be listed (among quite a few others).

That's all. The homework is described in hw.txt. Good luck and have fun.

1.10 Homework

To submit the homework, type pb152 submit in this directory. The first deadline is Monday next week (8.3.) at midnight.

The exercise is to write a shell script (save it as hw.sh). The script should do the following:
- create a directory called scratch
- change into the directory
- make a copy of all shell scripts from ~/pb152/01 (files ending .sh)
- make a copy of oslist.csv from ~/pb152/01
- count operating systems by Apple, Google and Microsoft
- print the vendor of the most recent operating system
- count the number of different vendors (man uniq)
- change back to the original directory
- erase everything in scratch and the directory itself

The script will not be running in ~/pb152/01. Use the entire path to refer to that directory, using ~ or $HOME. Do not use your login in the path!

The expected output would be:

OS count by Apple:
3
OS count by Google:
2
OS count by Microsoft:
3
Latest operating system by:
Google
Number of different vendors:
7

Please follow the above output format exactly.

That's it for today. See you next week!

2 Descent into C

Hopefully, you remember how to use the shell and how to browse these documents from last week. If you need to review, just run:
```
$ cat ~/pb152/00/intro.txt
```
The lecture for today, however, starts here:
```
$ cd ~/pb152/02
$ cat intro.txt
```

Today, we will get acquainted with the C compiler, the linker and we will write some very simple C programs.
The traditional UNIX name for the C compiler is cc. On most Linux systems (this one included), the default C compiler is known as gcc (short for GNU cc -- this is mainly because you could install gcc on computers that already had their own system cc).
You can usually run gcc using the command cc whenever it is the default compiler. I would actually recommend that you do it this way, unless you require the GNU compiler specifically. Not all modern UNIX systems use gcc as the default compiler, sometimes for a good reason.
You can read the documentation for how to run gcc using:
```
$ man gcc
```
Unfortunately, on this computer man cc does not work.
The linker is available as ld (probably from Link eDitor, some people think it is from LoaDer, but that's more tenuous). See also man ld.
The linker takes multiple binary artifacts (object files, libraries) and produces a single file which contains an executable program. How this works will be explained in more detail in the lectures.
We will not run the linker directly here. This is rarely needed or useful even in the real world.

You can continue by looking at hello.c:

#include <stdio.h>

The stdio.h header contains useful functions that let us print things to the screen. They are part of libc and use operating system facilities to do their work, like system calls. The functions are really complicated on the inside, but we don't have to worry about that yet. There is only one other thing in this very simple C program, and that is the main function. Every C program has one, but not every C source file does. We will learn about that later. When you run a C program, the operating system (via its libraries) ensures that the main function starts executing.

int main()

This is a simple function definition: it has the return type (int), the name of the function (main), the argument list (empty parentheses) and the body (starting on the next line).

$ cat compiler.txt

2.2 Compiler

Going from a source (text) file to something a computer can execute is a fairly complicated process. What we normally call a ‘compiler’ is, in fact a driver: a program that supervises the process.
The commands cc and gcc actually invoke this driver.
The usual steps the driver invokes are as follows:
1. preprocessor: cc -E
2. compiler proper: cc -S α. lexical and syntactical analysis β. semantic analysis and type checking γ. optimisation δ. assembly (machine code) generation
3. assembler: cc -c
4. linker: cc

The commands in the parentheses above are how to invoke that particular stage. Let's try:
```
$ cc -E -o hello.i hello.c
$ micro hello.i
```

The .i file contains the program after the first step of the compilation process.
All preprocessor directives have been processed. Those are mainly #include but also #if/#else/#endif and #define macros.
You can find our main function near the end of the .i file.
Now try:
```
$ cc -S hello.i
$ micro hello.s
```

And finally:

$ cc -c hello.s
$ objdump -r --disassemble hello.o

Let's move on to a slightly more interesting program.

2.3 `read.c` (source code)

This is a simple program that uses the POSIX API to work with a regular file. As usual, we start with #include statements. In this case, we keep the familiar stdio.h but add unistd.h. You may recall from the lecture that the latter contains prototypes for basic POSIX functions. Additionally, we will need one more #include to get access to the open function, since it is not part of unistd.h.

#include <stdio.h>   /* printf */ 
#include <unistd.h>  /* read, write */
#include <fcntl.h>   /* open */

Since this is a simple program, it only has a single function: main. As you may remember, this is the entry point -- execution starts here.

int main() 
{

We will start by opening a file. The function open takes 2 arguments: the path to the file that we wish to open, given as a string, and the mode, which tells the OS what we intend to do with the file later. We are only interested in reading, so we will say O_RDONLY which is short for ‘open read only’.

    int fd = open( "read.c", O_RDONLY );

fd is short for file descriptor, and this name is used quite commonly. The variable is of type int: it is specified by POSIX this way. We should check whether the open call worked as we expected it to, so let's do that. If we look at man 2 open and scroll a bit, we will find a section that's named return value:

    close( fd ); /* let's not forget to close the file descriptor */

With that out of the way, a number of things can happen. Let's turn to man 2 read. It tells us that if we get a positive number back, that is the number of bytes that were actually read. This will never be more than the third argument, but could be less for various reasons. For instance, the file may be shorter than 450 bytes.

    if ( bytes < 0 ) 
        return 2; /* we don't care much about errors just yet */

We now use the familiar puts function to print the data we obtained from the file. Here it is:

    puts( buffer );

That's all. Can you say what the program prints before you run it?

    return 0; 
}

We learned how to compile programs just a short while back, so perhaps you can do that on your own. But in case you forgot, let me remind you:

$ cat 1.txt

2.4 Exercise 1

Let's dust off some of the knowledge we gained last week. Let's write a shell script. Since we learned some new commands today, let's also use those: we will compile a program.
Write a shell script, compile.sh that does the following:
- create an empty directory, called scratch
- enter this newly created directory
- make a copy of read.c
- preprocess read.c into read.i
- compile read.i into read.s
- assemble read.s into read.o
- link read.o into a.out
- run the resulting program, a.out

Normally, most of the steps would be done by cc automatically. But we are trying to learn about how cc works, which is why we do all those extra steps.
To compensate for making you do these boring steps by hand, I'll teach you a cool trick. Write set -x near the start of your shell script (before any other commands it runs). This will make sh print every command as it is about to be executed by the shell script.

As usual with those exercises, feel free to volunteer. You may want to review the instructions here:

$ cat args.c

2.5 `args.c` (source code)

We will need to make a small detour from the filesystem. You remember how you can write cat file.txt to have cat work with file.txt? We call the things that come after a command ‘arguments’ (or sometimes, to be explicit about it, we call them ‘command-line arguments’). We will now learn how to work with those in our C programs. For now, we are back to only needing our favourite #include:

#include <stdio.h>

And again, this is a simple program, with only a main function. But now we will give main some (function) arguments:

int main( int argc, const char **argv )

By convention, those arguments are called argc and argv. The first is an integer and tells us how many command-line arguments there were. But be careful: this includes the program name itself. So for instance:

$ ./a.out file.txt

would give us argc = 2. The other (argv) is more complicated. Let's not worry about that for now. It is always OK to write it as it is written here.

We start by checking how many arguments are there. If there aren't any interesting ones, we give up right away.

    if ( argc < 2 )

It would be nice to tell the user what went wrong -- we will do that another time.

        return 1;

If we made it here, we know that there's at least one argument, in addition to the program name. We don't know much about C yet, but I'll tell you that the way to get the first argument is argv[1]. It is a string, just like

"Hello
world!"

was.

    puts( argv[1] ); 
    return 0;

This program was hopefully simpler than the last one. Unfortunately, it uses some mysterious syntax and it'll take us a few weeks to decipher that properly.

You can run this program by issuing these commands:

$ cat syslibs.txt

2.7 System Libraries

We will first look at some of the system libraries we have mentioned in the lecture. Let's look at /usr/lib first:
```
$ ls /usr/lib/*.a
$ ls /usr/lib/*.so
```

I want to show you an interesting new command. It is called file and its job is to identify the type of a file based on the content (not the file name). Let's try:

$ file /usr/lib/libc_nonshared.a
/usr/lib/libc_nonshared.a: current ar archive
$ file /usr/lib/libc-2.17.so
/usr/lib/libc-2.17.so: ELF 32-bit LSB shared object, [...]

Using file, we learned a few things. Even before that, however, we notice that the file names are all wrong. Unfortunately, there is no libc.a on aisa, because many people consider static libraries obsolete. The libc_nonshared.a is a helper library that only contains a small fraction of libc.
The shared version of the library has a version number in the file name. This is unusual, but libc on this computer is somewhat special. If you are interested, you can peek at the file named libc.so, which is not an actual shared library, as confirmed by file:
```
$ file /usr/lib/libc.so
$ ls -l /usr/lib/libc.so.6
```

The text file libc.so instructs the linker to look for libc.so.6, which is a symbolic link (we don't know those yet, but think of it as an alias for a different file). It points to our libc-2.17.so.
The other kind of versioning -- the 6 in libc.so.6 is common and this is how most shared libraries are. It means that this is the 6th major revision of libc.
The second thing we learned is that the shared version of libc is 32-bit. But surely, aisa is a 64b machine? Yes, and for historical reasons, the actual system libraries on aisa are located in /usr/lib64. I am sorry, life is complicated.

This was a little disappointing. Let's move on to making our own (static) libraries instead:

$ cat archives.txt

2.8 Archives

You might remember that static libraries are basically zipped-up object files. Hence, we will need some object files to make a static library.
I prepared two very simple math functions, each in its own C file. You can have a peek if you like:
```
$ cat factorial.c
$ cat fib.c
```

Let's compile those into object files:

$ cc -c factorial.c
$ cc -c fib.c
$ ls *.o

And make a static library out of them:
```
$ ar r libmath.a factorial.o fib.o
```

And we can use the simple math.c I prepared for you for our main function:
```
$ micro math.c
$ cc -c math.c
$ cc math.o libmath.a
$ ./a.out
```

It might be interesting to see what happens, if we try to link our math.o without the library:
```
$ cc math.o
math.c:(.text+0xe): undefined reference to `fib'
collect2: error: ld returned 1 exit status
```
The linker tells us that it did not find a definition for fib and cannot produce an executable for this reason. We also see that the compilation driver in cc invoked ld internally, and told us that ld failed with an exit code 1.

Now we should be ready to work out exercise 3:

$ cat 3.txt

2.9 Exercise 3

In this exercise, we will practice working with object files and archives (static libraries) a little.
We will start by creating an empty directory. Let's call it ex3.
In that directory, let's create a 3-part program, made of main.c, hello.c and bye.c. We will also need greetings.h.
In each of the .c files, there will be a single function. In hello.c, it'll be void hello(), in bye.c it will be void bye() and in main.c, it'll be int main().
Our greetings.h will look like this:
```
void hello( void ); 
void bye( void );
```
There will only be those two lines. We can use greetings.h by using an #include statement, like this:
```
#include "greetings.h" 
```

Each of hello() and bye() will simply write a message using puts. You can choose the message. And main() will simply call hello() and bye(), in this order.
Now let's build our program in stages. We will want main.o, bye.o and libhello.a. Then link them all together to form a.out, as usual.
Check that a.out works as expected.

We can now go back to reading and writing files in C. Let's have a look:

$ micro stdio.c

2.10 `stdio.c` (source code)

In this program, we will work with ‘standard input’ and ‘standard output’. We will also look at the relationship between puts and write. As usual, we need some #include statements:

#include <stdio.h> 
#include <unistd.h>

Notice that we did not #include <fcntl.h>. We will not need open.

int main() /* we won't need command line arguments either */ 
{

First we print a message using puts, as we did before.

    puts( "hello from puts" );

We now need to take a small precaution. Near the start of today's lecture, I told you that puts does complicated things on the inside. One of those things is called buffering: it will collect characters that you asked it to print in a piece of memory (a buffer) instead of printing them right away. It will print the buffer only when it fills up, or when we ask it to do that explicitly. We can use fflush for that. The stream into which puts sends the string is called stdout. A stream is basically a file descriptor along with the buffer.

    fflush( stdout );

Internally, what fflush does (on a UNIX system) is call write with the content of the buffer that was stored away by puts. The buffers are always flushed by the standard library when the program exits normally. We can also try to call write directly. The file descriptor that hides behind the stdout stream is also called stdout and is, by convention, assigned the number 1. We could also call it STDOUT_FILENO, but we don't need to bother. The 18 in the third argument is, however, more of a problem. We will let it slide for now. The \n in the string means a newline -- unlike puts, write won't add that for us.

    write( 1, "hello from write\n", 18 );

We will try to read from stdin now, and we need space for that, so let's make some.

    char buffer[20] = { 0 };

stdin is file descriptor 0. Let's try reading from it.

    ssize_t bytes = read( 0, buffer, 19 ); 

    if ( bytes >= 0 )
    {

And just print back whatever we got back to the user.

        write( 1, "what we got from stdin:\n", 24 ); 
        write( 1, buffer, bytes );
    }
    else
    {

Something failed. I promised earlier that I will tell you what to do with errors. This is not very pretty, but since we are using write already, let's stick with that. The fd number 2 is called stderr and that's where we send error messages.

        write( 2, "reading stdin failed :(\n", 24 ); 
        return 1;
    }
}

Now compile and run this program as follows:

$ cat redirects.txt

2.11 Redirects

We have already seen pipes last week. But pipes are not the only way to redirect inputs and outputs of programs. The other is redirections, which work with files.
A simple thing to try is redirecting the output of cat, like this:
```
$ cat hello.c > redirected.c
$ cat redirected.c
```

We see that the output that cat would normally send to the screen, or to another program using a pipe, was written into the file redirected.c. This was easy.
In the other direction, we can read things from files instead of from the keyboard, or from other programs via pipes. Consider:
```
$ cut -d, -f3 < ../01/oslist.csv
```

You see that the input of cut came from the file we specified using <.
Let's try something slightly different. There are special files on UNIX, and one of those special files is /dev/null. This is a file that cannot be read and throws away anything you write to it:
```
$ echo hello > /dev/null
$ cat /dev/null
```

We still have the a.out which came from stdio.c. Let's try redirections with that one:
```
$ ./a.out < /dev/null
$ ./a.out < /dev/null > /dev/null
```

Can you explain the output?

$ ./a.out < read.c

2.13 Homework

There will be two parts to the homework.
1. Write a C program, write.c.
2. Write a shell script, test.sh.

From this point on, use relative paths unless directed otherwise. The script should do its work in the directory in which it is executed. This will also apply to future assignments. Do not assume existence of files, outside of those submitted, that you did not create.

The C program will work as follows:
1. It will read (at most) one kilobyte of data from its standard input.
2. It will (create) and write this data into a file that is given to it as the first argument.
3. It will complain using stderr if something fails, and return a non-zero exit code.

A hint: to create and write into a file, you will want to use these flags to open: O_RDWR | O_CREAT. This is the second parameter. You also must pass a third argument to open for this to work. This will be 0666 (a number) and will be explained later.

Now for the shell script:
- compile write.c into a.out in the 4 stages
- the files write.i, write.s and write.o should exist
- use a.out to make a copy of write.c under copy.c
- show that a.out prints an error when writing fails (try to write into a directory instead of a file, or to create a file in a directory that does not exist, or somewhere where you don't have the permission to create files, say /bin)

That is all. Happy hacking.

$ micro variables.sh

3.1 `variables.sh` (source code)

This is a shell script that will demonstrate basic variable substitution. We will be ahead of the lecture a fair amount, but the things we will do are not hard. I hope you remember pipes from the first week (we also used those in the second week, they are written as program1 | program2). Let's also ask the shell to tell us what is going on. We saw this last week:

set -x

There are two types of variables that we are interested in. First are shell variables, the second are environment variables. There is a fair amount of overlap, which can be confusing. We need to be careful about the distinction.

variable=value

The above syntax sets the value of VARIABLE to be value. In a traditional shell, there are no types: everything is a string. There must be no spaces around the = ‘assignment operator’. To use a variable, we need to use special syntax. Compare

echo variable
echo $variable

You can also set a variable to the output of a program. Incidentally, the $(command) construct can be used in other places too, we may encounter that later.

variable=$(echo hello from echo)
: $variable

The command : does nothing. It is useful when we are using

set
-x

because it shows up in the trace, along with all the arguments (which are $-expanded). You should run this script now. The invocation is:

$ sh variables.sh

: "That's it for now. Continue to: env.sh"

3.2 `env.sh` (source code)

As I just mentioned, there are 2 types of variables floating around in UNIX. The other are the environment, a set of variables passed around to each program. In a short while, we will write a C program to look at them, so you see that they are not the same as shell variables. In any case, the program env prints all environment variables. You can run it like this:

echo '# looking for PATH in the environment'
env | fgrep PATH

Shells typically make it possible to add their own variables into the environment, and they make all environment variables available as their own (via $ expansion).

echo '# doing echo $PATH'
echo $PATH

Adding variables into the environment is done with export:

echo
echo '# setting VAR, echo-ing it and looking for it in env'
VAR=17
echo $VAR
env | fgrep VAR

echo
echo '# exporting VAR, echo-ing it and looking for it in env'
export VAR
echo $VAR
env | fgrep VAR

echo
echo '# changing the value of VAR'
VAR=13
echo $VAR
env | fgrep VAR

echo
echo 'That said, let us go on to getenv.c'

3.3 `getenv.c` (source code)

This is a C program, I am sure you remember those. They start with some #include statements -- you are also familiar with stdio.h, but we now encounter a new one: stdlib.h. This is because we want to use the standard C function getenv which is declared in this header.

#include <stdio.h>  /* printf */ 
#include <stdlib.h> /* getenv */

int main( int argc, const char **argv )
{

You probably remember how command-line arguments are passed into C programs. The argc and argv arguments to main contain the number of arguments and the strings representing them. That said, argv[0] is the name of the program itself. So we only care for argv[1], argv[2] and so on. What follows is a for loop: you don't need to be able to write one, but hopefully you can read the one here without much trouble. It just runs its body (the printf) for every value of i between 1 and argc (the latter not included).

    int i; 

    for ( i = 1; i < argc; ++i )
    {

We have seen printf before, too, when we printed numbers in our simple math programs last week, but we always used puts to print strings until now. Well, using printf to do this is often more convenient. Like with the for loop, for now you only need to be able to read what this does and not write your own printf calls.

        printf( "%s → %s\n", argv[ i ], getenv( argv[ i ] ) ); 
    }
}

In case you forgot, compile this with:

$ cat 1.txt

3.4 Exercise 1

Write a shell script that does the following:
1. count and print how many environment variables are set
2. set the variable ENV_SIZE to this value
3. make ENV_SIZE part of the environment
4. compile getenv.c from earlier (in 1 step)
5. run it with ENV_SIZE as argument

That's it, now we can move on to test.sh.

3.5 `test.sh` (source code)

This shell script demonstrates basic control flow: conditional statements, and how to write some basic conditions that you may want to test. The basic structure is simple:

if condition; then
commands; else commands; fi

. We can either use semicolons or newlines. Different people prefer different styles. The condition part of the if statement is a command. If the command is successful (i.e. if it returns exit code 0), the then branch of the conditional is taken. Otherwise, the else branch is. The else branch is optional.

echo 'we first try some if statements and the ‹test› program'
echo

The program that is most often used in if conditions is called test. You can use it to check basic properties of files (existence, whether they are regular files or directories, if one file has been modified more recently than another and so on). It can also test properties of numbers (basic relational operators) and strings (emptiness and equality). A few examples follow.

if test -e file.txt; then
    echo file.txt exists
fi

if test -f file.txt; then
    echo file.txt exists and is a regular file
fi

if test 3 -gt 4; then
    echo "3 is greater than 4"
fi

if test 3 -lt 4; then
    echo "3 is less than 4"
fi

There is another way to call test, and that is [. Yes, there is a command with the name ‘left square bracket’. It is usually a (sym)link to test. In some shells, it is also built-in (both test and [).

if [ -e file.txt ]; then
    echo file.txt still exists
else
    echo file.txt does not exist
fi

The closing square bracket is an argument to [ and it is required: [ will complain if you don't supply it, like this:

$ micro chaining.sh

3.6 `chaining.sh` (source code)

In this script, we will demonstrate how commands can be chained. This is sometimes useful in if conditions, but also in other scenarios. We already know pipes: those are a form of chaining too. But we are now interested in simpler things. You know that if you enter two commands on separate lines (in a script, or in an interactive session), those commands will run in a sequence. A semicolon does the same thing:

echo foo ; echo bar

That is not very interesting, but sometimes useful, if you want to run a sequence of commands at once and each of them takes a while and you don't want to watch for each to finish so you can type in the next one. The next two ways are more interesting: the && and || operators:

true  && echo "this command will run"
false && echo "this command will not run"

Just like in C, && is short-circuiting: if the first command fails, the result must always be false and running the second command is not needed. Same (but opposite) with ||:

true  || echo "this command will not run"
false || echo "this command will run"

We can form a longer chain, too:

true  && echo "this will run"  || echo "but this won't"
false && echo "this won't run" || echo "but this will"

Of course, you can use those operators in conditionals:

if true && false; then echo "not happening"; fi

if test -f file.txt && test -f file.bak; then
    echo "both file.txt and file.bak exist and are regular files"
fi

This is all I wanted to say about chaining for now. We might look at the & operator later in the semester, if there's enough time. For now, let's move on to loops:

$ micro loops.sh

3.7 `loops.sh` (source code)

We have already seen simple for loops, even though we didn't quite understand variables. For loops are quite simple:

for var in x y z; do
    echo $var
done

By now, you should understand what echo $var means, and the for loop simply means this: set var to x, run the body of the loop, then set it to y, run the loop again and then set it to z and repeat once more. The basic structure is like with if: you can use semicolons or newlines, as you prefer. Of course, the above form of a loop can be useful, but the things to iterate will often be files. You may remember that we can use globs (patterns) to match multiple files. This works universally, not just in a for statement:

echo *.mk
echo

for file in *.txt; do
    if test $file -nt make.txt; then
        echo "$file is newer than make.txt"
    else
        echo "$file is older than make.txt"
    fi
done

The last form of a loop that I want to show you is a while loop. It is written like this:

echo
touch scratch.txt
while test $(wc -l < scratch.txt) -lt 5; do
    echo . >> scratch.txt
done
wc -l scratch.txt

We can also pipe things to and from loops:

for i in 1 2 3 4 5; do
    echo .
done | wc -l

echo

Especially useful (when piping things into loops) is the

while
read

construct. The read command is provided by the shell to read inputs and put them into variables. command | read var is similar to var=$(command). You can try running:

$ cat 2.txt

3.8 Exercise 2

Write a shell script, that:
1. creates and enters a scratch directory
2. writes lines file1 through file5 into list.txt
3. creates (touches) all files listed in list.txt
4. renames each fileN to fileN.x (use globs)
5. rename each fileN.x to fileN.y (use list.txt)
6. clean up

A small hint: For b., use for i in 1 2 3 4 5. You can also use seq 5, like this: for i in $(seq 5). If you use seq, you can easily make more files without typing yourself to death.

Go on to make.txt:

 $ cat make.txt

3.9 `make`

We have built C programs before by hand. This is OK if they are small.
But the compiler is not the fastest program in the world. We often want to compile and test our programs after modifying them.
If the program is big (it has many C files), it is wasteful to compile every one of those every time we want to test.
This is where make can help us.
We would like to only compile files that have been modified since their respective object files were built.
For instance, we have foo.c, bar.c and baz.c and we build the program, so we also have foo.o, bar.o and baz.o. We edit foo.c and want to build the program. We only need to compile foo.c and link the object files.
However, consider there is a header file, which is included by bar.c and baz.c. If we edit this header, both bar.o and baz.o need to be rebuilt. Do you understand why?
The program make can deal with exactly those problems. It reads a makefile -- a script of sorts, in a special language -- and only updates files that need to be updated based on this makefile.
The makefile can be called anything, and if we have build.mk, we can tell make to process it like this:
```
$ make -f build.mk
```
However, if the name is makefile or Makefile, it is enough to say:
```
$ make
```
You can try renaming any of the subsequent makefiles that we encounter to makefile (or Makefile) and run make without -f.
Now you can continue to build.mk.
```
$ micro build.mk
```

3.10 `build.mk` (source code)

This is a makefile (a script for the make program) that builds a simple C program. You should make a copy of this file, along with getenv.c and you can then build getenv using this command:

$ make -f build.mk Like shell, make has variables. We will use them from the start, because they are very useful. One of the common variables we will encounter is CFLAGS, which is what make passes to the C compiler. We can use the += operator to add things to existing variables. Here, we will tell the compiler to emit additional warnings and to switch to C99 mode.

CFLAGS += -Wall -std=c99

Make uses rules, which describe how to build files from other files. In make, the rules have 3 parts: the target (what we want to build), followed by a colon and a list of ingredients that are required to build the target (so-called dependencies or prerequisites). On the next line, a sequence of shell commands follows. We will get to those in a short while. The first rule in the file is special, because it is the one that executes when we just ‘run’ the makefile. It is often called all, so the command then looks as make all or just make. In our case, to make all, we ask make to build getenv but nothing else. This is written so:

all: getenv

In this case, all is a phony target: it is not a file. Most targets in a makefile are, however, files. We will only deal with files from now on. For instance, let's describe how to build getenv.o from getenv.c:

getenv.o: getenv.c
	cc $(CFLAGS) -c getenv.c

The ‘body’ (the shell commands) of the rule is like a small shell script. But you can't use shell variables, because make runs each command in a new shell. Each line of the body starts with a <TAB> character. Using 8 spaces does not work. Please take note that variable expansion works differently in make than it does in shell: we write $(VAR). In a shell, this would mean ‘run the command VAR’, but is just standard variable expansion in a makefile. While we talk about variables: make variables are not environment variables. Unlike in a shell, we also can't get environment variables using $-expansion. Sorry for the detour. Now we also want to tell make how to build getenv from the object file:

getenv: getenv.o
	cc -o getenv getenv.o

That is all. You should be able to write a simple makefile yourself. Try 3.txt.

3.11 Exercise 3

Write the programs (or take them from your solutions from last week) from points 1-6 of ../02/3.txt (i.e. exercise 3 from the last week)
Write a simple makefile that tells make how to build hello.o, main.o and bye.o. We will not build a library at this time.
Make it so that a binary called hello is built from these 3 object files.
Make sure that greetings.h appears in the prerequisites where it is required.
Use touch on various source files and check that make does the right thing.

You can now continue to 4.txt.

3.12 Exercise 4

Write a shell script that mimics what the makefile from previous exercise does. That is, compiles the three object files and links them, but only does those things if either the output files do not exist, or the inputs have changed since the outputs have been generated.

3.13 `suffixes.mk` (source code)

We start with a rule to make getenv again:

getenv: getenv.o
	cc -o getenv getenv.o

We also add -std=c99 to CFLAGS:

CFLAGS += -std=c99

Out of the box, make knows how to compile C sources (.c files) into object files (.o files). This time, however, we will tell it how to do that anyway. The traditional way to tell make that .c and .o are suffixes (which are special for make) is this:

.SUFFIXES: .c .o

If .c and .o are suffixes, we can write special rules (called inference rules) which tell make how to go from any .c file to a corresponding .o file like this:

.c.o:
	cc -c $(CFLAGS) -o $@ $<

We have used some magic variables there. The $@ is the name of the target to make and is defined in any rule. The $< is the name of the prerequisite (the .c file in this case) and may or may not be available in standard (non-suffix) rules.

3.14 `autodep.mk` (source code)

This file is a makefile again. You can run it using

make -f
autodep.mk

CFLAGS += -std=c99

Building programs in C poses some special challenges when header files are involved. Recall that the right hand side of a make rule lists the ingredients that enter the build process. For C programs, this list should contain any header files that are included: if the header file changes, we should rebuild all object files that could possibly be affected by the change. But maintaining such lists is tedious and error-prone. Nobody wants to do it. Fortunately, compilers and make offer some features to automate this tedious task. When compiler builds an object file, it can also produce a makefile fragment that describes what header files were used in the process. The flags to do that are -MD -MP. This makefile fragment will be stored in a file called file.d (assuming we build file.o). You can read more in man gcc.

CFLAGS += -MD -MP

Being able to refer to all sources (and object files) that entail a single program is often very useful. We will call the list of sources SRC and the list of object files OBJ. But we are lazy programmers and we do not want to repeat ourselves. We just list the source files and tell make that the object files are the same files, but with .c replaced with .o. Notice the syntax: we use curly braces here. The syntax is ${VAR:old=new} where old and new are suffixes: each whitespace-separated part of the variable is checked and if it ends with the string old, that part is replaced with new in the output.

SRC = getenv.c foo.c
OBJ = ${SRC:.c=.o} # getenv.o foo.o
DEP = ${SRC:.c=.d} # getenv.d foo.d, see below for an explanation

getenv: $(OBJ) # we need all the object files
	cc -o getenv $(OBJ)

Recall that make already knows how to build .o files from .c files. It also knows to use $(CFLAGS), so it will use our special flags that we specified at the start and cc will generate a .d file for each .c file. This .d file will contain dependency information (in makefile format). We add another (phony) rule, so we can do make clean to remove all the files we built and start over if we wish.

clean:
	rm -f $(DEP) $(OBJ) getenv

The final ingredient is the -include directive, which causes make to include rules from other files. In this sense, it is quite similar to the #include directive from C. The - at the start means that it is not an error if some of the files do not exist. After you run this makefile, you may want to inspect the file getenv.d using a text editor.

-include $(DEP)

That's it. You should be ready to work out the homework now, described in hw.txt.

3.15 Homework

Write a makefile that builds a static library and a program using automatic dependencies (.d files should be created during compilation). Use libmath.a from last week, along with the math.c we had which used it. The name of the resulting program should be math. It should be possible to (at least) run make libmath.a and make math.

$ micro stat.c

4.2 `stat.c` (source code)

We will try to reproduce (part of) what our shell script did in C. If you didn't open man 2 stat yet, now would be a good time. We start by using the #include statement from the manual page (and add stdio.h for a good measure):

#include <sys/stat.h> 
#include <stdio.h>

Let's go straight for main().

int main() 
{

First, we need a place for the OS to put the information about the file(s). This is an aggregate type, with a number of fields in it which correspond to the items printed by the stat user command. The type is called struct stat:

    struct stat st;

In the man page, we immediately notice that there is a function called stat which takes a path, and a function called fstat which takes a file descriptor. This is a common pattern. For now, we will stick with the version that uses paths:

    if ( stat( "intro.txt", &st ) == 0 ) 
    {

A quick glance at the ‘return value’ section of the manpage reveals that 0 means success. Anything else would be an error (but the only other thing that we can get should be -1).

        printf( "stat() on intro.txt returned OK:\n" );

If the above stat call succeeds, it fills in the struct stat that we passed to it. The fields are described in detail in man 2 stat, we will print some of them here:

        printf( "size: %lu\n", st.st_size ); 
        printf( "inode: %lu\n", st.st_ino );
        printf( "links: %lu\n", st.st_nlink );
        printf( "uid: %u\n", st.st_uid );
        printf( "gid: %u\n", st.st_gid );
        printf( "mode: %o\n", st.st_mode ); /* incl. permissions */
    }
    else

This would also be a good time to introduce the function perror, which is short for ‘print error’. Often, if a system call (like stat above) fails, we want to tell the user that it failed and how. This is what perror does.

        perror( "calling stat on intro.txt" );

We could do the same thing for another file, but I think you get the point. This was relatively easy, but also hard to compare with the shell code. If all we wanted to do was print the info for the user to read, it was a lot more work in C. If we wanted to act on the information, doing that in shell would be more difficult.

Before continuing with directories, let's make a very brief detour to errno.c. The perror above should not be called, but we want to know what it does.

4.3 `errno.c` (source code)

This is a very simple program that demonstrates what perror does (and also introduces errno).

#include <sys/stat.h> 
#include <errno.h>
#include <stdio.h>

The global(-ish) variable errno tells us what went wrong in case a system call or a libc function fails. This is how perror knows what to say. I suggest you take a brief look at man 3 errno. Note that different systems put errno in different sections of the manual. You can also try

whatis
errno

. See also man whatis. Sorry for the digression.

int main() 
{
    struct stat st;

    stat( "/does/not/exist", &st );

Please note that errno can only be safely used immediately after the failing call; store it away because we will want to look at it later; errno is of type int.

    int error = errno; 
    perror( "stat" ); /* prints an error based on ‹errno› */

    if ( error == ENOENT )
        fputs( "the stat error was ENOENT\n", stderr );
}

Having sorted that out, let's continue with directories:

4.4 `directories.sh` (source code)

In this shell script, we will try creating and removing directories, and look at listing their content. We already know most of the commands involved, so this will be mainly review.

set -x # let's ask the shell to print commands for us

We first create and enter a scratch directory (it is an error if it already exists)

mkdir scratch ; cd scratch

The -p switch to mkdir tells it to create any missing parent directories. It is also not an error if the target directory already exists, in this case.

mkdir -p some/subdir
mkdir -p some

We also know how to list directory contents, using ls:

ls some

The rmdir command is the inverse of mkdir: it will remove an empty directory; it is an error if the directory is not empty, see:

rmdir some

However, those are okay:

rmdir some/subdir
rmdir some

cd ..
rmdir scratch

Let's move on to the C programs again: we'll now look at rmmkdir.c. We will skip ls for now because it's a little more complicated. But we will get to it near the end of the seminar today.

Let's start by ‘borrowing’ the #include directives (adding stdio.h):

#include <sys/stat.h>  /* mkdir */ 
#include <sys/types.h> /* also mkdir */
#include <errno.h>     /* errno */
#include <unistd.h>    /* rmdir, chdir */
#include <stdio.h>     /* perror */
#include <stdlib.h>    /* exit */

int main()
{

We quickly scan the manpage to learn that, once again, return value 0 (from both mkdir and rmdir) means success. Let's run with that. For mkdir, the prototype makes it clear that we need to pass in a path (of the directory to make). The second argument is a little more mysterious: you may remember a similar thing from second assignment, when we called open with O_CREAT. This is essentially the same thing. Let's pass 0777 this time, without explanation for now.

    if ( mkdir( "scratch", 0777 ) != 0 ) 
        perror( "mkdir scratch" ), exit( 1 ); /* let's give up */

The C version of cd is called chdir. Details, as usual, in man chdir. Like before, 0 means success.

    if ( chdir( "scratch" ) != 0 ) 
        perror( "chdir scratch" ), exit( 1 );

There is no equivalent of mkdir -p in the standard library. We have to do it by hand. Hold on to your hats, errno is coming. The value for ‘this file or directory already exists’ is EEXIST.

    if ( mkdir( "some", 0777 ) != 0 && errno != EEXIST ) 
        perror( "mkdir some" ), exit( 1 );

    if ( mkdir( "some/dir", 0777 ) != 0 && errno != EEXIST )
        perror( "mkdir some" ), exit( 1 );

Now try mkdir some again. It will fail but errno will be set to EEXIST.

    if ( mkdir( "some", 0777 ) != 0 && errno != EEXIST ) 
        perror( "mkdir some" ), exit( 1 );

    if ( rmdir( "some/dir" ) != 0 )
        perror( "rmdir some/dir" ), exit( 1 );

}

We keep some of the stuff around, so that you can check that it actually exists by hand. You can then remove the mess by typing:

$ cat 1.txt

4.6 Exercise 1

This exercise has 2 parts. First, write a script:
1. create directory scratch and change into it
2. create directories dir1/sub and dir2/sub/dir
3. compile the C program from ../remove.c (see point 2 below)
4. run ./remove dir1 and check that it fails
5. run ./remove dir1/sub and check that it succeeds
6. remove the remaining directories using remove in a for loop
The program remove.c should do the following:
1. remove the directory given by its first argument
2. if this fails, it should print an error and give non-0 exit
3. otherwise, it should print <directory> removed to stderr

That's it. Let's move to rm.sh.

4.7 `rm.sh` (source code)

This script demonstrates the rm command, which primarily removes (unlinks) files. It can, however, also remove directories and it can remove entire directory trees recursively. First some simple cases:

set -x

touch file # recall that this creates an empty file
ls file    # does it exist?
rm file    # remove it
ls file    # check again
rm file    # this prints an error, since ‹file› no longer exists

rm -f file # not an error -- this just ensures the file is not there
mkdir dir
rm dir     # this is an error
rm -d dir  # this works (but is non-standard, even if common)

touch file
rm -d file # this also works

Now let's try something more complicated (and dangerous!):

mkdir -p dir/subdir/stuff
touch dir/subdir/stuff/file
touch dir/file
rm -d dir  # fails
rmdir dir  # also fails
rm -df dir # still fails
rm -r dir  # finally! r is for recursive

For extra power (and danger), you could say rm -rf. Always remember that rm does not ask before deleting all your data. If you are unsure about the command, use rm -i, like this:

touch some_file
rm -i some_file

That's about as much as we need to know about removing files and directories using user-level commands. Let's have a look at unlink.c now.

$ micro links.sh

4.10 `links.sh` (source code)

Let's have some fun with hardlinks and symlinks.

set -x

touch file # create a test file
stat file  # let's check how it looks

To create a hard link, we can use the command ln (short for link). After a hard link is created, you can no longer tell which link is the original and which is ‘the link’: they are the same in all respects.

ln file hardlink # make ‹hardlink› refer to the same inode as ‹file›
stat file        # observe the link count and the inode number
stat hardlink    # same here

We can now unlink the original file and see what happens?

rm file
stat hardlink    # this should be unchanged, save for link count
mv hardlink file # get back to where we started (‹mv› renames files)

mkdir dir
ln dir dirlink # this fails, directory hardlinking is forbidden

Let's look at soft (symbolic) links now. They are created using ln -s (where s stands for soft/symbolic):

ln -s file softlink
ln -s dir dirlink

Both these worked, let's look at stat:

stat softlink
stat dirlink

There is one more command that is interesting with regards to symlinks:

readlink softlink
readlink dirlink
readlink dir # fails, ‹dir› is not a symlink

rm -df file dir dirlink softlink

Let's look at some weird things that can happen with symlinks too:

4.12 `links.c` (source code)

The C functions that correspond to ln are link and symlink. There is also readlink. They are all system calls and their manuals live in section 2. Let's #include things we'll need:

#include <unistd.h>   /* link, symlink, readlink */ 
#include <sys/stat.h> /* mkdir */
#include <fcntl.h>    /* open */
#include <stdio.h>    /* perror */
#include <stdlib.h>   /* exit */

int main()
{
    if ( open( "file", O_CREAT, 0666 ) < 0 ) /* make a test file */
        perror( "creating 'file'" ), exit( 1 );
    if ( mkdir( "dir", 0666 ) != 0 )         /* and a directory */
        perror( "creating 'dir'" ), exit( 1 );

As usual, we check the exit codes in man pages. For link and symlink, they are the usual 0 = success. But readlink behaves like read, returning the number of bytes, so we have to watch out there.

    if ( link( "file", "link" ) != 0 ) 
        perror( "creating link" ), exit( 1 );

    if ( link( "dir", "dirlink" ) != 0 )
        perror( "creating dirlink" ); /* we expect this to fail */

Now let's look at symlinks.

    if ( symlink( "file", "symlink" ) != 0 ) 
        perror( "creating symlink" ), exit( 1 );

    if ( symlink( "dir", "dirlink" ) != 0 )
        perror( "creating (symbolic) dirlink" ), exit( 1 );

    char buffer[ 60 ] = { 0 };
    if ( readlink( "dirlink", buffer, 59 ) < 0 )
        perror( "readlink dirlink" ), exit( 1 );

    printf( "readlink on dirlink gave %s\n", buffer );
    if ( readlink( "dir", buffer, 59 ) < 0 )
        perror( "readlink dir" );
}

We do not clean up so that you can check the resulting files and links with stat or ls -l. However, you will have to clean up by hand. I am sure you know how (you can use

rm
-d

). That was links in C. Please go on to 3.txt.

4.13 Exercise 3

Write a C program called destroy, that:

takes a single filename as an argument,
unlinks the file; if it is a symlink, also unlinks the target,
prints whether the file is gone, or some hardlinks remain.

There are a few improvements that could be done. See 4.txt.

4.14 Exercise 4

Improve the solution of the previous exercise in following ways:

4.15 `readdir.c` (source code)

To do what ls does, we need a set of somewhat complicated C functions. They are like open and read but for directories, and are called opendir and readdir accordingly. Please check the man pages: they are in section 3. The readdir in section 2 is an obsolete system call on Linux, please ignore it!

#include <dirent.h> /* opendir, readdir */ 
#include <stdio.h>  /* puts */
#include <stdlib.h> /* exit */

int main()
{

Let's open the current working directory (cwd for short). The manual suggests that on failure, cwd will be a NULL pointer.

    DIR *cwd = opendir( "." ); 

    if ( !cwd )
        perror( "opendir ." ), exit( 1 );

We have cwd open and we can read the individual directory entries. The dirp below is a pointer to a single directory entry.

    struct dirent *dirp; 

    while ( 1 ) /* forever */
    {
        dirp = readdir( cwd ); /* obtain the next directory entry */

If readdir returns NULL, it means that there are no more directory entries (i.e. we read all of them). In that case, let's stop the loop using break here.

        if ( !dirp ) 
            break;

There are a few items in the directory entry (like the inode number), but we only want to print the name for now. We can use puts for that.

        puts( dirp->d_name ); 
    }

Free up the resources related to the directory handle (analogous to close).

    closedir( cwd );

That's it. This program, when compiled and executed, will print the names of all files and directories in its working directory (in no particular order).

Please note that in modern programs, we would most likely use readdir_r instead of readdir, because there might be threads in such programs, and readdir doesn't work very well with those. But we won't worry about that yet. This was the last bit I wanted to show you. Only the homework remains, in hw.txt.

4.16 Homework

Implement list.c, an extended version of readdir.c, which prints the contents of the working directory.

Each entry goes on a single line. The format is as follows (each column is separated from the next by a single space):
1. one character, f for files, d for directories, l for symlinks
2. the inode number
3. the number of links
4. the size of the file (0 if not applicable)
5. the file name
6. if the file is a symlink, ... target (otherwise nothing)

That's it for today. Have a nice day!

$ micro proc.sh

5.1 `proc.sh` (source code)

We will first learn how to run a process in the background and how to get its process ID (pid). The first step is running a command without waiting for the result, which is achieved by ending the command with a single ampersand, like this:

while true; do sleep 1; echo .; done &

Now the entire compound command is running in a separate process (the shell has forked): the parent processes continues to execute this script (i.e. the commands below) while the child runs the command above (it runs in the background, but it can still print to the terminal). To do anything useful with the running process, we need to learn its process identifier, which is stored in the shell variable $!.

child=$!
echo $child

Then we wait a bit, so that the loop above can print a few lines. The sleep command, as you probably figured out by now, does nothing for a given number of seconds (5 in this case).

sleep 5

Now we request that the child process terminates. This is a ‘nice’ request, in the sense that the operating system notifies the process that the user asked for its termination. The process may or may not honour the request (it usually will, but it may perhaps try to clean things up before terminating).

kill $child

As usual, you can look up kill in the manual. That's it, we will look at a few more process management commands now:

5.2 `ps.sh` (source code)

Now that we know how to run a process from shell, we will look at a few more commands that work with processes. So far, we have only seen kill, which, given a process ID, sends a signal to the process. You can learn more about signals in man 7 signal -- unfortunately, we won't have time to get into details in the seminar. First we will create a process and get its pid, like we did before:

chmod +x spin.sh
./spin.sh &

set -x

pid=$!

The first command that I want to show you is pgrep, which allows you to find processes by looking at the commands those processes are executing. Of course, you can look at man pgrep. Without -f, it checks only the command itself, with -f it will also match on the command line arguments. To make things a little more complicated, executable scripts sometimes have their filename as the command name, and sometimes the interpreter name, depending on the operating system. On aisa, it is the former, so the first command will work as we expect.

pgrep spin.sh
pgrep -f spin.sh

The pgrep command has a companion called pkill, which does the same thing but instead of printing the process id's of the matching processes, it kills them (sends them a ‘please terminate’ signal, also known as TERM). Killing processes forcibly can be done with the -KILL switch to either kill or pkill. It is usually better to not use -KILL. The last user command for process management that you need to know is ps, which is short for ‘processes’ and simply prints a list of all processes. By default, only interactive processes owned by you are shown:

ps

If we want to see more details, we can add u to the options. Notice that unlike most other unix programs, the switches to ps can be specified without the dash.

ps u

But if you are more comfortable with the dash, you can have it, too:

ps -u # the same as the previous command

We can request to also see processes that do not have a controlling terminal.

ps -x

And finally, we can also request to see all the processes in the system with ps -a, but we won't run that in the script because there are many. You can try running it by hand if you like. Another command to try yourself would be top, which interactively shows current processes in the system and tells you how much CPU and memory they use. Okay, let's wait for a bit to check our process is running. Be sure to notice our process in the process listings from ps.

sleep 3
kill $pid

Let's do an exercise, 1.txt.

5.3 Exercise 1

Write a shell script that:
1. creates a background process that writes a line with a dot into file.txt every second
2. create another process that counts the lines in file.txt and prints the count to the terminal every second
3. use ps -u and grep to print information about both processes every second for 10 seconds
4. kill both background tasks

That is all for the user-level part. Let's move on to C:

int main() 
{
    printf( "before fork()\n" );

fork takes no arguments and returns a pid_t, which is an integer with platform-defined size (big enough to hold any valid process id)

    pid_t pid = fork();

The following message will appear twice, once from the parent process and once from the child process. In the parent process, fork() returns the pid of the newly created child. In the child process, it returns 0. If an error happens, it returns -1 and no process is created: hence, in this case, fork() only returns once.

    printf( "after fork(), result = %d\n", pid );

Be sure to understand the output of this program. We will use fork() in a more complicated scenario shortly.

Let's see one more example before moving on to exercises:

5.5 `wait.c` (source code)

In this program, we will learn how to check the result of the child process. Then we will move on to an exercise that will use fork().

#include <unistd.h>   /* fork */ 
#include <sys/wait.h> /* waitpid */
#include <stdio.h>    /* printf */

int main()
{
    puts( "entered main()" );
    pid_t pid = fork();

    if ( pid < 0 )
    {

Things went south; you can look at man fork to check what could cause fork to fail (specifically section ERRORS near the end of the manpage).

        perror( "fork" ); 
        return 1;
    }

    if ( pid == 0 )
    {
        puts( "in the child process" );
        return 13; /* terminate with exit code 13 */
    }

    if ( pid > 0 )
    {
        printf( "in the parent process, child pid = %d\n", pid );

The child process is running, but we don't know when it finishes or how. To find out, we can use the waitpid system call. It is only allowed to call waitpid on processes that are direct descendants (via fork) of the current one. As usual, see man waitpid. Besides waiting for the child process to finish, we can learn the exit status, or whether the process exited normally or was killed.

        int status; 
        if ( waitpid( pid, &status, 0 ) == -1 )
        {
            perror( "waitpid" );
            return 1;
        }

The integer status passed to waitpid is filled in with information about how the process terminated. Unfortunately, it is a single integer, not a structure like struct stat, which makes things a little odd and complicated. We have to use macros called WIFEXITED and WEXITSTATUS to learn whether the program terminated normally and what was the exit code. The remaining macros (described in man waitpid) deal with signals -- you can study those on your own, or perhaps in a more advanced course on POSIX programming.

        if ( WIFEXITED( status ) ) 
            printf( "child terminated with return code %d\n",
                    WEXITSTATUS( status ) );

        return 0; /* terminate with exit code 0 */
    }
}

Now we are ready for an exercise: check 2.txt.

5.6 Exercise 2

Write a C program that uses fork() to create two processes. The parent:
1. creates a file called file1.txt
2. loops until file2.txt appears -- you can use stat and sleep( 1 )
3. unlinks both file1.txt and file2.txt
4. waits for the child process and prints its exit code
The child:
1. waits until a file called file1.txt appears
2. creates a file file2.txt
3. exits with a code of your choosing
Print a message in both the parent and the child after each step (identifying which process is talking).

That's that. We are ready for the final chapter for today, and that is exec:

l is for list (of arguments)
whereas v is for varargs (we won't need to deal with those yet)
the suffix p means ‘use PATH to find the command’ and
the suffix e means ‘let me pass a new environment to the program’.

For now, we will just use execl. As is easily checked in man execl, we need to include unistd.h:

#include <unistd.h> /* execl */ 
#include <stdio.h>  /* puts */

int main()
{

We just print a ‘hello’ to make it clear we are running.

    puts( "hello from main" );

Now we use execl to replace the current process with another program, and pass a few arguments to that program. The first argument is the path to the program to execute, the rest of the arguments is what becomes argv of the new main() in the new program. Remember that argv[0] should be the name of the program itself: we need to pass that explicitly! The last argument must be NULL, to let the function know when to stop looking for more arguments. Notice that we do not pass argc -- the arguments are counted by the system.

    if ( execl( "/bin/echo", "echo", "hi from echo", NULL ) == -1 ) 
    {
        perror( "exec" );
        return 1;
    }

You may have read in man execl that ‘The exec() functions return only if an error has occurred.’ This is because if they work as intended, the program executing in this process has been replaced, and is no longer this program. The following line will therefore never execute -- when the new program ends, the process terminates as usual.

    puts( "back to main" ); 
}

Let's now look at the usual combination of fork + exec, which lets us execute a program and wait for it to finish. The code is in exec2.c.

5.8 `exec2.c` (source code)

This program demonstrates the code for ‘running’ a program in the usual sense: create a new process and then use exec to replace the program in the process with a new one. As usual, includes first:

#include <unistd.h>   /* fork, execl */ 
#include <stdio.h>    /* printf */
#include <sys/wait.h> /* waitpid */
#include <stdlib.h>   /* exit */

int main( int argc, const char **argv )
{
    puts( "hello from main" );
    if ( argc != 2 )
        puts( "please pass in a path to the program to execute" ), exit( 1 );

Let's start by creating a new process using fork().

    pid_t pid = fork(); 

    if ( pid < 0 ) /* don't forget to check for errors */
        perror( "fork" ), exit( 2 );

    if ( pid > 0 ) /* parent */
    {
        puts( "hello from parent" );
        int status; /* to hold the exit status of the child */

We wait for the child process to terminate and get the status.

        if ( waitpid( pid, &status, 0 ) == -1 ) 
            perror( "waitpid" ), exit( 2 );

And print the result to the user.

        if ( WIFEXITED( status ) ) 
            printf( "child terminated with exit code %d\n",
                    WEXITSTATUS( status ) );
    }

    if ( pid == 0 ) /* child */
    {

As you have seen in exec1.c, we need to pass the program name twice to execl. We don't want any further arguments, so we just pass the terminating NULL.

        if ( execl( argv[1], argv[1], NULL ) == -1 ) 
            perror( "exec" ), exit( 2 );

We never get here.

    } 
}

There are a few things we can try with this program:

$ ./a.out /bin/echo 
$ ./a.out /bin/false
$ ./a.out /bin/true
$ ./a.out /does/not/exist
$ ./a.out true

Let's do an exercise with fork() and exec(): 3.txt.

5.9 Exercise 3

Write a C program that forks into two processes.
The parent waits for 3 seconds and then kills the child.
The child executes ./spin.sh.

That wasn't too hard, was it? The homework is described, as usual, in hw.txt.

5.10 Homework

Write a C program sync.c and a shell script sync.sh.
The C program should fork() & the child should then exec() the script.
The parent process:
1. waits until file1.txt appears
2. when it does, it copies its content into file2.txt
3. waits for the script to finish and prints its exit code
4. use POSIX read() and write() above
The shell script:
1. prints its own pid (found in $$) into file1.txt
2. waits until file2.txt appears
3. waits a second to make sure the content is written
4. reads file2.txt and checks that its content matches
5. if it matches, ends with exit code 0, otherwise 1
6. the shell built-in exit 0 and exit 1 can be used for (e)

That is all! Or is it.

$ micro forkfd.c

6.1 `pipes.sh` (source code)

This script simply serves to remind you how anonymous pipes work in shell.

TBD

6.2 `forkfd.c` (source code)

Since we want to talk about IPC, we will need multiple processes. Hence, we will need fork() to create those processes. If you don't remember how fork() works, plase check out ../05/fork.c (and the following units) again.

#include <unistd.h> /* fork, read */ 
#include <stdio.h>  /* printf */
#include <fcntl.h>  /* open */

int main()
{

The purpose of this C program is to demonstrate how file descriptors are inherited across a call to fork. Therefore, we will first open a file for reading.

    int fd = open( "intro.txt", O_RDONLY ); 

    if ( fd < 0 )
        return perror( "open intro.txt" ), 1;

And call fork, to create a second process.

    pid_t pid = fork(); 

    if ( pid < 0 )
        return perror( "fork" ), 1;

You might remember from the lecture that file descriptors are per-process (and this is indeed true). However, fork creates an identical copy of a process, including all its open file descriptors. Two processes are running from this point on. The read call is performed twice, each in one of the processes, but the fd is still shared: we will read different parts of the file.

    char buf[ 36 ] = { 0 }; 
    if ( read( fd, buf, 35 ) != 35 )
        return perror( "read" ), 1;

We will print the process id of each process that made it this far, by using the getpid function, which just returns the pid of the calling process. As usual, see man getpid.

6.3 `pipe.c` (source code)

We reminded ourselves about fork and explained what it means for file descriptors. We will set that aside for a while, because we only need 1 process in this unit, where we explain the pipe system call. In the next part, we will combine what we know about both to build a simple IPC mechanism.

#include <unistd.h>  /* pipe, read, write */ 
#include <stdio.h>   /* printf */

int main()
{

To create an unnamed pipe, we will need 2 file descriptors: one for the reading end and one for the writing end. Let's prepare memory for that: what follows is an array of 2 integers.

    int fds[ 2 ];

Now the creation of the pipe: unlike open, the pipe syscall does not return a file descriptor, because it actually needs to return 2 of those. The return value only indicates success (0) or error (-1). For the file descriptors, it uses the same approach as stat: we pass a pointer to memory we prepared above.

    if ( pipe( fds ) == -1 ) 
        return perror( "pipe" ), 1;

The file descriptors are not interchangeable. One is the reading end (from which you can only read) and the other is the writing end (to which you can only write). A pipe is a unidirectional device -- data only flows in one direction. To make the remaining code more readable, we will give names to the ends.

    int read_fd = fds[ 0 ], write_fd = fds[ 1 ];

Let's write something (the string hello) into the write end now.

    if ( write( write_fd, "hello", 5 ) != 5 ) 
        return perror( "write" ), 1;

We try reading from the read end to see what comes out. Knowing how pipes work, we expect hello to be the result.

6.4 `pipe-ipc.c` (source code)

#include <unistd.h>  /* fork, pipe, read, write */
#include <stdio.h>   /* printf */

int main()
{

We already know how to create a pipe. Let's just repeat the process.

    int fds[ 2 ]; 

    if ( pipe( fds ) == -1 )
        return perror( "pipe" ), 1;

    int read_fd = fds[ 0 ], write_fd = fds[ 1 ];

Like before, we keep the pipe ends as read_fd and write_fd.

    pid_t pid = fork(); /* fork & check for errors */ 
    if ( pid == -1 )
        return perror( "fork" ), 1;

    if ( pid > 0 ) /* parent */
    {

The first thing we should do is close the end of the pipe that we are not going to use. This is important, because closing the write end of the pipe is the only way to indicate the end of file to the read end. And overall, it is just bad form to keep unused fd's lying around.

        close( read_fd );

Having that sorted, we just write a message for the child to read.

        if ( write( write_fd, "hello", 5 ) != 5 ) 
            return perror( "write" ), 1;

        close( write_fd ); /* indicate that we are done */
    }

    if ( pid == 0 ) /* child */
    {
        close( write_fd ); /* see parent */

there should be a message by the parent, let's try to read it

        char buf[ 6 ] = { 0 }; 
        if ( read( read_fd, buf, 5 ) != 5 )
            return perror( "read" ), 1;
        printf( "child obtained message: %s\n", buf );

That should have worked okay. Now let's check that there is no more data in the pipe (because the parent closed the writing end).

        if ( read( read_fd, buf, 1 ) == 0 ) 
            printf( "child got eof on the pipe, as expected\n" );
        else
            printf( "oops, unexpected data in the pipe!\n" );
    }

    return 0;
}

It is time to work out an exercise, see 1.txt.

6.5 Exercise 1

Write a C program that creates 2 pipes.
Then it forks off 2 child processes.
Sends each of them a hello through the pipe.
The children both print their pid and the message.

Now let's learn about dup and dup2, so that we can replicate what shell does when we use |, but in C. That is, we will use pipe, fork and exec together. See dup.c.

6.6 `dup.c` (source code)

The familiar preamble:

#include <unistd.h> /* dup, read */ 
#include <fcntl.h>  /* open */
#include <stdio.h>  /* printf */

int main()
{

The dup family of functions manipulates (duplicates) file descriptors. We will first try dup on a file and then on stdout.

    int fd1 = open( "intro.txt", O_RDONLY );

The dup call makes a copy of fd1, with a new number.

    int fd2 = dup( fd1 );

We will need two buffers for reading data.

    char buf1[ 22 ] = { 0 }, buf2[ 15 ] = { 0 };

Now we will read from the file descriptors in turn. The man page for dup says that both descriptors share the same ‘open file’ object (the same entry in the table of open files). This means, among other things, that the read position is shared between them.

    if ( read( fd1, buf1, 21 ) != 21 ) 
        return perror( "read fd1" ), 1;

    printf( "reading from fd1: %s\n", buf1 );

    if ( read( fd2, buf2, 14 ) != 14 )
        return perror( "read fd2" ), 1;

    printf( "reading from fd2: %s\n", buf2 );

However, closing one of the descriptors does not affect the other. You could read from fd2 after closing fd1, for instance. We will see that later.

    if ( close( fd1 ) ) 
        return perror( "close fd1" ), 1;
    if ( close( fd2 ) )
        return perror( "close fd2" ), 1;

    return 0;
}

Let's check out dup2 now, in dup2.c.

6.7 `dup2.c` (source code)

First some includes:

#include <unistd.h> /* dup2 */ 
#include <fcntl.h>  /* open */
#include <stdio.h>  /* puts */

int main()
{

We start by opening an output file.

    int fd = open( "dup2_out.txt", O_WRONLY | O_CREAT, 0666 ); 

    if ( fd == -1 )
        return perror( "open" ), 1;

The only difference between dup and dup2 is that in the latter, you get to choose the number that will be used for the copy of the file descriptor. If that number is an existing fd, the old one will be closed and replaced by the new one. Try to think about what you would expect the following dup2 call to do.

    if ( dup2( fd, 1 ) < 0 ) 
        return perror( "dup2" ), 1;

At this point, stdout (fd number 1) was replaced by a copy of fd, which points to a file open for writing. Let's use familiar puts to print something to stdout. Normally, that would go to screen, but now it should probably end up in the file dup2_out.txt?

    puts( "hello world" ); 
}

Now we are ready to move on to another exercise, 2.txt.

6.8 Exercise 2

First, a hint: you will want to use execvp( argv[1], argv + 1 ) to execute a program along with arguments given to you on the command line.
If you prefer a complete example, check out execvp.c.
Write a C program that uses pipe and dup2 to send the string "hello\nworld\n" as the input to a command given to it as described in point 1 above.
Check that the program works as expected by running ./a.out cat and ./a.out wc -l.

We can now move on to sockets, see

$ micro socketpair.c

6.9 `socketpair.c` (source code)

This program demonstrates the use of the socketpair function. The prototype is in sys/socket.h:

#include <unistd.h>     /* fork, read, write */ 
#include <sys/socket.h> /* socketpair */
#include <stdio.h>      /* printf */

int main()
{

Socket pairs are basically just like pipes, but bidirectional. The interface to create them is similar, but there are some new elements. However, the idea with two file descriptors is unchanged.

    int fds[ 2 ]; 
    char buf[ 6 ] = { 0 };

There are some extra arguments when we compare socketpair to pipe. The first is so-called address family. There are basically 3 that we could consider, those are AF_UNIX (unix domain sockets), AF_INET (internet protocol, IP) and AF_INET6 (IPv6). The first type, AF_UNIX, is a local socket, that connects programs that run on the same computer. It is the only valid option for socketpair. The other two types connect programs across a network, and we will deal with them in a later lecture. The second new bit is socket type, of which there are two: stream sockets (SOCK_STREAM, which behave like pipes, i.e. as byte streams) and datagram sockets -- SOCK_DGRAM, which are essentially a message-passing device (datagram sockets send entire packets, not individual bytes). The third option to socketpair is protocol -- the only reasonable value here is 0, which lets the system choose something appropriate.

    if ( socketpair( AF_UNIX, SOCK_STREAM, 0, fds ) == -1 ) 
        return perror( "socketpair" ), 1;

For socketpair, the file descriptors are equivalent -- there is no distinction between a writer and a reader.

    int parent_fd = fds[ 0 ], child_fd = fds[ 1 ]; 

    pid_t pid = fork(); /* fork & check for errors */
    if ( pid == -1 )
        return perror( "fork" ), 1;

    if ( pid > 0 ) /* parent */
    {

Like with a pipe, we will close the end of the socket pair that is not used by this process.

        close( child_fd );

Having that sorted, we just write a message for the child to read.

        if ( write( parent_fd, "hello", 5 ) != 5 ) 
            return perror( "write" ), 1;

        if ( read( parent_fd, buf, 5 ) != 5 )
            return perror( "read" ), 1;

        printf( "parent obtained message: %s\n", buf );

        close( parent_fd ); /* clean up */
    }

    if ( pid == 0 ) /* child */
    {
        close( parent_fd ); /* see parent */

There should be a message by the parent, let's try to read it:

        if ( read( child_fd, buf, 5 ) != 5 ) 
            return perror( "read" ), 1;

        printf( "child obtained message: %s\n", buf );

        if ( write( child_fd, "world", 5 ) != 5 )
            return perror( "write" ), 1;

        close( child_fd );
    }

    return 0;
}

Let's look at a ‘real’ socket now. See client.c.

6.10 `client.c` (source code)

This program will serve as the client side to an AF_UNIX socket. All it does is connect to the socket and send a simple message. The content of the message is given by the command line.

#include <unistd.h>     /* write */ 
#include <stdio.h>      /* perror */
#include <string.h>     /* strlen */
#include <sys/socket.h> /* socket, connect */
#include <sys/un.h>     /* sockaddr_un */

int main( int argc, const char **argv )
{
    if ( argc != 2 )
        return fputs( "need exactly 1 argument\n", stderr ), 1;

Sockets are symmetric, in the sense that the first thing we always do is create a socket. That socket can then become a server or a client, depending on what happens next. So let's create a socket: the arguments should now be familiar, because they are exactly the things that we passed to socketpair just a short while ago.

    int sock_fd = socket( AF_UNIX, SOCK_STREAM, 0 ); 

    if ( sock_fd == -1 )
        return perror( "socket" ), 2;

The socket itself is not very useful. To talk to some other program, we need to connect to its listening socket, and for that, we need to tell the system the address on which the server is listening. Addresses for AF_UNIX sockets are described by structures of the type sockaddr_un.

    struct sockaddr_un sa = { .sun_family = AF_UNIX, 
                              .sun_path = "./socket" };

The above describes the address, now let's ask the socket to connect to it. The function to do that is called connect, and takes the socket file descriptor and the address, like this:

    if ( connect( sock_fd, ( struct sockaddr * ) &sa, 
                  sizeof( sa ) ) == -1 )
        return perror( "connect to ./socket" ), 2;

    size_t bytes = strlen( argv[1] ) + 1;
    if ( write( sock_fd, argv[1], bytes ) != bytes )
        return perror( "write" ), 2;

    close( sock_fd );
    return 0;
}

Before compiling and executing this program, we will need also the server part. So let's read that first and then we can try both pieces:

6.11 `server.c` (source code)

This is the server part of an example client-server system. It creates a socket, listens on it and prints any messages it receives from clients.

#include <unistd.h>     /* read */ 
#include <stdio.h>      /* perror, printf */
#include <string.h>     /* strcpy */
#include <sys/socket.h> /* socket, connect */
#include <sys/un.h>     /* sockaddr_un */

int main( int argc, const char **argv )
{

Like before, we start by creating a socket.

    int sock_fd = socket( AF_UNIX, SOCK_STREAM, 0 ); 
    if ( sock_fd == -1 )
        return perror( "socket" ), 1;

To become a server, we also need an address, but this time it will be the address to listen on. The code to prepare the address is, however, the same as before.

    struct sockaddr_un sa = { .sun_family = AF_UNIX, 
                              .sun_path = "./socket" };

Now the differences start. Instead of calling connect, we will call bind which tells the system to turn this socket into a server socket with the given address. First, however, let's unlink any previous sockets that may have been left behind.

    unlink( "./socket" );

Now for the bind itself.

    if ( bind( sock_fd, ( struct sockaddr * ) &sa, sizeof( sa ) ) ) 
        return perror( "bind ./socket" ), 1;

The socket is bound and we can listen for connections on it now. The function to do that is called, unsurprisingly, listen. This is a simple function that allows clients to start connecting to the above address. The second parameter says how many clients can ‘queue up’ to talk to us. We'll understand why in a short while.

    if ( listen( sock_fd, 5 ) ) 
        return perror( "listen" ), 1;

The final ingredient is called accept, and is a little tricky, because there might be multiple programs trying to talk to us at once. We may call accept multiple times, and every time we do that, we get a new file descriptor, which behaves like one end of a socketpair (the other end being the socket that the other program called connect on). The accept call will wait for a new connection (i.e. it will not return until there is a client that wants to talk to us).

    int client_fd; 
    char buffer[ 64 ];
    while ( ( client_fd = accept( sock_fd, NULL, NULL ) ) >= 0 )
    {

The return value of accept is a file descriptor, so let's try reading some data (the example client just sends us a message).

        if ( read( client_fd, buffer, 64 ) >= 0 ) 
            printf( "received: %s\n", buffer );
        else
            perror( "read" );

We clean up the connection and accept another client.

        close( client_fd ); 
    }
}

That is all for theory today, let's work out another exercise: 3.txt.

6.12 Exercise 3

You will need the execvp trick that we used in exercise 2.
Connect to the server that we already have (in server.c).
Send the output of the program given in arguments to the server.
Check that things work as expected, e.g.: ./a.out ls

The homework is specified in hw.txt.

6.13 Homework

Write a C program, listen.c, which listens on a Unix socket:
1. the name of the socket should be ./socket
2. repeatedly accept connections from clients
3. each client will send a single null-terminated path
4. the server will execute this program in a new process
5. the path buffer should be at least 200 bytes

As a test client, you can use client.c from the exercise.

7 Threads, Synchronisation

We will look at threads that run within a single process today.
To create and work with threads, we will use the pthread API.
In addition to thread creation and management, there is a number of pthread functions that implement synchronisation primitives.
We will try out communication using shared memory (global variables).
In the second half, we will try mutexes, semaphores and barriers.

Let's start by creating a thread:

$ micro thread.c

7.1 `thread.c` (source code)

We will need a new (unfamiliar) header file: pthread.h.

#include <pthread.h> /* pthread_create, pthread_join */ 
#include <stdio.h>   /* puts, perror */
#include <unistd.h>  /* sleep */

Let us first declare a function which will become the 'main' function of a thread which we will create in main(). However, we want to first look at how main works, so we will define the function thread later.

void *thread( void * );

The main function is the same as always -- using threads does not affect the prototype.

int main() 
{

Threads have numeric identifiers, just like processes do. For a process, the numeric identifier is called a pid (as you may recall from last two weeks), and is of the type pid_t. For threads, the identifier is usually called tid (for thread id) and is of the type pthread_t.

    pthread_t tid;

We'll print out some messages to see what is going on in the program.

    puts( "main, before pthread_create" );

To create a new thread, we will use the function pthread_create (you can, as always, look into

man
pthread_create

for details). This function is quite different from fork: it returns only once, in the main thread (i.e. the parent). The child will start executing the function passed to pthread_create as the third argument (in this case, the function thread that we declared above). The remaining arguments to pthread_create are: an address of a variable that will hold the thread identifier of the newly created thread (this is the first argument), then come optional attributes of the thread (we will not need to set any of those) and finally a void * value (a pointer of an unspecified type), which is passed as the argument to the function given by the third argument. Since we do not need to pass anything to the thread, we will just use NULL for now.

    if ( pthread_create( &tid, NULL, thread, NULL ) ) 
        return perror( "pthread_create" ), 1;

Let's print some messages and sleep for a bit in between, so that we can clearly see that the newly created thread and the main thread execute concurrently.

    puts( "main, after pthread_create" ); 
    sleep( 2 );
    puts( "main also still here" );
    sleep( 1 );
    puts( "about to pthread_join" );

Now that we have done all that we wanted, we should synchronize with the thread, i.e. wait until it finishes execution. The pthread_join call is the thread equivalent of the waitpid syscall, which waits for a child process to finish. That is, pthread_join will wait until the thread specified by the first argument stops executing. An optional return value from the thread's main function can be obtained via the second argument. We will use this later.

    if ( pthread_join( tid, NULL ) ) 
        return perror( "pthread_join" ), 1;

    return 0;
}

That is all for the main thread, now let's look at the child. Its execution starts with the function thread below the moment the main thread calls pthread_create above. The thread has its own stack, but otherwise, it has access to the same memory as the main thread. We will also look at this later in more detail.

void *thread( void *arg ) 
{

The code in the child thread is quite boring: all it does is print a few messages and sleep some in between. Then it returns NULL, which the main thread discards anyway.

    puts( "thread" ); 
    sleep( 1 );
    puts( "thread still here" );
    sleep( 1 );
    puts( "about to end" );
    return NULL;
}

Unlike all the C programs until this point, this one needs a little more than just the standard C library (libc). For this reason, we need to compile it with an additional switch (this will hold for all programs that we encounter today):

volatile int x = 0;

Our thread will be very simple: it will print the value of x, sleep for a bit, then print the value of x again.

void *thread( void *arg ) 
{
    printf( "x = %d\n", x );
    sleep( 1 );
    printf( "x = %d\n", x );
    return NULL;
}

int main()
{
    pthread_t tid;
    x = 17; /* start by setting the value of x */

We first start the thread, the same way we did before.

    if ( pthread_create( &tid, NULL, thread, NULL ) ) 
        return perror( "pthread_create" ), 1;

Now let's wait for a bit, so the thread has a chance to run. This is definitely not something you would do in a normal program. The sleep here does not guarantee that the other thread will run, or that it will get past the first printf even if it does. To demonstrate how things work, however, it's an okay thing to do.

    usleep( 200000 ); /* 200ms */

Change the value of x.

    x = 33; 
    printf( "main changed x to 33\n" );

And wait for the thread to finish.

    if ( pthread_join( tid, NULL ) ) 
        return perror( "pthread_join" ), 1;

    return 0;
}

This program has demonstrated that global variables are shared between different threads. For fun and profit, you should try to do the same thing with processes, to convince yourself how threads are different from processes (and how shared memory is different from isolated, per-process memory). In fact, this is the first exercise, in 1.txt.

7.3 Exercise 1

Take the program in shvar.c and rewrite it to use fork and processes instead of threads.
Run the result to demonstrate that the variable x is not shared -- the parent changing the variable will not affect its value in the child process.

We can move on to data races now, see race.c.

volatile int x = 0;

The thread simply tries to increment x a million times.

void *thread( void *arg ) 
{
    for ( int i = 0; i < 1000000; ++i )
        ++ x;
    return NULL;
}

int main()
{
    pthread_t tid;

Let's start the thread.

    if ( pthread_create( &tid, NULL, thread, NULL ) ) 
        return perror( "pthread_create" ), 1;

Now let's decrement the variable x a million times. Since we increment and decrement the same number of times, we would expect the result to be 0, regardless of the order in which the increments and decrements get executed in the two threads.

    for ( int i = 0; i < 1000000; ++i ) 
        -- x;

We wait for the child thread to finish, so we know that both loops have executed the same number of times.

    if ( pthread_join( tid, NULL ) ) 
        return perror( "pthread_join" ), 1;

Here, the value of x should be zero. Let's check.

    printf( "x = %d\n", x );

Oops. Notice that the actual value is also different every time you run the program. That is quite unfortunate! Let's try to fix that.

We will look at mutexes, which solve this (and related) problems. See mutex.c.

7.5 `mutex.c` (source code)

In this program, we will demonstrate how a mutual exclusion device works. Afterwards, we will combine global shared variables with a mutex, which is one of the correct approaches for communicating in a threaded program.

#include <pthread.h> /* pthread_create, pthread_join */ 
#include <stdio.h>   /* puts, perror */
#include <unistd.h>  /* sleep */
#include <stdlib.h>  /* exit */

First of all, we need a variable to represent the mutual exclusion device itself. The variable should be of type pthread_mutex_t and it can be either initialized by a macro like we do here, or by calling the function pthread_mutex_init. The lock and unlock methods of the mutex are implemented by the functions pthread_mutex_lock and pthread_mutex_unlock respectively. We will see an example below.

pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;

The thread will just print messages and sleep in-between to demonstrate how mutual exclusion operates.

void *thread( void *arg ) 
{
    puts( "thread started" );
    sleep( 3 );

Now we lock the mutex, entering a critical section. All mutex (and semaphore, barrier and similar) functions take the address of the mutual exclusion device as their parameter. This is because the device (the mutex) must be the same mutex in all threads -- passing the mutex by value would make a copy, and could not do what we want (that is, lock and unlock the shared mutex stored in the variable above).

    if ( pthread_mutex_lock( &mutex ) ) 
        perror( "pthread_mutex_lock" ), exit( 1 );

Now that we are in the critical section, print a message and sleep for a few seconds, so it is clear that the main thread cannot enter the critical section while we are in it.

    puts( "thread in the critical section" ); 
    sleep( 3 );
    puts( "thread spent 3 seconds in the critical section" );

We are satisfied with our stint in the critical section, let's unlock the mutex...

    if ( pthread_mutex_unlock( &mutex ) ) 
        perror( "pthread_mutex_unlock" ), exit( 1 );

... print a few more messages ...

    puts( "thread is out of the critical section" ); 
    sleep( 3 );
    puts( "thread finished" );

... and stop

    return NULL; 
}

int main()
{
    pthread_t tid;

We start the other thread first:

    if ( pthread_create( &tid, NULL, thread, NULL ) ) 
        return perror( "pthread_create" ), 1;

Let's try to enter the critical section repeatedly (every second) and print a message whenever we get in.

    for ( int i = 0; i < 10; ++i ) 
    {

The call to lock a mutex is the same in all threads.

        if ( pthread_mutex_lock( &mutex ) ) 
            return perror( "pthread_mutex_lock" ), 1;

        puts( "main entered critical section" );

Likewise for unlocking it.

        if ( pthread_mutex_unlock( &mutex ) ) 
            return perror( "pthread_mutex_unlock" ), 1;

        sleep( 1 );
    }

And wait for the thread to finish.

    if ( pthread_join( tid, NULL ) ) 
        return perror( "pthread_join" ), 1;

    return 0;
}

As promised, the next program will show how to use mutexes together with global variables to communicate properly. See mtxvar.c.

7.6 `mtxvar.c` (source code)

A couple #include statements:

#include <pthread.h> /* pthread_{create,join} */ 
                     /* pthread_mutex_{lock,unlock} */
#include <stdio.h>   /* printf, perror */

We no longer need the volatile keyword, because we will use a mutex to guard access to the variable x, which among other things tells the compiler to generate correct code for multiple threads accessing it.

int x = 0;

Like before, we set up a global, shared mutex.

pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;

The child thread tries to increment x a million times, but this time it protects the increment with the mutex -- that is, it turns the increment into a critical section.

void *thread( void *arg ) 
{
    for ( int i = 0; i < 1000000; ++i )
    {
        if ( pthread_mutex_lock( &mutex ) )
            return perror( "pthread_mutex_lock" ), NULL;

        ++ x;

        if ( pthread_mutex_unlock( &mutex ) )
            return perror( "pthread_mutex_unlock" ), NULL;
    }
    return NULL;
}

int main()
{
    pthread_t tid;

First of all, let's start the thread.

    if ( pthread_create( &tid, NULL, thread, NULL ) ) 
        return perror( "pthread_create" ), 1;

Like in the child thread, the parent should guard the decrement on x with the same mutex (and hence the same critical section).

    for ( int i = 0; i < 1000000; ++i ) 
    {
        if ( pthread_mutex_lock( &mutex ) )
            return perror( "pthread_mutex_lock" ), 1;

        -- x;

        if ( pthread_mutex_unlock( &mutex ) )
            return perror( "pthread_mutex_unlock" ), 1;
    }

We wait for the child thread to finish, so we know that both loops have executed the same number of times.

    if ( pthread_join( tid, NULL ) ) 
        return perror( "pthread_join" ), 1;

At this point, the value of x should be zero. It wasn't before, and we hope that we fixed the problem. Let's see.

    printf( "x = %d\n", x );

Yay! The program takes appreciably longer to run, but it works.

Now that we know how to combine global variables and mutexes to communicate, let's try another exercise. Check out 2.txt.

7.7 Exercise 2

Write a C program with threads. Set up a global (shared) mutex and a character array (a string) 32 bytes long.
The main thread should:
1. start the child thread (see next point)
2. sleep for half a second
3. read the first 31 bytes of intro.txt into the global character array (in a critical section protected by the mutex)
4. wait for the child thread to finish
The child thread should:
1. print the content of the global character array
2. sleep for a second
3. print the content again
Both reads from the array in the child thread should be protected by the mutex.
Run the program to demonstrate it works as expected.

We can now move on to more complex synchronisation devices: semaphores and barriers. First, however, let's do one more thing with mutexes: busy waiting. See busy.c.

7.8 `busy.c` (source code)

This program demonstrates how to busy-wait for a condition using a shared global variable and a mutex.

#include <pthread.h> /* pthread_create, pthread_join */ 
#include <stdio.h>   /* puts, perror */
#include <unistd.h>  /* sleep */
#include <stdlib.h>  /* exit */

Set up a global variable and a mutex to protect it.

int proceed = 0; 
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;

Thread announces that it started, then waits for proceed to become non-zero and then finishes.

void *thread( void *arg ) 
{
    puts( "thread started" );
    while ( 1 )
    {

We enter the critical section:

        if ( pthread_mutex_lock( &mutex ) ) 
            perror( "pthread_mutex_lock" ), exit( 1 );

        if ( proceed )
        {
            puts( "thread allowed to finish" );

We must always unlock the mutex that we locked, otherwise we could cause deadlocks. This is very important!

            if ( pthread_mutex_unlock( &mutex ) ) 
                perror( "pthread_mutex_unlock" ), exit( 1 );

            return NULL;
        }

And leave the critical section again:

        if ( pthread_mutex_unlock( &mutex ) ) 
            perror( "pthread_mutex_unlock" ), exit( 1 );

    }
}

The main function just sets up the thread, waits for a while and then lets it proceed.

int main() 
{
    pthread_t tid;

Start the thread:

    if ( pthread_create( &tid, NULL, thread, NULL ) ) 
        return perror( "pthread_create" ), 1;

    puts( "main thread sleeping for 3 seconds" );
    sleep( 3 );

    if ( pthread_mutex_lock( &mutex ) ) /* critical section */
        return perror( "pthread_mutex_lock" ), 1;

    puts( "main thread setting proceed = 1" );
    proceed = 1;

    if ( pthread_mutex_unlock( &mutex ) ) /* critical section end */
        return perror( "pthread_mutex_unlock" ), 1;

And wait for the thread to finish.

    if ( pthread_join( tid, NULL ) ) 
        return perror( "pthread_join" ), 1;

    puts( "all done" );
    return 0;
}

We can now go on to semaphore.c.

7.9 `semaphore.c` (source code)

In this program, we will look at a semaphore, which is a generalization of a mutex. The structure of the program is the same as the one that demonstrated mutexes: mutex.c.

#include <pthread.h>   /* pthread_create, pthread_join */ 
#include <stdio.h>     /* printf, perror */
#include <unistd.h>    /* sleep */
#include <stdlib.h>    /* exit */
#include <semaphore.h> /* sem_init, sem_wait, sem_post */

The type for a semaphore is sem_t. Unlike pthread mutexes, semaphores can be used in multi-process programs too, but this also requires allocation of inter-process shared memory. You could try doing this as an optional exercise (you will need shm_open and mmap). In this program, we will use a semaphore with threads in a single process.

sem_t semaphore;

Unlike earlier programs, this one will run multiple threads (i.e. multiple child threads, instead of just one). All the child threads will, however, execute the same function. We will make use of the argument to tell those threads apart.

void *thread( void *arg ) 
{
    char *thread_name = arg;

    sleep( 3 );

Waiting for a semaphore enters the critical section. The number of threads that can simultaneously enter the critical section is configured by the parameters to sem_init, which is called by the main thread below.

    if ( sem_wait( &semaphore ) ) 
        perror( "sem_wait" ), exit( 1 );

Now that we are in the critical section, print a message and sleep for a few seconds, so it is clear that the main thread cannot enter the critical section while we are in it.

    printf( "%s in the critical section\n", thread_name ); 
    sleep( 3 );

The equivalent to unlocking a mutex is posting the semaphore, and the function to do so is called sem_post.

    if ( sem_post( &semaphore ) ) 
        perror( "sem_post" ), exit( 1 );

    printf( "%s done\n", thread_name );
    return NULL;
}

int main()
{
    pthread_t tid1, tid2, tid3;

Semaphores must be initialized by a call to sem_init. The function takes 3 parameters: the address of the semaphore object, whether this is an inter-process semaphore (not in our case, so we pass 0) and the number of threads that can get into the critical section at once (2, in our case).

    sem_init( &semaphore, 0, 2 ); 

    if ( pthread_create( &tid1, NULL, thread, "thread 1" ) ||
         pthread_create( &tid2, NULL, thread, "thread 2" ) ||
         pthread_create( &tid3, NULL, thread, "thread 3" ) )
        return perror( "pthread_create" ), 1;

Let's try to enter the critical section repeatedly (every second) and print a message whenever we get in.

    for ( int i = 0; i < 10; ++i ) 
    {
        if ( sem_wait( &semaphore ) )
            return perror( "sem_wait" ), 1;

        puts( "main entered critical section" );

        if ( sem_post( &semaphore ) )
            return perror( "sem_post" ), 1;

        sleep( 1 );
    }

And wait for the threads to finish.

    if ( pthread_join( tid1, NULL ) || 
         pthread_join( tid2, NULL ) ||
         pthread_join( tid3, NULL ) )
        return perror( "pthread_join" ), 1;

    return 0;
}

I think it is quite clear how a semaphore works; let's look at a barrier now, in barrier.c.

7.10 `barrier.c` (source code)

Barriers are dual to semaphores, in the sense that semaphores allow through at most a given number of threads. Barriers let through a given minimum number of threads instead -- a sufficient number of them must reach the barrier for any to be able to continue.

#include <pthread.h>   /* pthread_{create,join} */ 
                       /* pthread_barrier_{init,wait} */
#include <stdio.h>     /* printf, perror */
#include <unistd.h>    /* sleep */
#include <stdlib.h>    /* exit */

Like with mutexes and semaphores, there is a dedicated type for barriers, called pthread_barrier_t. Like with a semaphore, it must be initialized by a call to a function, pthread_barrier_init.

pthread_barrier_t barrier;

Like with semaphores, we will run multiple threads. They will all wait a certain number of seconds (given to them in their argument) and then proceed to synchronise on the barrier.

void *thread( void *arg ) 
{
    int n = (int) arg;

    printf( "thread %d sleeping for %d seconds\n", n, n );
    sleep( n );

The call to pthread_barrier_wait is quite simple, and unlike with mutexes and semaphores, is not paired. We don't check for errors, since the only error that could happen is that we pass an invalid barrier. Additionally, checking for the return code is more complicated, because the return value of pthread_barrier_wait is different in different threads. See the man page for details.

    pthread_barrier_wait( &barrier );

All threads should make it to this point at once.

    printf( "thread %d got past the barrier\n", n ); 
    return NULL;
}

int main()
{
    pthread_t tid1, tid2, tid3;

Barriers, like semaphores, are initialized by a call to an initialization function, in this case pthread_barrier_init. The arguments are: the barrier to initialize, an optional attribute object (which we do not care about) and the number of threads to synchronize on the barrier.

    pthread_barrier_init( &barrier, NULL, 4 ); 

    if ( pthread_create( &tid1, NULL, thread, (void *) 1 ) ||
         pthread_create( &tid2, NULL, thread, (void *) 2 ) ||
         pthread_create( &tid3, NULL, thread, (void *) 3 ) )
        return perror( "pthread_create" ), 1;

    printf( "all threads running\n" );
    printf( "main thread sleeping for 10 seconds\n" );

    sleep( 10 );
    pthread_barrier_wait( &barrier ); /* we also wait */

And wait for the threads to finish.

    if ( pthread_join( tid1, NULL ) || 
         pthread_join( tid2, NULL ) ||
         pthread_join( tid3, NULL ) )
        return perror( "pthread_join" ), 1;

    printf( "everything done\n" );

    return 0;
}

Again, it should be quite clear how barriers work. Let's try an exercise:

$ cat 3.txt

7.11 Exercise 3

The barrier provided by the operating system is quite smart: threads that wait are sleeping, not busy-waiting, making the entire program more efficient.
But in principle, a barrier is very simple. Let's try to make one ourselves.
Write a C program which only uses mutexes and shared variables to implement a barrier.
The waiting threads can busy-wait.

That's the last exercise for today, the homework is, as usual, in hw.txt.

7.12 Homework

In the seminar, we implemented a simple busy-waiting barrier.
Let's do that again, but this time with a semaphore.
You should create two files, ssem.c and main.c.
In ssem.c (for spin semaphore), implement the following functions:
1. ssem_init( int c ) which initializes the semaphore, c being the maximal number of threads to let in
2. ssem_wait() which waits for the semaphore
3. ssem_post() which unlocks the semaphore
4. there should not be a main function in ssem.c
The global variables that you need should also live in ssem.c. Note that there is only a single global (spin) semaphore, not a data type and methods for this type. This is unlike sem_t from the standard library.
In main.c, write a simple program that checks that semaphore works:
1. call ssem_init() so that 2 threads can enter at once
2. create 2 new (child) threads
3. all threads (including the main one) try to enter a critical section
4. the critical section is guarded by ssem_wait
5. each thread, upon getting into the critical section calls in_critical()
6. the function in_critical() simply prints a message to stdout

8 Networking

Today, we will learn a bit about networking.
In the first half, we will look at networking from the user perspective.
That includes a few diagnostic tools (ping, traceroute), general-purpose utilities (nc a.k.a. netcat and telnet).
But also clients for well-established protocols, like HTTP (curl and wget), SSH and DNS (dig).
In the second half, we will look at network (AF_INET) sockets.

Let's start with ping and traceroute: cat ping.sh.

8.1 `ping.sh` (source code)

There is a number of diagnostic utilities for IP networks, mostly based on the ICMP protocol. The most basic of those is ping, which asks another computer whether it's alive. The packets it sends are ICMP echo request, and the target host is supposed to send an ICMP echo reply back. Given no switches, ping will send a packet every second or so, forever. But we don't have that much time, so we request it only sends 5 and then stops, like this:

ping -c 5 1.1.1.1

The 1.1.1.1 above is an IP address. This one is particularly easy to remember -- it's a recursive DNS server operated by cloudflare. Notice the numbers that ping prints: those are statistics about how long it took for the packet to get there and back (rtt = round-trip time). Notice that it's only a few milliseconds -- that means the server is not very far from us. Magic is involved (and IP addresses that belong to different computers depending on where you are) Anyway, to prevent Google getting jealous, let's also ping their recursive DNS server, which lives at the address 8.8.8.8 (not as flashy as the cloudflare one, but not bad either):

ping -c 5 8.8.8.8

Let's just round it off with my favourite public recursive resolver, ran by Level3, living at the address 4.2.2.2 (I don't actually recommend using it for DNS... I just like to ping it):

ping -c 5 4.2.2.2

That's it for ping (with IP addresses), let's look at traceroute now, which is a little more interesting. The script lives in traceroute.sh.

8.2 `traceroute.sh` (source code)

Like ping, traceroute sends out ICMP echo request packets. However, unlike ping, it also sets the TTL field in the IP header -- short for Time To Live. This number is decremented by each router along the way, and the packet is thrown away when its TTL drops to 0. The router doing the dropping also helpfully sends back an ICMP ‘time to live exceeded’ packet. Someone clever noticed that you can send ICMP echo request with TTL set to 1 to learn the address of the first router along the way to the destination, then set TTL to 2 to learn about the second closest one, and so on. This is what traceroute does, and it prints information about each of the hosts it encounters. Let's try it on the trio of computers we pinged earlier: By default, traceroute tries to find a hostname for each IP address. But we don't know enough about DNS to worry about that yet. So we just ask it to print numeric addresses, using -n.

traceroute -n 1.1.1.1
traceroute -n 8.8.8.8
traceroute -n 4.2.2.2

Now that we tried both ping and traceroute, let's try some DNS queries. See host.sh.

8.3 `host.sh` (source code)

In this script, we will first learn about the host utility, which does basic DNS resolution for us. For instance, we can ask what is the IP address of aisa like this:

host aisa.fi.muni.cz.

We can also use the shorthand, since we are in the same domain:

host aisa

This has some interesting consequences:

host www

What do you think the last query means? Besides translating names to addresses, DNS can also go the other way: give a hostname for a numeric address. We can simply pass the IP address to the host utility, like this:

host 4.2.2.2

This uses some magic behind the curtain (though the magic also leaks into the user interface some). There is a special domain called in-addr.arpa, which has subdomains for all the possible IP addresses -- in reverse. So when we ask host to find the name for 4.2.2.2, it is in fact looking for 2.2.2.4.in-addr.arpa. There's magic involved because the normal recursion rules don't quite work in this domain. Nonetheless, there are DNS records (of type PTR) and they contain the ‘canonic‘ hostname of the computer that lives at the given IP address. You usually want the name to resolve back to the same IP address. In fact, let's refresh our shell scripting skills with an exercise (in 1.txt). After that, we'll look at dig which is another DNS tool and then we'll move on to nc.

8.4 Exercise 1

Write a shell script that does the following:
1. takes one argument, an IP address (see also below)
2. resolves the address to a hostname
3. resolves the hostname back to an (IPv4) address
4. prints ‘yes’ if they match
5. prints ‘no’ if they are mismatched
6. prints ‘error’ if an error happens
You can run the script as sh chkrdns.sh 4.2.2.2.
The address will be available in the variable $1.
Anything other than ‘yes’, ‘no’ and ‘error’ should go to stderr (but you can print what you like there).
To redirect the stdout to stderr for a command, you can use 1>&2 after the command, like so: echo hello 1>&2
Hints: this should be quite doable with just cut and $(command).

Let's continue to dig.sh.

8.5 `dig.sh` (source code)

Dig is a somewhat more flexible tool for doing DNS queries from the command line. The output is also more DNS-like so to understand it, you need to know a little about DNS. For our use-case, it should be sufficient to know that there are different types of records in DNS (carrying different types of info about hostnames): NS stands for name server (i.e. a DNS server responsible for a given subdomain), A stands for IPv4 address, AAAA stands for IPv6 and MX stands for mail exchanger. Let's try some basic queries:

dig aisa.fi.muni.cz -t A

The above performs a DNS query using the default system DNS recursive server. It does not perform recursion itself. The output is fairly long, so let's look at what's in there. Let's ignore the header, all the way until QUESTION SECTION. The question section shows us what the question we asked was. In this case, we would like the A record of aisa.fi.muni.cz filled in for us. What follows is an ANSWER SECTION, which contains the answer to our query: it tells us that the A record for aisa.fi.muni.cz contains the address 147.251.48.1. What follows after that is the list of name servers (NS records) responsible for the domain fi.muni.cz. and finally we get some A and AAAA records for those name servers too. In case we wanted to directly consult them. We will actually stop the script here, so you can dig (ha-ha) through all the output ; when you are ready to continue, just hit enter.

read

Since output from dig is so long and tedious, let's only do one more. This replicates what I have shown you in the lecture (but that was somewhat redacted, to fit in the slide). Let's also ask for a different type of record for the fun of it. The +trace tells dig to perform recursion itself and print all the intermediate results too.

dig fi.muni.cz -t MX +trace

The MX record tells us who handles mail for the given domain. So when you send an email to me (using my @fi.muni.cz address), the MTA that sends email for you will do an MX lookup for fi.muni.cz to find out which computer it should contact. In this case, it'll see:

8.6 `netcat.sh` (source code)

We will look at the client side of nc (netcat) in this script. We left off with dig when we learned that email for fi.muni.cz is handled by relay.muni.cz. Let's look at what would happen next in our hypothetical mail scenario. The MTA would learn the destination IP address and initiate an SMTP session. I won't get into much detail, but let's talk about nc for a bit. What nc does is basically what cat does, but then sends the data over a network. It can both listen to connections (with -l) or connect to existing services. We will now use the latter mode. In either case, whatever you send to nc, it will forward into the network, and what falls out of the network connection, nc will print to its stdout. Let's try with some SMTP:

{
    sleep 5; # wait a bit for the remote server to greet us
    echo HELO aisa # tell them who we are (hostname)
    echo "MAIL FROM:<not@me>" # and our mail address
    echo "RCPT TO:<yes@me>"   # and the recipient of the email

At this point, the SMTP server will stop talking to us because it does not want to send mail to yes@me -- that is fine, we didn't want to actually spam anyone. The rest of the SMTP session would mostly involve the actual message.

} | nc relay.muni.cz 25 # 25 is the SMTP port

Let me just note that nc is not quite standardized. It's present on many/most Unix systems, but the exact behaviour and switches might differ. Be careful and read man pages. Let's try looking at nc in server mode, in listen.sh.

8.7 `listen.sh` (source code)

set -x # ask the shell to print out what is going on

This is a simple script that demonstrates nc in listening mode. To wit:

port=2222
nc -l $port > file.txt &

Now unlike most of the things we did earlier, you may not be able to run this script. That's because port numbers are a global, shared resource, and if one person is listening on port 2222 on aisa, this port is no longer available for others. You may want to try with a different port number (must be greater than 1024, too). But since the script is brief, you might get lucky. Now let's send something to the listening nc using another nc, this time in client mode:

echo "hello world on port $port" | nc localhost $port

localhost is a special name, reserved to mean ‘this computer’. But we could have used aisa.fi.muni.cz just fine, too. Anyway, let's check the content of file.txt:

wait # wait for the backgrounded ‹nc -l› to finish
cat file.txt # print the content of the file

Let's do another exercise: 2.txt.

8.8 Exercise 2

Write a shell script that:
1. listens, using netcat, on port 2222
2. someone (e.g. you from another terminal) will send a message to that port, in this form: host1 host2 port
3. fetch, using http the / document from host1 (see also below)
4. send the result to host2 on port port
You can use a temporary file and cut to extract the information.
To fetch a document using HTTP, you simply send GET / on port 80, like this: echo GET / | nc www.fi.muni.cz 80.

This is almost all for the user part. We will have a brief look at ssh and curl before we delve into C again.

$ micro ssh.sh

8.9 `ssh.sh` (source code)

This is another simple script which shows that ssh can also be used for moving data across the network, like nc, but much more securely. Of course, there is some overhead involved, but in 98% of the cases, ssh is a much better choice.

set -x

First notice that ssh works non-interactively if we give it a command to execute, like this:

ssh aisa echo hello world

Of course we can pipe the output to a (local) command:

ssh aisa echo hello world | cut -f1 -d' '

We can also pipe stuff to the stdin of ssh and it appears as the stdin of the remote command, like the following. Unfortunately, we have to do double escaping, because the command is given to a shell at the remote end, but the quotes were already stripped by the local shell (when constructing argv for ssh).

echo hello world | ssh aisa "cut -f2 -d' '"

That's all for ssh. Let's move on to curl and then finally some C:

Please note that curl is not standard (but can be installed on most Unix systems). We can start by a simple curl invocation to fetch a document (the -s option tells curl to not print progress information to stderr):

echo fetch http://ix.io/1FWS
curl -s http://ix.io/1FWS
echo

Out of the box, curl prints the document to stdout -- this is convenient if we want to process the output immediately using some shell magic. But if we prefer to have it stored in a file, we can use curl -O, which will select a filename for us:

echo fetch https://www.fi.muni.cz/index.html
test -f index.html && exit 1 # die if the file is already here
curl -s -O https://www.fi.muni.cz/index.html
test -f index.html || exit 1 # die if the file did not materialize
ls -hl index.html
rm index.html
echo

The last option to curl (related to GET requests) to look at is -L, for ‘location‘. In some instances, the HTTP server may return a code like 301 which tells the client that they should be looking elsewhere. For instance:

echo fetch http://www.fi.muni.cz
curl -s http://www.fi.muni.cz
echo

In this case, we get a 301 Moved Permanently, with the instruction to look at the encrypted version of the site (i.e. https://www.fi.muni.cz). Since we don't want to handle such matters ourselves, we can use curl -L which will automatically follow the redirect:

echo "fetch http://www.fi.muni.cz (with redirects)"
curl -s -L http://www.fi.muni.cz | wc -c

That's for GET requests with cURL, let's go on to curl-post.sh.

8.11 `curl-post.sh` (source code)

This is a very simple script, which demonstrates how curl can be used to send data to a remote server, using so-called POST requests. This script can be really useful in case you want to send the output of a command or the content of a plain text file to someone across the internet. We use a web service called ix.io which will hold text for us, and lets us send it using a simple POST request. Basically a no-frills pastebin.

echo hello world | curl -F 'f:1=<-' http://ix.io

The -F argument to curl is short for ‘form’, and uses the POST protocol in a manner similar to how a web browser would use it to send data you filled in an HTML form. f:1 is the name of the key and the <- is an instruction to read the value for that key from stdin. After the script runs, you can try fetching (with just curl) the resulting URL. You should get back hello world. That's all for the user-level part, let's do some C:

$ micro ghbn.c

8.12 `ghbn.c` (source code)

This program demonstrates the use of the gethostbyname C function. It works a little like the host program: it gets a hostname as the first argument and prints the address it obtains from gethostbyname on standard output.

#include <netdb.h> /* gethostbyname, hstrerror, h_errno */ 
#include <stdio.h> /* fputs, fprintf */

int main( int argc, const char **argv )
{
    if ( argc != 2 ) /* check the number of arguments */
        return fputs( "expected 1 hostname\n", stderr ), 1;

The gethostbyname function is almost the simplest interface to the resolver that you could imagine. A string goes in and you get back a pointer to a structure with the address(es). If an error occurs, the pointer will be NULL.

    struct hostent *he = gethostbyname( argv[1] ); 

    if ( !he ) /* deal with a possible error */
    {

Unlike most C functions that we encountered so far, gethostbyname does not set errno. Instead, it has a similar variable called h_errno and a function hstrerror to turn the numeric value of h_errno into a human-readable string. You can try calling this program on a non-existent domain to see how this works.

        fprintf( stderr, "error resolving host %s: %s\n", 
                argv[1], hstrerror( h_errno ) );
        return 2;
    }

If we got here, hostname resolution worked and we have an address in he. Thing is, one host can have more than a single IP address, so what gethostbyname gives you is a list that works a bit like argv. But we only care about a single address now, so let's just print it. See also

man
gethostbyname

    unsigned char *addr = he->h_addr_list[0]; 
    printf( "host %s has address %d.%d.%d.%d\n",
            argv[1], addr[0], addr[1], addr[2], addr[3] );
    return 0;
}

We can now go on to http-get.c.

$ printf "%02x%02x%02x%02x" 147 251 48 1

const uint32_t REMOTE_IP   = 0x93fb3001; 
const uint16_t REMOTE_PORT = 80;

int main( int argc, const char **argv )
{

We first create a socket. The address family is INET (for IPv4 sockets) and the type is STREAM, for pipe-like operation. This pretty much implies TCP. As usual, we left the protocol to be filled in by the OS.

    int sock_fd = socket( AF_INET, SOCK_STREAM, 0 ); 

    if ( sock_fd == -1 ) /* something went wrong, bail */
        return perror( "socket" ), 2;

We now set up the address. The structure is slightly more complicated than the sockaddr_un that we used two weeks ago. There is a further bit of complication stemming from the fact that the networking API uses so-called network byte order -- also known as big endian -- to represent numbers. But the CPU in aisa (and in most contemporary computers) represent numbers the other way around, as little endian. There are a few helpful functions to convert between those formats. We will use two: htons and htonl which are short for Host TO Network Short and Host TO Network Long and convert 16 and 32 bit numbers respectively. The opposite direction is done by ntohs and ntohl. You must always remember to use those functions when filling in sockaddr_in with native integers, otherwise bad things will happen.

    struct sockaddr_in sa = { .sin_family = AF_INET, 
                              .sin_addr = { htonl( REMOTE_IP ) },
                              .sin_port = htons( REMOTE_PORT ) };

From this point, there's nothing really new. Let's connect the socket to the remote host/port.

    if ( connect( sock_fd, ( struct sockaddr * ) &sa, 
                  sizeof( sa ) ) == -1 )
        return perror( "connect to 147.251.48.1:80" ), 2;

Send the HTTP request.

    const char *request = "GET / HTTP/1.0\r\n\r\n"; 
    int reqlen = strlen( request );
    if ( write( sock_fd, request, reqlen ) != reqlen )
        return perror( "write" ), 2;

Okay, I lied a little. This part is somewhat new, though we know all the ingredients. What it does is read everything that the remote side sent and print it out to stdout, in chunks of at most 32 bytes.

    char buf[ 32 ]; 
    int bytes;

    while ( ( bytes = read( sock_fd, buf, 32 ) ) > 0 )
        write( 1, buf, bytes );

    close( sock_fd );
    return 0;
}

That's all of the demos for today, let's do another exercise and then you can get started on the homework. The exercise is in 3.txt.

8.14 Exercise 3

Combine gethostbyname with the socket client code to fetch a document given on the command line.
The host to contact is the first argument.
The filename part of the URL to fetch is the second argument.
Print the reply to stdout.

E.g. ./a.out ix.io /1FWS will send the following data:

 GET /1FWS HTTP/1.1\r\nHost: ix.io\r\n\r\n
     ^^^^^ the url           ^^^^^ hostname

The homework is, as usual, in hw.txt.

8.15 Homework

Write a program, echo.c, which:
1. connects to the TCP port 2222 of localhost
2. reads a message of up to 20 bytes from the connected socket
3. sends back the string "you said <message>" over the socket
4. closes the connection and exits with status 0
The IP address of localhost is 127.0.0.1
Write a shell script echo.sh, which:
1. compiles the program from point 1 into a.out
2. listens on port 2222 using nc
3. starts a.out from point 3.a
4. sends message ping to a.out from 3.c, using the listening nc instance started in 3.b
5. prints the message from a.out onto stdout
6. waits until nc from point 3.b terminates

9 Scripting Redux

Today will be slightly different than usually. We will only do exercises, since you already know most of the ingredients.
You can refer to examples from previous seminars, lecture slides and of course man pages.
In case a program you don't know could be useful, I will give you a name (so you can look it up), or a very short example.
If you remember a keyword, a program name or any other fragment of information from previous seminars, but aren't sure where it was, a good command to try is grep -r keyword ~/pb152.
There are 10 exercises today, plus the homework, in hw.txt.

That said, let's start with 01.txt.

9.1 Exercise 1

Let's refresh our memory of shell scripts and pipes. Write a script, that:

uses ~/pb152/01/oslist.csv as the input,
prints the number of operating systems by Microsoft,
prints the year and name of all operating systems by Apple,
prints the list of all mono- and microkernel operating systems.

That should have been fairly easy. Let's move on to 02.txt.

9.2 Exercise 2

Let's look at compiling programs. And also writing some very simple C:

write a C program that:
1. prints its first argument (if present),
2. prints no arguments if it got no arguments,
write a shell script that:
1. compiles the program in steps (preprocess, compile, assemble, link),
2. the binary name should be a.out,
3. runs the program with all shell scripts in ~/pb152/08 as arguments,
4. checks that ./a.out hello world prints hello,
5. if it does, the script prints it works,
modify the program to print its second argument instead (print nothing if there is only one) and re-run the script from point 3 (submit this version).

9.3 Exercise 3

Write a script that:
1. creates an empty file called file.txt,
2. creates a symlink to this file, called symlink.txt,
3. creates a hardlink to file.txt called link.txt,
4. writes foo into symlink.txt,
5. prints the target of symlink.txt.
Write a script that:
1. checks if the target of alink.txt exists,
2. if it does, it prints the target,
3. otherwise, it prints alink.txt is broken.
Create a valid and a broken alink.txt and run the script from point 2 to check that it works.

9.4 Exercise 4

Write a C program that prints the i-node number of a file it gets as its first argument.
Write a makefile that compiles this program.

9.5 Exercise 5

Write a shell script that prints the vendor, year and name (in this order) of each operating system from oslist.csv, sorted by vendor first and name second.

Hint: try the following command:

$ echo 'hello,world' | sed -re 's:([^,]*),([^,]*):\2,\1:'

9.6 Exercise 6

Write a script that renames each file in the current directory to only use minuscule letters. E.g. IMG_32.jpg would be renamed to img_32.jpg.
Create some test files and run the script to check it works.
Hint: try the following command:
```
$ echo Hello W0rLD | tr A-Z a-z
```

9.7 Exercise 7

Write a script that, for each file in the current directory, moves and/or renames files like this:
- example_file.txt becomes example/file.txt,
- bar.txt stays unchanged,
- some_longer_name.sh ends up as some/longer/name.sh.

Create any directories that are missing.

Try the following commands:

$ echo some_longer_name.sh | sed -e s,_,/,g
$ dirname some/longer/name.sh

9.8 Exercise 8

Write a script that removes all broken symlinks from the current directory. For each symlink, print whether it is OK or broken (and hence is being erased).

9.9 Exercise 9

Write a script that prints files in the current directory, sorted by their suffix. Print directories first (sorted by name).

9.10 Exercise 10

Write a script that builds a program from multiple sources.
It takes a list of files as arguments (i.e. any number of arguments, one file name per argument).
If the file is a .c or .s, compile it to an object.
If the file is a .o, just keep it.
Link all the .o files (including those compiled from sources) into a single executable (named a.out).
If an argument does not exist or is not a .c, .s or a .o file, print an error and exit with a non-zero code.
Make the script print the actions it takes, test it.

9.11 Homework

Write a script, hw.sh, that prints names of files in the current directory.
The output should be sorted by i-node number in ascending order.
Use the program from exercise 4 to obtain i-node numbers.
Assume the program can be executed as ./04.

10 Access Control

In this seminar, we will look at some of the access control concepts.
Unfortunately, most of the interesting stuff is off limits for us, since we are mere users (and not root).
This means we can't create users or groups, or change ownership of files.
We can look at some of the APIs and some of the basic commands though.

Let's start with something simple: see perm.sh.

10.1 `perm.sh` (source code)

Let's first look at how standard UNIX permissions are encoded in the user interface. We will look at the underlying numeric representation later on.

set -x
touch file.txt

We have created an empty file, so that we can look at the permissions it has. The encoding might seem odd at first, so let's explain how it works.

ls -l file.txt

The first column of the output from ls -l shows the permissions, and it reads -rw-r--r--. This is the ‘human readable’ encoding of UNIX permission bits. The first dash is for file type, dash meaning a regular file (d would mean a directory, l a symlink, and so on). However, what follows is more interesting for us now. The remaining 9 letters form 3 groups of 3 letters each, like this: rw-, r-- and r-- again. Each of the letter groups encodes access rights of a particular set of users: the first belongs to the owner of the file (in this case, ourselves), the second belongs to the owning group and the last triplet applies to all users that do not belong to the owning group of the file. The r indicates reading, while w indicates writing: the owner (us) can read and write the file, while both remaining sets of users can only read the file. The third common letter is x and denotes the ability to execute the file (as a program). The letters have fixed positions: r always comes in the first position, w in the second and x in the third. A dash (-) in the corresponding position means this particular right is denied. We can check:

ls -l /bin/echo

The owner of /bin/echo is root and the owning group is also root (those are the third and fourth columns of the output above). The permission bits are rwx (the owner can read, write and execute the file), r-x (the root group can read and execute, but not write) and again r-x (in this case, this applies to us). Specifically, we are not allowed to write into /bin/echo:

echo hello > /bin/echo # fails with a permission error

However, we could read and execute it. Let's try the latter:

/bin/echo hello # this works okay

Finally, let's double-check that file.txt cannot be executed (it lacks any x bits):

./file.txt

Let's try changing some permission bits now (in chmod.sh).

10.2 `chmod.sh` (source code)

The chmod command serves to change permission bits of files (well, i-nodes, so that covers directories, specials and so on). Let's start by creating an empty file:

echo creating file.txt...
touch file.txt
echo hello > file.txt # this also works
cat file.txt # this works as expected

We have checked that we can read the file and write into it -- this is normal and expected. However, we can change the permissions (using chmod) to remove either of those access rights (or both). First, we check what the initial permissions were:

ls -l file.txt

Let's look at chmod then. What it does is manipulate the individual bits, but using a somewhat different syntax. The formatting we see in the output of ls is optimized for output, but writing out the permissions in this form could be somewhat tedious, especially if all we want is to flip one of the bits. For this reason, chmod gives names to each of the triplet: u for user (owner), g for owning group and o for others. Beware that o does not mean the owner! What we can do now is tell chmod which bits we are interested in and what to do with them. We can either set bits with + or remove them with -. For instance, g+w would mean ‘add the w bit to the group triplet’, or in other words, let the users in the owning group write into this file. To remove the write permission from ourselves (the owner), we will use u-w, like this:

echo
echo removing the write permission and testing...
chmod u-w file.txt
ls -l file.txt
echo world >> file.txt

Likewise, we can remove the read permission from ourselves, using u-r. Then we won't be able to read from the file either:

echo
echo removing the read permission and testing...
chmod u-r file.txt
ls -l file.txt
cat file.txt # permission error

Multiple bits can be specified in a single group. Let's restore the original permissions:

echo
echo "restoring permissions (and testing again)"
chmod u+rw file.txt
ls -l file.txt
echo world >> file.txt
cat file.txt

If you want to specify multiple groups in a single chmod command, that can be done too: separate the groups with a comma, like this: g+r,o-w. You can also set the permissions directly, using assignment, or combine the two alternatives: chmod u=rw,g+x). With that out of the way, let's try an exercise (in 1.txt).

10.3 Exercise 1

Write a script that creates files with the following permissions (using touch and chmod):
1. r-xr-xr-x file1.txt
2. rwxr--r-x file2.txt
3. ---r--r-x file3.txt
4. rw-r-xr-- file4.txt
5. rwxr-xr-- file5.txt
6. rwxr-xr-x file6.txt

Which of the above permission sets make sense? What would you say is the property that sensible permission sets have?

Let's look at encoding permissions in octal next, in octal.sh.

0 → 000
1 → 001
2 → 010
3 → 011
4 → 100
5 → 101
6 → 110
7 → 111

The digits are written in the same order as with the human readable form: owner, group and others, from the left. Hence:

touch file.txt

printf "755: "
chmod 755 file.txt
ls -l file.txt

printf "750: "
chmod 750 file.txt
ls -l file.txt

printf "644: "
chmod 644 file.txt
ls -l file.txt

Let's have a quick look at how directory permissions work now: dir.sh.

10.5 `dir.sh` (source code)

Let's first create a directory and look at its default permissions.

mkdir dir
touch dir/file.txt
ls -ld dir
echo

set -x # print what is going on

Then remove all the ‘execute’ bits:

chmod 644 dir
cd dir # permission error
ls dir # ok
echo

Let's now try with only the execute bits:

chmod 111 dir
cd dir # ok
ls # not ok
cd ..
echo

How about r-x------? i.e. deny writing to ourselves

chmod 500 dir
rm dir/file.txt # not ok
echo

Go back to where we started and clean up.

chmod 755 dir
rm dir/file.txt
rmdir dir

Next up: umask.sh.

10.6 `umask.sh` (source code)

Unfortunately, creating files and then changing their permissions is a race condition: some other user could come at the wrong moment, and open the file while it had incorrect permissions (between creation and chmod). In most cases, it is not a huge risk, but it is a risk and in certain circumstances can lead to serious security problems. There are multiple mechanisms that can be used to close this loophole, one of them being umask. The other is specifying the permissions at the time the file is being created, but this is mainly an option for C programs. On the user side of things, not many tools allow the user to specify the permissions they want for a particular new file. This is where umask comes in: the operating system maintains a per-process permission mask (called umask) which is subtracted from permissions of each newly created file. Most shells allow the user to manipulate this mask, and all programs executed by the shell inherit the mask from the shell (since it is inherited across fork). When a file is created, all the bits that are set in the umask are cleared:

echo umask 777
umask 777
touch umask.txt
ls -l umask.txt
rm -f umask.txt

The clearing of permission bits from umask happens atomically: a file with the permission bits set does not exist, not even temporarily. Of course, 777 makes little sense as an actual umask, but it does demonstrate what is going on. A more reasonable (though also somewhat paranoid) option would be 077, which means new files are only accessible to the owner (i.e. the creator of the file, i.e. us):

echo
echo umask 077
umask 077
touch umask.txt
ls -l umask.txt
rm -f umask.txt

Let's try to create a file that is marked executable by the tool that creates it. A compiler is a good example:

echo
echo umask 077 with cc
echo 'int main() {}' > empty.c
cc empty.c
ls -l a.out
rm -f a.out

A somewhat less reasonable umask (again only useful for demonstration purposes) would deny any executable bits:

echo
echo umask 133 with cc
umask 133
cc empty.c
ls -l a.out
rm -f a.out

And finally, a fairly common default would be 022: this prevents anyone but the owner from writing into new files, but anyone else can read and/or execute them if they wish so.

echo
echo umask 022
umask 022
touch file.txt
cc empty.c
ls -l a.out file.txt
rm -f a.out file.txt

Possibly useful: if you run umask without arguments, it will print the currently active umask. That said, let's look at creating files in C now, and specifically setting their permissions upon creation. See perm.c.

10.7 `perm.c` (source code)

Let's look at how to set permissions when we create files in C.

#include <fcntl.h>    /* open */ 
#include <sys/stat.h> /* chmod, fchmod */
#include <unistd.h>   /* close */

int main()
{

You might remember that we used open to create new files. At the time, the mysterious third argument remained without a proper explanation. As you can surely see now, this is nothing else but an octal representation of permissions. In C, a leading 0 in a numeric literal means the number is written out in octal (just like 0x means what follows is hexadecimal). Please keep in mind that the operating system will apply umask to the permissions given here in the third argument.

    int fd1 = open( "create1.txt", O_CREAT | O_RDWR, 0711 ); 
    int fd2 = open( "create2.txt", O_CREAT | O_RDWR, 0600 );
    int fd3 = open( "create3.txt", O_CREAT | O_RDWR, 0600 );
    int fd4 = open( "create4.txt", O_CREAT | O_RDWR, 0600 );

Besides setting the permission bits at file creation time, we can change them using two system calls that resemble the chmod program. However, in C, we always must supply the permission bits as a number (and for convenience, we usually stick to octal literals). The chmod syscall is straightforward: it takes a file path and a number. See

man
2 chmod

. Of course it might fail, but we currently don't care.

    chmod( "create3.txt", 0 );

The fchmod function is a little different, because it does not use a file path to change the permission bits. Rather, it takes a file descriptor, which corresponds to an i-node reference and changes the i-node via this fd. Again, the function may fail.

    fchmod( fd4, 0100 );

That's it, let's clean up.

    close( fd4 ); 
    close( fd3 );
    close( fd2 );
    close( fd1 );
}

With this sorted out, let's look at another exercise, this time in C: 2.txt.

10.8 Exercise 2

Write a C program that creates an empty file with permissions given on the command line. The name of the file comes first, then comes a single octal number representing the permissions.
To convert the number from a string to an actual number, use strtol with base set to 8 and endptr set to NULL. See also man strtol.
You will want to call the C function umask, which works the same as the shell builtin we have seen before.

This is pretty much all about permissions / access modes. Let's have a look at users using shell commands for a bit -- we will look at C APIs later on. Next stop: users.sh.

10.9 `users.sh` (source code)

While we can't manipulate users, there is a bunch of commands that print information and which are available to us. Let's have a look. The first command to check out is called simply w. Note that the command head as used here discards all but the first 5 lines of output.

echo output from w
w | head -n 5

The command prints information about users who are logged on, along with commands that they are currently executing. A more basic (and more standard) variant is a command called who, which only prints a subset of the info provided by w. Most notably, the currently executed command is missing:

echo
echo output from who
who | head -n 5

Feel free to experiment with those commands yourself. To learn your own username, you can use the command whoami (who am i?), like this:

echo
echo info about the current user
whoami

To print the list of groups attached to the active session, we can use groups. There can be more than one.

groups

Finally, the info from the two previous commands, along with some extras, can be obtained from id. Unlike groups, this tells us which group is primary and also gives numeric identifiers for each piece of information. The output of id is meant to be machine readable.

id

The last two commands can be also invoked on other users (i.e. not yourself). This works as follows:

echo
echo info about the user kontr
groups kontr
id kontr

This is about all the user- and group-related commands that we will look at. Let's have a quick look at passwd.txt before going on to C.

10.10 User Database

A simple, local database of users and groups is, on UNIX computers, stored in text files in the /etc directory. This database is usually present even when it is supplemented by a network-based database like LDAP (which is the case on aisa and on school computers in general).

10.13 `getpwent.c` (source code)

The first thing we need to do is set a so-called feature macro, because out of the box, the getpwent function is not visible. Just consider this to be a magic line that lets us use getpwent.

#define _POSIX_C_SOURCE 201800 
#define _BSD_SOURCE /* needed for getpwent */

#include <pwd.h>   /* getpwent */
#include <stdio.h> /* printf */

int main()
{

Each time getpwent is called, it returns the record corresponding to the next line of /etc/passwd. We will call it 3 times and hence print out 3 user records.

    int i; 

    for ( i = 0; i < 3; ++i )
    {
        struct passwd *pw = getpwent(); /* obtain the next entry */

We have seen structures like this before (e.g.

struct
stat

). The structure has various fields, the most interesting of which we will print. There isn't much else to this.

        printf( "name: %s\n", pw->pw_name ); 
        printf( "uid: %d\n", pw->pw_uid );
        printf( "gid: %d\n", pw->pw_gid );
        printf( "home: %s\n", pw->pw_dir );
        printf( "shell: %s\n", pw->pw_shell );
        printf( "\n" );
    }
}

Next: getpwuid.c.

10.15 `getpwnam.c` (source code)

#define _POSIX_C_SOURCE 201800 /* as before */
#include <pwd.h>   /* getpwnam */
#include <stdio.h> /* printf */

This program prints the basic information about a user whose login name is given to it as a command-line argument.

int main( int argc, const char **argv ) 
{
    if ( argc != 2 ) /* check command line arguments */
    {
        fputs( "expected 1 username as an argument\n", stderr );
        return 1;
    }

Like getpwuid, the getpwnam function looks up user records, but this time by their login (user name).

    struct passwd *pw = getpwnam( argv[1] ); 
    printf( "name: %s\n", pw->pw_name );
    printf( "uid: %d\n", pw->pw_uid );
    printf( "gid: %d\n", pw->pw_gid );
    printf( "home: %s\n", pw->pw_dir );
    printf( "shell: %s\n", pw->pw_shell );
    printf( "\n" );
    return 0;
}

Next: getgrent.c.

int main() 
{
    int i, j;

We first skip the first 10 entries because they are quite boring.

    for ( i = 0; i < 10; ++i ) 
        getgrent();

    for ( i = 0; i < 3; ++i )
    {
        struct group *grp = getgrent();

Groups have names and identifiers, just like users... let's print those first.

        printf( "name: %s\n", grp->gr_name ); 
        printf( "gid: %d\n", grp->gr_gid );

However, users only have a fixed number of fields attached to them. The situation with groups is somewhat different, because they contain a list of all users which belong to the group. We don't know beforehand how many such entries there are. The information is stored just like argv: an array of strings. We know we hit the end when we encounter a NULL string (pointer). Let's print all the member users then:

        for ( j = 0; grp->gr_mem[j]; ++j ) 
            printf( "member: %s\n", grp->gr_mem[j] );
        printf( "\n" );
    }
}

Next: 3.txt.

10.17 Exercise 3

Write a C program that prints the groups of the current user (not the current process, i.e. not what getgroups does).
You can either use getgrent and iterate, or you can use getgrouplist. However, you should know that the latter is not part of POSIX.
You will also need getuid and getpwuid.
Recall that strings in C are compared using the function strcmp.

That's all for today. Follows homework, in hw.txt.

10.18 Homework

Write a C program, user.c, which does the following:
1. if given an argument, that argument is a user name
2. print the numeric id of that user
3. print the numeric id of the primary group of the user
4. print the name of the primary group of the user
5. if no argument is given, print the name of the current user, along with c. and d.
You can use any of: getpwent, getpwnam, getpwuid, getgrent, getgrnam and getgrgid.

Print each piece of info on a separate line, with no extra text.

11 C API Redux

Like we did two weeks ago with shell, we will review some of the C programming that we did over the course. Since C is somewhat harder and more time-consuming than shell, there are only 8 exercises this week.

11.1 Exercise 1

Implement a C program that does what ln does: take 2 arguments and create a hardlink of a file given by the first under the path given by the second.

11.2 Exercise 2

Implement a small C program that does what tee does: copy its standard input onto its standard output, while also copying it into a file given to it as an argument (unlike standard tee, your program only needs to write into a single file).

11.3 Exercise 3

Implement a C program that runs another program given in the first argument (passing all other args to this program). Recall execvp from seminar 6. Print for how many seconds this other program was running (see also man 3 time).

11.4 Exercise 4

Implement a C program that limits the time for which another program may run. The first argument is the number of seconds for the time limit (recall atoi), the remaining arguments are the program to run along with arguments.

	pts	given	try 1	try 2	try 3	try 4	try 5	try 6
01	2	5.3.	8.3.	10.3.	12.3.	15.3.	17.3.	19.3.
02	3	12.3.	15.3.	17.3.	19.3.	22.3.	24.3.	26.3.
03	2	19.3.	22.3.	24.3.	26.3.	29.3.	31.3.	2.4.
04	3	26.3.	29.3.	31.3.	2.4.	5.4.	7.4.	9.4.
05	3	2.4.	5.4.	7.4.	9.4.	12.4.	14.4.	16.4.
06	4	9.4.	12.4.	14.4.	16.4.	19.4.	21.4.	23.4.
07	3	16.4.	19.4.	21.4.	23.4.	26.4.	28.4.	30.4.
08	3	23.4.	26.4.	28.4.	30.4.	3.5.	5.5.	7.5.
09	2	30.4.	3.5.	5.5.	7.5.	10.5.	12.5.	14.5.
10	2	7.5.	10.5.	12.5.	14.5.	17.5.	19.5.	21.5.
11	3	14.5.	17.5.	19.5.	21.5.	24.5.	26.5.	28.5.
12	2	21.5.	24.5.	26.5.	28.5.	31.5.	2.6.	4.6.

	pts	given	deadline	review	resubmit
02	3	12.3.	26.3.	2.4.	9.4.
04	3	26.3.	9.4.	16.4.	23.4.
05	3	2.4.	16.4.	23.4.	30.4.
06	4	9.4.	23.4.	30.4.	7.5.
07	3	16.4.	30.4.	7.5.	14.5.
08	3	23.4.	7.5.	14.5.	21.5.
10	2	7.5.	21.5.	28.5.	4.6.
11	3	14.5.	28.5.	4.6.	–

A Preliminaries

A.1 Intro

A.2 Rules of the Game

A.2.1 Homework

A.2.2 Assignment Grading

A.2.3 Plagiarism

A.2.4 Summary

A.3 Assignment Guidelines

A.3.1 Submitting Solutions

A.3.2 Evaluation

A.4 Peer Reviews

A.4.1 Requesting Reviews

A.4.2 Writing Reviews

1 First Contact

1.1 A Few Useful Tricks

1.2 Files

1.3 Command-Line Interfaces

1.4 files.sh (source code)

1.5 Exercise 1

1.6 Exercise 2

1.7 Pipes

1.8 Exercise 3

1.9 Exercise 4

1.10 Homework

2 Descent into C

2.1 hello.c (source code)

2.2 Compiler

2.3 read.c (source code)

2.4 Exercise 1

2.5 args.c (source code)

2.6 Exercise 2

2.7 System Libraries

2.8 Archives

2.9 Exercise 3

2.10 stdio.c (source code)

2.11 Redirects

2.12 Exercise 4

2.13 Homework

3 More Shell Scripts, make

3.1 variables.sh (source code)

3.2 env.sh (source code)

3.3 getenv.c (source code)

3.4 Exercise 1

3.5 test.sh (source code)

3.6 chaining.sh (source code)

3.7 loops.sh (source code)

3.8 Exercise 2

3.9 make

3.10 build.mk (source code)

3.11 Exercise 3

3.12 Exercise 4

3.13 suffixes.mk (source code)

3.14 autodep.mk (source code)

3.15 Homework

4 Working with Files

4.1 stat.sh (source code)

4.2 stat.c (source code)

4.3 errno.c (source code)

4.4 directories.sh (source code)

4.5 rmmkdir.c (source code)

4.6 Exercise 1

4.7 rm.sh (source code)

4.8 unlink.c (source code)

4.9 Exercise 2

4.10 links.sh (source code)

4.11 badlink.sh (source code)

4.12 links.c (source code)

4.13 Exercise 3

4.14 Exercise 4

4.15 readdir.c (source code)

4.16 Homework

5 Processes

5.1 proc.sh (source code)

5.2 ps.sh (source code)

5.3 Exercise 1

5.4 fork.c (source code)

5.5 wait.c (source code)

5.6 Exercise 2

5.7 exec1.c (source code)

5.8 exec2.c (source code)

1.4 `files.sh` (source code)

2.1 `hello.c` (source code)

2.3 `read.c` (source code)

2.5 `args.c` (source code)

2.10 `stdio.c` (source code)

3 More Shell Scripts, `make`

3.1 `variables.sh` (source code)

3.2 `env.sh` (source code)

3.3 `getenv.c` (source code)

3.5 `test.sh` (source code)

3.6 `chaining.sh` (source code)

3.7 `loops.sh` (source code)

3.9 `make`

3.10 `build.mk` (source code)

3.13 `suffixes.mk` (source code)

3.14 `autodep.mk` (source code)

4.1 `stat.sh` (source code)

4.2 `stat.c` (source code)

4.3 `errno.c` (source code)

4.4 `directories.sh` (source code)

4.5 `rmmkdir.c` (source code)

4.7 `rm.sh` (source code)

4.8 `unlink.c` (source code)

4.10 `links.sh` (source code)

4.11 `badlink.sh` (source code)

4.12 `links.c` (source code)

4.15 `readdir.c` (source code)

5.1 `proc.sh` (source code)

5.2 `ps.sh` (source code)

5.4 `fork.c` (source code)

5.5 `wait.c` (source code)

5.7 `exec1.c` (source code)

5.8 `exec2.c` (source code)

6.1 `pipes.sh` (source code)

6.2 `forkfd.c` (source code)

6.3 `pipe.c` (source code)

6.4 `pipe-ipc.c` (source code)

6.6 `dup.c` (source code)

6.7 `dup2.c` (source code)

6.9 `socketpair.c` (source code)

6.10 `client.c` (source code)

6.11 `server.c` (source code)

7.1 `thread.c` (source code)

7.2 `shvar.c` (source code)

7.4 `race.c` (source code)

7.5 `mutex.c` (source code)

7.6 `mtxvar.c` (source code)

7.8 `busy.c` (source code)

7.9 `semaphore.c` (source code)

7.10 `barrier.c` (source code)

8.1 `ping.sh` (source code)

8.2 `traceroute.sh` (source code)

8.3 `host.sh` (source code)

8.5 `dig.sh` (source code)

8.6 `netcat.sh` (source code)

8.7 `listen.sh` (source code)

8.9 `ssh.sh` (source code)

8.10 `curl.sh` (source code)

8.11 `curl-post.sh` (source code)

8.12 `ghbn.c` (source code)

8.13 `http-get.c` (source code)

10.1 `perm.sh` (source code)

10.2 `chmod.sh` (source code)

10.4 `octal.sh` (source code)

10.5 `dir.sh` (source code)

10.6 `umask.sh` (source code)

10.7 `perm.c` (source code)

10.9 `users.sh` (source code)

10.11 `getid.c` (source code)

10.12 `getgroups.c` (source code)