Unix standard streams, a basic unix process that always tied with (stdin, stdout, and stderr). Probably you may use it in your software without even knowing it. Let's understand deeper and how you can make use of the standard streams and apply into your workflow to gain the development productivity. This article won't explain and cover the standard streams in details, however the main aim of this article would be focused on using the three standard streams and the basic unix Standard I/O (Input / Output) operator like pipeline and redirection.

Standard Streams

Starting with the Standard Streams, it is the most basic fundamentals in the Unix and Unix-Like Operating system like Linux, BSD and Mac OS. By default, it consists of three main file process such as.

1. stdin that means standard input,

2. stdout that means standard output, and

3. stderr that means standard error.

Every application must use of those three standard I/O above to process input and output. As we know, every application have many processes. In each process, they use three standard file descriptors in the unix-like operating system, stdin as the input, stdout as their basic output, and stderr as the error output. To visualize what are those our computer, it can be demonstrated using the picture below.

File Descriptor in standard streams

In the picture above, there is a process will interact with a three numbers, which 0 that belong to stdin, 1 to stdout, and 2 to stderr. The number represented is the file descriptor opened in each process. Since unix system follow the philosophy of "everything as a file", then the standard streams would be accessible from /dev/stdin as the input stream file, /dev/stdout as the standard output stream, and /dev/stderr as the standard error output stream. Digging deeper in the golang standard library, the Stdin, Stdout, and Stderr files are predefined with path explained before.

Let's write a simple demo program using go that use stdin, stdout, and stderr.

package main

import (
    "fmt"
    "io"
    "os"
    "strings"
)

func main() {
    stdin, err := io.ReadAll(os.Stdin)

    if err != nil {
        panic(err)
    }
    str := string(stdin)

    lowercased := strings.ToLower(str)

    fmt.Fprintf(os.Stdout, "Hi %s", strings.ReplaceAll(lowercased, "\n", ", "), )
    fmt.Fprintf(os.Stdout, "\n")
    fmt.Fprintf(os.Stderr, "There is no error, this is just an example\n")
}

The simple go code above would do the following actions:

1. Read all input given in the standard input.

2. Convert the input string to be all lowercase and concatenate them if the inputs is more than one line.

3. Print the lowercased string into standard output.

4. Print something into standard error just for example.

While running those program, we will get the input from the os.Stdin (in golang, it is the standard input stream file. The input then will be buffered into the stdin variable that turned into lowercased string later.

To print an output to the terminal, we prefer using the fmt.Fprintf() function instead of fmt.Println() because under the hood, the fmt.Println method also write bytes to the os.Stdout (golang standard output file). It is the same though, it just to simplify our learning points about the unix basic standard streams. To print out to the standard error output, we also use fmt.Fprintf() function to print the error message to os.Stderr that will be explained later in this post.

In this post, to make sure that you will be learning something, the post will prefer using screenshot only for the command used in the example. This is intended to aim you to try and type the command by yourself.

Standard Input (stdin)

First thing first, let's build the program above using the following command.

go build main.go

The output binary file should be named main in the same directory. Before execution, we will explain several standard input operators such as pipe operator that can be denoted with the | and the < less than operator.

Input Redirection < Operator

Input redirection means that we can replace the input using the files or custom input we decided to use. In this example we use a separated file named names.txt that contains three names in the file.

To redirect the standard input, we can use < (less than) operator. The convention of its full command would be the binary < input . It can be shown in the example below.

The picture above show that we can redirect the input using the < operator to replace the input data streams from /dev/stdin to names.txt.

Pipe | Operator

Pipe will be useful to combine two commands, it will take the output of the first command as the inputs of the second command. Let's take the example of the pipe operator usage in the terminal history below.

The commands used in the picture above is echo that print something to the standard output, and the main command (the previous simple application) will take the echo command as the input of the program.

Hi hai,

There is no error, this is just an example

In the first run, the output "Hai" will be used in the main command as the standard input. Our main command do the lowercasing of the "Hai" input and printed out as "hai" with the following additional output in the terminal.

In the second run, the output from echo command is "Rendy" that will be lowercased by executing the main command as it gave the output below.

Hi rendy,

There is no error, this is just an example

Take another look with another command, ls to show our files in the current directory. Combining with pipe operator and main command gave us similar output.

The ls command giving output of the files inside the directory and piped with the main command. All of the files would nicely concatenated using comma.

Standard Output (stdout)

Standard output has two operators such, that is the output redirection with replacing file and output redirection with append file. The replace operator use the > (greater than), and append operator use >> (double greater than) symbols.

Redirect Output > Operator

First example, we will use the standard unix command like echo to redirect the output into a file.

In the example above, we can understand that the string "Hello" is written through the file named hello.txt. If we do the the same thing with a different string, the hello.txt file will be replaced with the new string. It can be shown by the example below.

As the example above, we don't intentionally delete the files but the old hello.txt file that contains "Hello" got replaced by a new string "Rendy". If we don't want to replace the string, then we should use the append output redirection operator >> .

Append Output >> Operator

In the second attempt, we will using the same file hello.txt and write another string to the this file but using the append >> operator.

In the example above, we can see that the >> operator did not change the file contents of the hello.txt. Instead it will append the "George" string below the existing file contents.

The standard output and standard input redirection operator can be used together with the main program we wrote before.

With the command above, you can see that the Hi rendy string is not printed in the terminal, instead it written in the main.log file. Hence the example error line "There is no error, this is just an example" that is printed in the standard error would still shown in the terminal output because we only redirect the standard output.

Standard Error (stderr)

Basically, redirecting the standard error would still the same intuition as the standard output. Remember that the standard error using 2 as the file descriptor number in this picture? We only need to redirect them using 2 prefix in each operator. For instance, to redirect the standard error we can use 2>, and to redirect append output of a standard error we can use 2>>. Let's try that using the main command.

The picture above shown that we already successfully redirect the standard error to the error.log file. While using the append operator, we will got the similar result as the picture below.

That is the basic usage of the standard input, standard output, and standard error operator in bash system.

Advanced Use Cases

After understanding its function and available operators in the standard streams, there is a bunch another use case combination of its that may help improving our productivity. Utilizing the main program we wrote before, we can redirect the standard output to a file, and the standard error to another file like the picture below.

In some cases, there is needed a bigger input, usually from a file. It is possible by leveraging the < operator as the example below.

There another use case that need to combine all of the output of standard error and standard output in a one file.

Usually we have a bunch of files in the same directory (usually logs). To querying some string, we can use the cat command to list all of the files that match the pattern, and find the relevant information using grep command.

That's all. Understanding the basic standard input and output streams of unix system is worth it. We can improve the productivity because we don't need to open the IDE that may heavy, or even impossible in the production server.