Handling Multiple Connections: Multithreading (C Networking Part 4)

Introduction

In part 1, we learned the basics of C networking, including how to create and listen on a socket. In part 2, we learned how to send a response back to the client. Finally, in part 3, we learned how to handle multiple connections with looping, forking and select.

Now let’s learn how we can use multithreading to process multiple connections with better resource usage.

C Multithreading Basics

We want to handle multiple tasks at the same time without the overhead of creating a new process. That’s where multithreading comes in. The C multithreading API is in the pthread.h import:

...
#include <pthread.h>
...

To spawn a thread, we tell it what function to execute. To do so we pass in a pointer to a function. The function should look like this:

void *thread_function(void* ptr) {
    return NULL;
}

It both takes and returns a void*. This means that the argument and return types aren’t specified in the function signature itself. It’s the responsibility of the caller to correctly handle typing. Let’s update our example to pass in an int. In this case we’ll pass in a thread ID.

void *print_thread_id(void *ptr) {
    int *threadIdPtr = (int *) ptr;
    printf("Hello from thread %d\n", *threadIdPtr);
    return NULL;
}

The thread function needs to assume that the pointer it’s given points to valid input data, in this case an int representing thread ID. The first thing the function does is cast the input memory location to the expected data type. In this example, we only print out the thread ID.

Now that we’ve built a function for threads to execute, we’ll spawn a thread in the main function.

int main(void) {
    pthread_t thread;
    int threadId = 1;
    pthread_create(&thread, NULL, print_thread_id, &threadId);
    pthread_join(thread, NULL);
    puts("Success");
    return 0;
}

This is all it takes to spawn a thread. Declare the thread variable itself as a pthread_t, then call pthread_create with the a pointer to the thread, the name of the thread function, and a pointer to any arguments to pass in. The second argument chooses thread attributes, but we pass in NULL to use defaults. (Real production code should have error checking, but for simplicity we leave it out here).

Next we join the thread. pthread_join takes the thread as its first argument. The second argument takes a memory location to store a return value from the thread. In this example, the thread function does not return anything, so we just pass in NULL.

Why does pthread_create need a pointer to the thread variable, but pthread_join is fine with just the pthread_t itself? Simplify put, pthread_create needs to modify the thread variable with information about the created thread. On the other hand, pthread_join’s only job is to block until the thread finishes and acccess the return value. Since it doesn’t modify the thread variable, it doesn’t need a pointer to it.

A Complete Example

Putting everything together, we have the following:

#include <stdio.h>
#include <pthread.h>

void *print_thread_id(void *ptr) {
    int *threadIdPtr = (int *) ptr;
    printf("Hello from thread %d\n", *threadIdPtr);
    return NULL;
}

int main(void) {
    pthread_t thread;
    int threadId = 1;
    pthread_create(&thread, NULL, print_thread_id, &threadId);
    pthread_join(thread, NULL);
    puts("Success");
    return 0;
}

Output:

cbrown@typhoon ~/temp % gcc main.c
cbrown@typhoon ~/temp % ./a.out
Hello from thread 1
Success

Depending on your system, you might need to link in the pthread library using the -pthread option:

cbrown@typhoon ~/temp % gcc -pthread main.c && ./a.out
Hello from thread 1
Success
cbrown@typhoon ~/temp % clang -pthread main.c && ./a.out
Hello from thread 1
Success

Multiple Threads

To spawn and join multiple threads, we can modify our thread variable and input arg to be arrays.

int main(void) {
    const int NUM_THREADS = 8;
    pthread_t threads[NUM_THREADS];
    int threadIds[NUM_THREADS];
    for (int i = 0; i < NUM_THREADS; i++) threadIds[i] = i;
    for (int i = 0; i < NUM_THREADS; i++)
        pthread_create(&threads[i], NULL, print_thread_id, &threadIds[i]);
    for (int i = 0; i < NUM_THREADS; i++) pthread_join(threads[i], NULL);
    puts("Success");
    return 0;
}

Result:

cbrown@typhoon ~/temp % gcc -pthread main.c && ./a.out
Hello from thread 0
Hello from thread 2
Hello from thread 1
Hello from thread 3
Hello from thread 5
Hello from thread 6
Hello from thread 7
Hello from thread 4
Success

Notice how the threads are not executed in any particular order. They all run at the same time, and the ultimate order each thread prints will change each time, and is up to the operating system.

Handle Client in a Thread

Let’s build on top of the loop solution from part 3. The goal is to handle connections simultaneously in different threads, instead of looping in one single thread and handling them sequentially. Here’s the client handler we made:

void handleClient(int clientfd) {
  // Wait to receive data
  char receiveBuffer[1024];
  ssize_t bytesReceived =
      recv(clientfd, receiveBuffer, sizeof(receiveBuffer) - 1, 0);
  receiveBuffer[bytesReceived] = '\0';

  // Send HTTP response
  const char *response =
      "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n"
      "<html><h1>HTML Response</h1><p>The server works!</p></html>";
  send(clientfd, response, strlen(response), 0);
  close(clientfd);
}

Now let’s turn this into a function we can use with pthread_create. The argument and return types need to be void*. Since our handler doesn’t return anything, changing the return type is just a matter of adding a star to the function signature, and then return NULL; to the function end.

Updating the argument is trickier but we can still follow the example from earlier. Instead of directly passing in the client file descriptor (an int), we pass in a void* that then gets immediately converted.

void *handleClient(void *arg) {
  // Wait to receive data
  int *clientfdptr = (int *)arg;
  int clientfd = *clientfdptr;
  char receiveBuffer[1024];
  ssize_t bytesReceived =
      recv(clientfd, receiveBuffer, sizeof(receiveBuffer) - 1, 0);
  receiveBuffer[bytesReceived] = '\0';

  // Send HTTP response
  const char *response =
      "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n"
      "<html><h1>HTML Response</h1><p>The server works!</p></html>";
  send(clientfd, response, strlen(response), 0);
  close(clientfd);
  return NULL;
}

Now let’s consider the main loop:

while (true) {
    int clientfd =
        accept(socketfd, (struct sockaddr *)&clientAddress, &clientAddressSize);
    handleClient(clientfd);
}

We update this to dispatch a thread to call the client handler, instead of directly doing so. We will declare a pthread_t and then call pthread_create with it. Next we use pthread_join to wait for the thread to complete. Here’s the full, updated example:

#include <netinet/in.h>
#include <pthread.h>
#include <stdbool.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <unistd.h>

#define PORT 80

void *handleClient(void *arg) {
  // Wait to receive data
  int *clientfdptr = (int *)arg;
  int clientfd = *clientfdptr;
  char receiveBuffer[1024];
  ssize_t bytesReceived =
      recv(clientfd, receiveBuffer, sizeof(receiveBuffer) - 1, 0);
  receiveBuffer[bytesReceived] = '\0';

  // Send HTTP response
  const char *response =
      "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n"
      "<html><h1>HTML Response</h1><p>The server works!</p></html>";
  send(clientfd, response, strlen(response), 0);
  close(clientfd);
  return NULL;
}

int main() {
  // Create socket
  int socketfd = socket(PF_INET, SOCK_STREAM, 0);

  int opt = 1;
  setsockopt(socketfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
  setsockopt(socketfd, SOL_SOCKET, SO_REUSEPORT, &opt, sizeof(opt));

  // Assign address to socket
  struct sockaddr_in socketAddress = {
      .sin_family = AF_INET,
      .sin_port = htons(PORT),
      .sin_addr = INADDR_ANY,
  };
  bind(socketfd, (const struct sockaddr *)&socketAddress,
       sizeof(socketAddress));

  // Listen for and start queueing up connections on socket
  listen(socketfd, 64);

  while (true) {
    struct sockaddr_in clientAddress;
    socklen_t clientAddressSize = sizeof(clientAddress);
    int clientfd;
    clientfd =
        accept(socketfd, (struct sockaddr *)&clientAddress, &clientAddressSize);
    pthread_t thread;
    pthread_create(&thread, NULL, handleClient, (void *)&clientfd);
    pthread_join(thread, NULL);
  }

  close(socketfd);
  return 0;
}

The thread is created and then immediately joined, so this isn’t yet true multithreading. But we can use this to verify that our new client handler works. Compile and run the server, then visit localhost in a browser. You should see a message The server works!.

True Multithreaded Networking

Next, our server should actually handle client connections simultaneously.

So far, all we’ve done is call pthread_detach on the threads we create. But we also have the option of using pthread_detach. Joining causes the main thread to block until the child thread terminates. Detaching separates the child thread from the parent, useful for “fire and forget” type tasks.

It’s important to detach any thread you won’t join. Detaching indicates to the thread to automatically clean up its resources when it finishes. Not doing this leads to resource leaks.

Finally, we need to change int clientfd so that we dynamically allocate its memory, instead of declaring it on the stack. The loop context we fire off the thread ends right after, which means any variables allocated on the stack fall out of scope. Dynamic allocation lets us manage the memory lifecycle by hand. The thread function body accesses the memory, saves it on its own stack, and then calls free() to deallocate the memory.

These changes are shown below:

diff --git a/server.c b/server.c
index 3fba071..587a195 100644
--- a/server.c
+++ b/server.c
@@ -12,6 +12,7 @@ void *handleClient(void *arg) {
   // Wait to receive data
   int *clientfdptr = (int *)arg;
   int clientfd = *clientfdptr;
+  free(arg);
   char receiveBuffer[1024];
   ssize_t bytesReceived =
       recv(clientfd, receiveBuffer, sizeof(receiveBuffer) - 1, 0);
@@ -49,12 +50,12 @@ int main() {
   while (true) {
     struct sockaddr_in clientAddress;
     socklen_t clientAddressSize = sizeof(clientAddress);
-    int clientfd;
-    clientfd =
+    int *clientfd = malloc(sizeof(int));  // freed in handleClient
+    *clientfd =
         accept(socketfd, (struct sockaddr *)&clientAddress, &clientAddressSize);
     pthread_t thread;
-    pthread_create(&thread, NULL, handleClient, (void *)&clientfd);
-    pthread_join(thread, NULL);
+    pthread_create(&thread, NULL, handleClient, clientfd);
+    pthread_detach(thread);
   }

   close(socketfd);

Connect again to the server at localhost in a browser. Again, you should see a familiar The server works! message.

Conclusion

We now know how to use multithreading to handle multiple client connections at the same time. Armed with information from this entire series, we can now build a basic server in C from scratch.

From here, the sky is the limit. Here are some ideas to iterate on what we have so far.

• Error handling (skipped in this series for simplicity)
• Dynamic response to the client HTTP requests (right now every request gets the same static response)
• Support for application level protocols other than HTTP (for example, FTP)
• Logging (potentially in its own thread)
• Unit and integration testing

Real production-grade servers like Apache boast nearly two million lines of mostly C code, written over the course of 20 years.

A big thank you to everyone who followed the series. I hope it provides the kickstart into C network programming that I didn’t have.