We will mainly focus on how does the magic happen inside the NextJs App ?

The requirements :​

What we want to build is a nodal GUI handler for docker-compose data handling.

From this product need we can extract somes technicals answers :

• We need a good data handling of docker-compose
• We need a tool to help us with the nodal GUI
• We need a state management sytem that allow us complex deep nested object mutations, in an efficient way.

Choosing a data library to handle docker-compose efficiently​

Docker-compose is a really complex data structre, as it's deeply nested and somes bottoms objects are linked to top ones. I looked for any docker-compose but didn't find any in TS, so i builded mine from the ground.

It's curently not open-source but we do have a form to ask acces to it

The library work with holding all the states inside a big Compose class. managing the state of just one Compose instance, it becomes straightforward to stay up-to-date with all job-related data.

React state management​

From the start we knew we have to manage the Compose state on client side as there is a lot of state mutation made on the playground page.

But, what to choose between React UseState, Zustand, Recoil, Redux... ?

We chose Zustand for our state management because of its seamless integration with Next.js and its ability to modify state by passing a function instead of a new object.

Why does this matter? In the traditional approach, using setData(newData) would recreate the entire Compose object on every state update, like this:

const [compose,setCompose] = useComposeState()

setCompose(new Compose(whatever you change))

This would result in a new instance of Compose every time, which could lead to inefficiencies or unnecessary re-renders.

Instead, Zustand allows us to pass a modifier function, enabling changes to be applied directly to the existing object without creating a new instance:

const [compose,setCompose] = useComposeState()

setCompose((oldCompose)=>oldCompose.whateverChange(anyParamsNeeded))

This approach allows targeted updates and keeps the instance intact. For more granular changes, you can directly modify properties within the state:

const [compose,setCompose] = useComposeState()

setCompose((oldCompose)=>{
    oldCompose.service1.name = "new name"
})

By using this method, we maintain the simplicity and efficiency of state updates without compromising the integrity of the Compose instance

But you maybe wonder, what does the zustand store look like ?

"use client";

import { create } from "zustand";
import { Compose } from "@entrecompetents/composecraftLib";
import { generateRandomName } from "@/lib/utils";

interface ComposeState {
    compose: Compose;
    setCompose: (updater: (currentCompose: Compose) => void) => void;
}

export const useComposeStore = create<ComposeState>((set) => ({
    compose: new Compose({ name: generateRandomName() }),
    setCompose: (updater: (currentCompose: Compose) => void) => {
        set((state) => {
            updater(state.compose);
            return { compose: state.compose };
        });
    },
}));

The nodal GUI library​

The options was : JointJS, react flow, mermaidJS (their was maybe other options but we didn't go further in the searchs)

Here is a small recap table of our choose :

We finally choosed React flow for the interactivity and the fact it's open source.

But keep in mind it's not an easy way to go, as react-flow require a lot of custom dev.

The playground scheme :​

Conclusion :​

Building a nodal GUI for handling Docker Compose data in a Next.js application is a multi-layered process that combines efficient state management, a robust GUI library, and custom-built tooling. Through this exploration, we have:

1. Crafted a TypeScript library for managing Docker Compose data that suits the complexity and nested nature of the structure.
2. Optimized client-side state management using Zustand, allowing for efficient updates without recreating entire objects or instances.
3. Selected React Flow as the nodal GUI library, leveraging its interactivity and customizability while tackling the steep learning curve through targeted development.

This integration, represented in the final scheme, showcases a streamlined workflow where deeply nested state changes, GUI interactions, and backend communication seamlessly come together. The chosen tools not only meet the functional requirements but also align with performance, usability, and scalability goals.

These efforts culminate in a playground page that delivers a cohesive, interactive user experience while adhering to technical best practices.

The recent release of Traefik v3 brings a lot of new features. This article does not aim to explain all the new features of Traefik v3, but to show you how to use it with Docker compose.

Prerequisites​

Before starting, you need to have Docker and Docker compose installed on your machine. If you don't have them installed, you can follow the official documentation to install them.

Create a Docker compose file​

Let's creat a simple docker-compose.yml file to deploy a web server and Traefik v3.

version: '3.8'
services:

  traefik:
    image: traefik:v3.0
    networks:
      - web
    ports:
        - target: 80
          published: 80
          protocol: tcp
          mode: host
        - target: 443
          published: 443
          protocol: tcp
          mode: host
    environment:
      - TZ=Europe/Paris
    command:
        - --providers.swarm.endpoint=unix:///var/run/docker.sock
        - --providers.docker.exposedbydefault=false
        - --providers.swarm.network=fdp_web
        - --accesslog
        - --entryPoints.websecure.address=:443
        - "--certificatesresolvers.myresolver.acme.tlschallenge=true"
        #- "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
        - "--certificatesresolvers.myresolver.acme.email=organisation@lesfousdupeloton.com"
        - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
        - "--entryPoints.web.forwardedHeaders.trustedIPs=10.0.0.0/24,10.0.2.0/24,192.168.100.0/24"
        - "--entryPoints.websecure.forwardedHeaders.trustedIPs=10.0.0.2/24,10.0.2.0/24,192.168.100.0/24"
        - --accesslog.filepath=/log/acces/access.log
        - "traefik.http.middlewares.exclude-ip-log.ipwhitelist.sourceRange=217.72.195.109"  # IP address to exclude for ping
    volumes:
        - ./certs:/letsencrypt
        - /var/run/docker.sock:/var/run/docker.sock:ro
        - ./log:/log
    deploy:
        placement:
          constraints:
            - node.role == manager
        restart_policy:
          condition: on-failure
          delay: 5s
          max_attempts: 3
          window: 120s
        labels:
          - "traefik.enable=false"

  frontend:
    image: registry.gitlab.com/entrecompetents/lesfousdupeltoton/frontend:1.0.8
    hostname: frontend.lfdp
    networks:
      - web
    deploy:
      mode: replicated
      replicas: 1
      labels:
        - "traefik.enable=true"
        - traefik.http.services.frontend.loadbalancer.server.port=8080
        - traefik.http.routers.frontend.rule=Host(`lesfousdupeloton.com`)
        - "traefik.http.routers.frontend.entrypoints=websecure"
        - "traefik.http.routers.frontend.tls.certresolver=myresolver"
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/"]  # Health check command using curl
      interval: 5s  # Health check interval
      timeout: 2s    # Timeout for the health check
      retries: 3     # Number of retries before marking the container as unhealthy
      start_period: 10s  # Delay before starting the health checks after the container is started

  
networks:
  web:

When working with an API, it is important to be able to debug it.

API debugging tools can be divided into two classes: those that allow editing of the sent HTTP requests and those that inspect the network to see what has been sent retrospectively.

In the first category, there are well-known tools such as Postman, Insomnia or Curl.

In the second category, Wireshark overcomes most available solutions. Fortunately, it is open-source and free of charge.

When should we use Wireshark ?​

Wireshark allows us to intercept network requests, making it useful when:

• We want to know the network requests of a tool without having access to its code.
• We want to verify the content of requests sent by a service.

Below, it is emphasized that Wireshark is non-intrusive in the sending or receiving of HTTP requests; it acts as an observer of the network.

How to use Wireshark ?​

To install Wireshark, you can follow the official installation page.

Once installed and opened, Wireshark displays the available networks on the computer along with their activity (see screenshot below).

After selecting an interface, the user can start capturing requests. It's important to note that HTTPS requests will be captured but appear encrypted and unreadable.

As shown in the screenshot below, the user can also add filters to display only relevant requests.

The request inspection interface is particularly useful. It allows a detailed look at the content of the HTTP request, helping to understand what the recipient has received.

Wireshark's "non-intrusive" capability makes it a valuable ally in debugging applications.

Numerous tools are available for network debugging, from debugging a REST API to analyzing traffic on a router or local network.

A comparison can be found on our Wireshark article.

Tshark is the CLI version of wireshark, and lets you capture with your favorite tool when you don't have access to a GUI, or when the CLI server simply doesn't have a graphical environment.

The advantage is that you can export the network capture and open it on your PC with Wireshark! We'll look at this at the end of the article.

How to use Tshark

Installing tshark​

You can follow the official documentation, on debian-based just tap apt install tshark.

You can also get tshark by installing Wireshark. And like wireshark, tshark uses Dumpcap as its capture engine.

Tshark has the advantage of being a lightweight, powerful tool that can be installed on many platforms.

On installation, you can choose whether to give access to network analysis to all users or only to sudoers.

How do I capture packages with Tshark?​

Useful basic commands​

To see the list of capturable interfaces: tshark -D. You also have the advantage of seeing virtual interfaces.

To capture: tshark or tshark -i <interface_name>.

You can make complex filters with tshark -f "${filter} like tcp port 80, in the same way as filters on wireshark.

Further information​

The official documentation is very comprehensive, and the tshark --help command is very useful, as is the man.

Export and open capture on Wireshark​

A huge advantage of tshark is its intercompatibility and capture format.

To save a capture in a file, use the -w option like this: tshark -w <filename>.pcap.

To read a pcap file, you have two options:

• open in GUI with Wireshark
• open with tshark by coloring with: tshark -r <filename>.pcap --color.

docker commit [container name] [image name]:[tag]

This command creates an image by taking a kind of "snapshot" of the targeted container. However, I wondered what the differences were compared to docker build (aside from the Dockerfile).

I took the time to study a very simple case and examine the differences it produces:

Differences between Docker Commit and Docker Build:​

Both docker commit and docker build commands are used to create Docker images. However, docker commit creates an image from an existing container, while docker build creates an image from a Dockerfile.

Comparison Test Protocol for Docker Commit and Docker Build:​

1. Build an nginx:1.24.0 image with the addition of the iputils-ping package.

2. Make this addition in a single command.

The Dockerfile :

FROM nginx:1.24.0

RUN apt update && apt install iputils-ping -y

In this case, there is a difference of 1254 Bytes, approximately 0.0001 MB.

The exact sizes of the images are:

• 155 182 582 bytes for docker build
• 155 183 836 bytes for docker commit

Attribute Differences​

While the image sizes have changed very little, the attributes have been radically modified:

The image created with docker build has lost the following attributes:

• Cmd
• EntryPoint
• Env
• Labels
• Exposed ports
• Stop signal
• Docker version
• Parent

Conclusion on the Differences between Docker Commit and Docker Build:​

There is no better solution; they simply have different purposes:

Docker commit should mainly be used to customize a Docker image to simplify its later use.

Example:​

Adding a package, quickly modifying an image for testing purposes...

Docker build should be used when creating a new use case, especially when wanting to keep a scripted version of the image in the form of a Dockerfile. This is much more sustainable than an image stored in a registry.

The sources of my experience and the entire process are available on this repository: https://github.com/LucasSovre/sourcesCommitVsBuild/tree/main

This shouldn't affect the studied mechanics, but the tests were performed with an ARM (M1) processor.

Sources​

• Official Docker commit documentation
• Official Docker build documentation
• Official Dockerfile documentation
• Source Code

What is the point of theses tools ?​

Discover two tools designed for collecting email lists and sending bulk emails. Explore their features, benefits, and find the right solution for your mass email needs.

Where to find them ?​

official Listmonk website

official Mailchimp website

MailChimp an "all-in-one" tool​

MailChimp is an "all-in-one" tool. It includes mail servers, automation, data collection, registration forms... Almost nothing is missing.

Listmonk, a tool focused on emails.​

Listmonk doesn't burden itself with all the features of automation or a WYSIWYG editor (which would still be nice). But what it does, it does well, and most importantly, it uses external mail servers.

On the financial side:​

In terms of pricing, Listmonk is unbeatable since it is free. Even when factoring in the costs of servers and mail server providers, Mailchimp's prices remain much higher.

With a user base of 10,000 users and an average of 5 emails per person per month:

MailChimp, as of December 3, 2023, costs €125/month.

Assuming you use Amazon SES for emails (€0.09/1k emails), MailChimp would cost you €4.5 per month.

Your infrastructure costs (solely dedicated to Listmonk) would need to exceed €120 per month for MailChimp to be more cost-effective.

It's evident that in terms of savings, Listmonk can be highly beneficial, especially considering that with Listmonk, your subscriber lists are unlimited.

How to Address Listmonk's Feature Gaps:​

As mentioned earlier, Listmonk lacks user journey automation or automatic email sending capabilities. However, with a self-hosted n8n, you can easily create more advanced automations without the need for coding.

Below is an example of an n8n-listmonk automation workflow. This workflow is triggered by a webhook call, retrieves user information from the database, creates the user in the listmonk database if necessary, and then sends them an email.

In the following example, this workflow allows scheduling the sending of a transactional email to a user in n days|hours|minutes.

While Listmonk may lack the WYSIWYG editor found in MailChimp, a few email templates are likely to suffice. You can easily acquire them for a few euros, compensating for this specific feature gap.

You can find the n8n listmonk node in this npm package

How to install Listmonk ?​

The Listmonk installation page is comprehensive, but here's a quick summary:

If you don't already have a PostgreSQL database, you can use the following command:

mkdir listmonk && cd listmonk
sh -c "$(curl -fsSL https://raw.githubusercontent.com/knadh/listmonk/master/install-prod.sh)"

If you already have a database, just launch a container with the associated configuration.

How to Integrate Listmonk into Your Infrastructure:​

Here's a diagram that consolidates all the previous elements to provide an example of how to integrate Listmonk into your tools.

Sources​

• https://mailchimp.com
• https://listmonk.app
• https://listmonk.app/docs/
• https://aws.amazon.com/fr/ses/
• https://n8n.io

SurrealDB is a powerfull database, which 1.0 has been released in 2023. It's a new gen database, which can be schemafull or schemaless. In this article, we will focus on the schemafull part of SurrealDB.

Even if the official SurrealDB documentation is quite complete about SurrealQL language querrying, it lacks a bit of information about the types available in a schemafull table. This article aims to fill this gap.

Reminder about creating a table and field​

To create a table in SurrealDB, you need to have previously defined a database and namespace. Then you can create a table with the following command:

DEFINE TABLE product SCHEMAFULL;

Then to define a field in a table, you can use the following command:

DEFINE FIELD my_field ON TABLE product TYPE data_type;

Types available in SurrealDB schemafull table

basic types​

Any​

DEFINE FIELD my_field ON TABLE product TYPE any;

Booleans​

DEFINE FIELD active ON TABLE product TYPE bool;

Numbers​

Any number​

DEFINE FIELD price ON TABLE product TYPE number;

Integer​

DEFINE FIELD price ON TABLE product TYPE int;

Float​

DEFINE FIELD price ON TABLE product TYPE float;

Decimal​

DEFINE FIELD price ON TABLE product TYPE decimal;

Null​

DEFINE FIELD price ON TABLE product TYPE null;

String​

DEFINE FIELD name ON TABLE product TYPE string;

Bytes​

DEFINE FIELD icon ON TABLE product TYPE bytes;

Uuid​

DEFINE FIELD ext_id ON TABLE product TYPE uuid;

Times related types​

Datetime​

DEFINE FIELD created_at ON TABLE product TYPE datetime;

Duration​

DEFINE FIELD duration ON TABLE product TYPE duration;

Point​

DEFINE FIELD location ON TABLE product TYPE point;

Record​

DEFINE FIELD created_by ON TABLE product TYPE record;

Composed types​

Composed types are formed by combining basic types with composed_type<type>.

Optional type​

By default, all fields are required. If you want to make a field optional, you can use the option type.

DEFINE FIELD description ON TABLE product TYPE option<string>;

Multiple types​

If you want to allow multiple types for a field, you can use the either type.

DEFINE FIELD additional_propertie ON TABLE product TYPE string|number;

Array​

DEFINE FIELD tags ON TABLE product TYPE array<string>;

Set​

DEFINE FIELD tags ON TABLE product TYPE set<string>;

Geometry​

DEFINE FIELD location ON TABLE product TYPE geometry<line>;

In this case the only types allowed are the following defined in the SurrealDB documentation Theses types respects the GeoJSON format (defined in rfc7946).

SurrealDB full course​

If you want to learn more about SurrealDB, you can subscribe to our newsletter to be informed about the course we are preparing, and get a discount when it will be released.

So, my needs where :

• A user system
• Somes analytics
• A blog sytem
• A documentation system
• A database that allow me to quickly try new things
• The stack has to be self-hostable easily

Choose a database​

The database, is a though choice. First should i go SQL or No-SQL ?

In my opinion, we should choose a database that fit our data shape, and not shape our data to fit in the database. And if you know a little of the docker-composes, that's a very complex data shape to store, as is it quite inconsistent (I mean, even if this is based on yaml, there is so much differents possibilities). To store it in an sql database would be a nightmare, the only "easy way" would to just considere it as a raw text and store it in a text cell (or byte but anyway).

Yes i could use the postgresql JSON cell to create a way to store it convientently, but remember our need : keep it simple.

So i choosed No-SQL for the ability to store non structured data.

But what No-SQL database ?​

Choosing a No-SQL database is just half the way to choose a database. There is still so much questions to pick one. But always remember this when you are a small team or solo dev : keep it simple.

So i will restrict my questions to :

• Is there an easy to use TypeScript Driver ?
• Is the database secure ?
• Can i self-host it ?
• Is there a big community ?

I thought about surealDB and mongoDB. They both have a TS sdk, they dont' have new CVE each morning, and they are fully self-hostable.

SurrealDB​

SurrealDB could be a really good way to go, has it can be used as an sql db for my user part and noSQL for my data-processing and job part.

But the big drawback is the lack of maturity. Even if the devs are quite fast on issue peoples submits, their is not so much stack-overflow or any knowledge place about it.

The final choice : MongoDB​

So, at the end i choosed mongoDB as it would allow me to quickly change the dataModel of my entities, and there is a much stronger documentation knowledge about it.

At this point i knew i will use nextJS + mongoDB, but what about the bloging system, the documentation, and the analytics ?

The self hosting part​

Before going further i started my first beta with very few user. So i needed to self-host the stack. My two biggest concern where : I want it cheap, and i want it easy.

I choosed hetzner.com has they are really cheap and reliable. 4.51 € /month for 2 Cpu, 4Gb Ram 20Tb bandwith and 40gb SSD. To manage my VPS i choosed coolify, has it's fully open-source and allow me to easily deploy new services.

The analytics, blog and documentation :​

The analytics choice was really straigth forward for me as i use Umami analytics from 2020. It's fully self-hosted, open source and GDPR complient.

My blog need where a little bit weird as i needed something headless but with an admin UI and self-hostable. I could go for Strapi, but i ended with Directus for no particular reason both could perfectly fit in my use case.

About the documentation, my choice has been pushed by : cheap and fast. So i could go with any Mardkown based documentation. Here is somes good choices bellow :

• Docusaurus
• Hugo
• mkDocs
• markdoc

I ended with Docusaurus because it's based on react, and in the future, if there is any need i could easily add custom components.

Is'nt there any SEO issues using a different techno for documentation and bloging ?​

To avoid SEO issue that are associated with multi-domain website such as could be composecraft.com, docs.composecraft.com, blog.composecraft.com.

I Opted for a different way to go :

First my blog system is more a library system. Directus only help me to manage the entries, my data contains a markdown field and directly feed my nextJS, so for the blog part (or library as you want) everything is under composecraft.com/library and nextJS handle it with the next-remote-mdx library.

But although i could used the same patern for my documentation, i choosed docusaurus. Why ? Because my documentation will quickly evolve, and i don't want to mess with any nextjs question when writting documentation. I just want to writte mardown file and do not care about mobile responsive, about performance or anything. Using Docusaurus automaticly enforce my documentation to be structured, easy to edit and easy to use by my users.

So i just setted my reverse-proxy to send any request under /doc to the Docusaurus docker that running managed by my coolify.

The reverse proxy :​

In my opinion, especialy for choosing a reverse-proxy for low/medium load website go with the one you love. There is no "compute" better choice between ngninx, trafeik, apache or caddy as you stay with a simple server and no load balancing or complex routing.

The backups ?​

How does i ensure my self-hosted database won't desapear when my server will burn because of my to many users ?

Until now my server is far away from burning from my to many user, but anyway i have integrated a backup mechnism. Which one ? Every day at 3 am a complete database backup is schedule and stored in a distant S3 on a cloud provider (a different from my VPS one to prevent from them putting the S3 bucket in the same datacenter as my production server).

Last concern : the CDN​

As a small product that is used worldwide i faced a problem : my US users complains about latency, and indeed the server is located in Europe.

Easy answer : i could have a US instance !

But ... remember it's a small projet and i want it cheap and easy. So i choosed to call cloudflare to help me with their free CDN. Yes it's not a fully self hosted privacy powered stack. But as a small product team it's the best cost/performance chose i cloud make.

The final scheme :​

Somes thoughts​

Could i builded it simpler ?

Yes i definitly clould : If my final client was someone non tech at all i would have worked more on the docs and reused directly directus to manage my docs in the same time as my libraries. And if the client do not want to infere at all with linux server : using mongoDb atlas and cloud solution for analytics, and something like vercel, netifly or fly.io to deploy my stateless nextjs app.

But as I am a linux lover and i am really not afraid of getting my hand on code, i thinks this is the most cost efficient and easy maintainable stack that fit me.

Wana go deeper ?​

I will soon create article about the CI/CD pipeline, and another one about the internal nextjs logic. keep updated about new blogs posts subscribing to the newsletter bellow :