If you own an Intel NUC and are struggling to reach deep power-saving states like C8 or C9, your issue might be related to PCIe Active State Power Management (ASPM). I recently ran into this problem with my Intel NUC 7th Gen, where the system was stuck at C3 states despite my best efforts to tweak Linux power settings. Through trial and error, I discovered that an outdated BIOS was blocking ASPM on my NVMe SSD, preventing the system from reaching deeper C-states. A BIOS update ultimately resolved the issue! If you’re facing a similar problem, this guide will walk you through the solution.

The Problem: Why Was My Intel NUC Stuck at C3?

I started with PowerTOP, an essential diagnostic tool for Linux power management. To install it on most Linux distributions, use one of these commands:

sudo apt-get install powertop

After installation, you can run PowerTOP with privileges:

sudo powertop

Checking Power States with PowerTOP

After running PowerTOP and going to the ‘Idle Stats’ tab, I noticed that my system was only reaching C3 states, instead of the deeper C8-C10 states that Intel CPUs support. So clearly, something was blocking my system from reaching deeper states. The first thing that I tried was to run auto-tune. PowerTOP includes this feature that applies various power-saving settings:

sudo powertop --auto-tune

This command applies recommended settings for all power management features. However, in my case, even after running auto-tune, the system remained stuck at C3 states, indicating a deeper issue.

Investigating PCIe ASPM with lspci

Since NVMe SSDs and PCIe devices can block deep C-states, I checked their ASPM support, and I easily found one device not being ASPM enabled.

sudo lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'

I noticed multiple things:

• My Samsung SM961 NVMe SSD (3a:00.0) supports ASPM L1, but it was disabled.
• The PCIe Root Port (00:1d.0) did not support ASPM.
• Another PCIe Root Port (00:1c.0) did support ASPM, but nothing was connected to it.

This suggested that my NVMe SSD was stuck on a PCIe port that does not support ASPM, blocking deep sleep states.

Forcing ASPM Enable

Before diving into BIOS updates, I tried a helpful script from GitHub called AutoASPM, which attempts to enable ASPM for all devices, hoping this would enable ASPM.

# Clone the repository
git clone https://github.com/notthebee/AutoASPM
cd AutoASPM

# Run the script (requires sudo)
sudo python3 autoaspm.py

This script tries to enable ASPM across all PCI devices by writing to the appropriate kernel interface files. However, in my case, since the BIOS had disabled ASPM at a hardware level, the script couldn’t overcome this limitation. After running again:

sudo lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'

I still noticed that my ASPM was disabled for the NVMe.

If you’re facing similar issues, try this script first—it may solve your problem without requiring a BIOS update.

Checking BIOS Configuration for ASPM

After trying software solutions without success, I decided to check if my BIOS was configured correctly for ASPM. I confirmed that PCIe ASPM Support was enabled in the BIOS.

Despite this correct configuration in the BIOS, my system still could not reach deeper C-states. This led me to conclude that either the port my NVMe SSD was using did not allow lower power states. However, this conclusion did not make sense to me—why would an Intel NUC engineer design it this way? So the next day, I decided to give it one last try and check if this was an issue that was solved by a BIOS update.

The Fix: Updating My BIOS

As a last resort, I checked Intel’s website for a newer BIOS version (which is now moved to ASUS) for my NUC model (NUC7i5BNK). I found a BIOS update here on the ASUS website.

I followed the instructions:

• Downloaded the ZIP file.
• Unzipped and added the .bio file to a USB flash drive.
• Booted my NUC with the USB flash drive while spamming F7.
• Selected the .bio file from the menu and let the system update itself.

Once the BIOS update was complete, I quickly verified that my BIOS settings were still untouched— PCIe ASPM Support was still enabled. After rebooting, I ran the same power state checks:

Now I was able to run AutoASPM and verify that it was working with:

sudo lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'

And after running:

sudo powertop

I noticed that the output now showed C8 and even C9 states, with 20% less power consumption! Okay, I’ll be honest with you— 3 watts saved. But hey, free energy savings!

Bonus: Enabling ASPM for Other Devices

Through this process, I became faster in checking the ASPM states of all devices, which really helped me in another project. For example, there I immediately identified one device that was not ASPM-enabled. Back then enabled it manually with:

echo 1 | sudo tee /sys/bus/pci/drivers/r8169/0000:01:00.0/link/l1_aspm

However, by going all the way on this NUC, I now discovered this useful Python script. I think for each system, you could run AutoASPM, along with powertop --auto-tune, and verify with the regex lspci command to get a very high success rate of reaching deep C-states.

Step 1: Install Homebrew (if not already installed)

First, if you don’t have Homebrew installed, open your terminal and enter the following command:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

This will install Homebrew, a package manager for macOS, making it easy to install LaTeX and other software.

Step 2: Install LaTeX via Homebrew

Once Homebrew is installed, you can install LaTeX by running the following command in your terminal:

brew install --cask mactex-no-gui

This command installs the MacTeX-no-GUI version, which includes the full LaTeX environment without any graphical applications (such as TeXShop). The download and installation can take a while to start, so be patient—it may seem like nothing is happening initially, but it will eventually begin downloading.

After installation, update LaTeX packages to ensure you have the latest versions. You will need to restart your terminal before running the following commands:

sudo tlmgr update --self
sudo tlmgr update --all

This ensures your LaTeX environment is fully up to date. If you encounter any issues during the installation or update process, check the terminal output for error messages and try to resolve them accordingly. For me, it helped to run eval "$(/usr/libexec/path_helper)" and restart my terminal.

Step 3: Install VSCode

If you haven’t installed Visual Studio Code yet, you can download and install it from the official VSCode website.

Once installed, open VSCode and prepare to add the necessary extensions for LaTeX.

Step 4: Install the LaTeX Workshop Extension

VSCode doesn’t natively support LaTeX, so we need to install an extension called LaTeX Workshop. This extension provides LaTeX syntax highlighting, compilation, previews, and other useful features.

To install it:

1. Open VSCode.
2. Go to the Extensions view by clicking on the Extensions icon on the sidebar or pressing Cmd + Shift + X.
3. Search for LaTeX Workshop.
4. Click Install.

Once installed, LaTeX Workshop will automatically manage compiling your LaTeX files and displaying previews.

Configuring LaTeX Workshop:

You can customize LaTeX Workshop by going to the settings (open Cmd + ,) and searching for LaTeX Workshop. You’ll find options for how you want your documents to compile, preview behavior, and more. For now, you can leave the default settings as they are.

Step 5: Enable Word Wrap in VSCode

LaTeX files can often contain long lines of text, and horizontal scrolling is inconvenient. To make your experience smoother, enable word wrap in VSCode so that long lines break automatically within the window.

To enable word wrap globally:

1. Open VSCode settings (Cmd + ,).
2. Search for word wrap.
3. Set Editor: Word Wrap to on.

Alternatively, you can set word wrap for individual sessions by using the keyboard shortcut Alt + Z.

Another way to enable word wrap is by using the Command Palette:

1. Open the Command Palette by pressing Cmd + Shift + P.
2. Type Word Wrap and select View: Toggle Word Wrap.

This ensures that lines automatically break without requiring horizontal scrolling.

Step 6: Syncing VSCode with Overleaf for Collaboration and Reviews

While VSCode offers a great environment for writing and compiling LaTeX documents, Overleaf is fantastic for collaboration, reviewing, and sharing with others. You can leverage both platforms by syncing your project between Overleaf and VSCode.

Setting up Overleaf for Git Synchronization

1. Create or Open a Project on Overleaf: Start by either creating a new project on Overleaf or selecting an existing project that you want to sync with.

2. Enable Git on Overleaf: Overleaf allows you to connect your project to a private Git repository. To do this:

   • Open your Overleaf project.
   • Go to the Menu (top left).
   • Scroll down to GitHub.
   • Follow the instructions to allow Overleaf to create a private Git repository for your project.

3. Clone the Repository in VSCode: Once Overleaf creates the repository, you can clone it to your local machine:

   git clone <your-overleaf-git-url>
   

   After cloning, you can open the project in VSCode.

4. Add a .gitignore file: Add a .gitignore file to your project to exclude unnecessary LaTeX-generated files (like .aux, .log, etc.). A nice .gitignore template for LaTeX projects can be found here.

5. Working in VSCode: You can now work on your LaTeX document in VSCode, take advantage of the full LaTeX Workshop experience, and perform heavy-duty tasks like editing large files or compiling complex documents.

6. Syncing with Overleaf: If you want to review or share a version, simply push your changes back to the Overleaf repository:

   git push origin main
   

   Overleaf will not automatically update with the latest version, for this you will have to manually pull the changes from GitHub in Overleaf.

   To do this:

   • Open your Overleaf project.
   • Go to the Menu (top left).
   • Scroll down to GitHub.
   • Press the button to Pull changes from GitHub.

   Make sure to resolve any conflicts that may arise during the pull process. The best way is to make sure everything is pushed to GitHub before pulling in Overleaf.

7. Collaborate and Review on Overleaf: Once your collaborators or reviewers have made changes on Overleaf, you can pull those changes back into your local repository:

   git pull origin main
   

   This way, you can synchronize changes between Overleaf and VSCode seamlessly, using Overleaf for easy collaboration and VSCode for more ease of work.

Step 7: Compile and Preview

Here is a quick preview of how it looks when you have set everything up. You can see the LaTeX code on the left and the preview on the right. By default, auto-compilation is enabled, which means the preview will update automatically when you save the file (or have autosave enabled). I recommend enabling autosave on all your projects. In this case, I switched my theme to a light theme because the dark-themed editor created too much contrast compared to the preview.

With LaTeX installed via Homebrew and configured in VSCode using the LaTeX Workshop extension, you now have a powerful setup for writing LaTeX documents. Additionally, by syncing your project with Overleaf, you can easily collaborate with others, review changes, and keep everything in sync using Git. The installation and setup may take some time, especially when using the brew command, but once it’s complete, you’ll have a smooth and customizable LaTeX environment.

Our Guidepost: A Django Application

To better understand the concept, consider this example docker-compose.yml for a Django application, equipped with PostgreSQL as its database and Redis:

`version: '3.8'

services:
  web:
    image: django:latest
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - app-data:/app
    ports:
      - "8000:8000"
    depends_on:
      - db
      - cache

  db:
    image: postgres:latest
    environment:
      POSTGRES_DB: sampledb
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    volumes:
      - db-data:/var/lib/postgresql/data

  cache:
    image: redis
    command: redis-server --save 20 1 --loglevel warning

volumes:
  app-data:
  db-data:

This configuration offers a Django web service (web) that leans on PostgreSQL (db) for its data storage and Redis (cache) for caching. It illustrates a typical setup that many web applications could use.

Why SSH

A virtual machine, named DevVM for this story, could be running in a cloud service, a secluded private data center, or even on a local grid. Whether you’re working on an on-premise SecureMachine or a cloud-based DevVM, SSH is the key. It provides a secure bridge to both, ensuring encrypted access to these remote resources.

Containers and Docker Compose

Containers have become instrumental in the modern software ecosystem due to their promise of portability. With many services now leaning towards containerized deployments, technologies like Docker, a popular choice among developers, ensure that applications are consistently executed, regardless of where they’re run. To further enhance this, tools like Docker Compose step in, allowing us to weave multiple containers into an interdependent stack. This makes booting up an entire software stack, with all its intricacies and dependencies, as straightforward as executing a single command.

Setting Up with VSCode

Now with the introduction out of the way, lets start explaining how to setup all this. I assume you have the following:

Prerequisites

• SSH key for your DevVM or SecureMachine
• Docker installed on your local machine
• Visual Studio Code (VSCode) with Docker extension installed on your local machine

Step 1: Clone the Project

We clone our project repository to our local machine:

git clone git@github.com:YourUsername/YourRepo.git

Step 2: Connect to DevVM using Docker and SSH

First, establish a Docker context for the VM:

docker context create DevVM --docker "host=ssh://user@DevVM:22"

In VSCode:

1. Open the Command Palette (Ctrl + Shift + P).
2. Type and select Docker Contexts: Use.
3. Choose the DevVM context.

Step 3: Configure the Development Environment

In your project root, create a .devcontainer directory. Inside, add a devcontainer.json:

{
   "name": "ProjectName",
   "dockerComposeFile": [
       "../docker-compose.yaml"
   ],
   "service": "web",
   "workspaceFolder": "/app/", 
   "overrideCommand": true
}

In this case I copied my Django project inside the folder /app when building my docker and therefore, I target this folder as the startup workspaceFolder. Note the overrideCommand: true. I added this to override the default command that is specified in the docker-compose file, to avoid the container from starting up the Django server. By doing this it will fall back to the default command specified in the Dockerfile, which is CMD ["/bin/bash"]. This will allow us to start the Django server manually for debugging purposes.

To ensure data persistence and easy access, introduce a volume, app-data, to your docker service. In the context of our setup, this volume I attached to the web service, making sure that when your docker fails you won’t lose your uncommited changes.

Step 4: Engage in Development

In VSCode:

1. Open the Command Palette.
2. Search and select “Dev Containers: Reopen in Container.” OR
3. Click on the green bottom left >< button all the way in the corner.
4. Select “Reopen in Container”

VSCode will now set everything up, and you’re all set to develop!

For PyCharm Aficionados

I have to admit, I love PyCharm and I should definitely mention that it also supports connecting to Docker via SSH, allowing you to work seamlessly with your project using Docker Compose. However, when I tried this setup I had trouble setting up the Docker Compose interpreter via a remote ssh Docker service. Maybe this is a bug in the current 2023.2 or I’m doing something wrong.

For the solo developer

You could also install VSCode Studio Server and setup a tunnel so you connect with your local VSC to it. I think it will make the setup a bit more smoother and less complex, however I choose for this solution since I had to share my machine. With this solution multiple developers can run a docker-stack on the same machine.

Wrapping Up

Remote development doesn’t have to be complex. With VSCode, Docker, and SSH, coupled with the power of VMs, developers can enjoy a flexible, secure, and consistent coding environment.

• WholeSlideData: Amazing useful package made by Mart van Rijthoven for handling vast pathology images and annotations with multiple backends.

• Shapely and Rasterio: For converting image masks to polygonal representations.

• GeoPandas: For spatial operations, making the overlap calculations efficient.

Loading the Mask

First lets load a mask that contains either tumor annotations or tumor predictions. In my case I used tumor predictions, made by one of my models.

#from wholeslidedata import WholeSlideImage
def load_mask(path, spacing):
    mask_obj = WholeSlideImage(path, backend='asap')
    mask = mask_obj.get_slide(spacing)
    return mask.squeeze()

In this method, a mask is loaded using the specific spacing. The function handles the nuances of image spacings and reads the mask at the desired resolution at once. The wholeslidedata package allows you to choose with what image backend you want to read the image. The default is OpenSlide, but in my case I use ASAP developed by Geert Litjens.

Mask to Polygon Conversion

#import shapely
#from rasterio import features
#import rasterio
def mask_to_polygons_layer(mask):
    all_polygons = []
    for shape, value in features.shapes(mask.astype(np.int16), mask=(mask > 0),
                                        transform=rasterio.Affine(1.0, 0, 0, 0, 1.0, 0)):
        all_polygons.append(shapely.geometry.shape(shape).buffer(0))

    all_polygons = shapely.geometry.MultiPolygon(all_polygons)
    return all_polygons

This function converts a binary mask into a set of polygons using rasterio and shapely. This geometric representation is crucial for efficient spatial operations later. Okay, I admit it, its maybe hard to read and it will not win a beauty award so let me guide you through this: Here, the function iterates through the shapes detected in the binary mask.

for shape, value in features.shapes(mask.astype(np.int16), mask=(mask > 0),
                                    transform=rasterio.Affine(1.0, 0, 0, 0, 1.0, 0)):

• mask.astype(np.int16): The mask, which is possibly in a boolean format, is converted into an integer format suitable for features.shapes().

• mask=(mask > 0): This specifies that we are interested in the features where the mask values are greater than 0 (i.e., the foreground regions).

• transform=rasterio.Affine(1.0, 0, 0, 0, 1.0, 0): This is an identity transformation. It means that there’s no spatial transformation applied to the mask.

Then for each detected shape, the following steps happen:

all_polygons.append(shapely.geometry.shape(shape).buffer(0))

• shapely.geometry.shape(shape): This converts the detected shape into a shapely polygon.

• .buffer(0): The buffer(0) trick is a common technique in GIS operations. It’s used here to fix potential topology errors in the polygons. Even though it might seem like it does nothing (since the buffer distance is 0), it can help in ensuring the created polygon is topologically valid.

Computing Cell Density

GeoPandas is an open-source Python library that makes working with geospatial data easier. While pandas is loved for data manipulation, it doesn’t have built-in capabilities to handle spatial data. GeoPandas, on the other hand, equips DataFrames with a geometry column, which can store spatial objects like points, lines, and polygons, allowing for sophisticated spatial operations. Just what I needed in this case!

Much like how pandas allows for efficient data operations due to its internal use of optimized C and Cython operations, GeoPandas spatial operations are vectorized, making spatial calculations very fast, alsoGeoPandas uses spatial indexing (via the R-tree data structure) for quick bounding box queries. When you’re trying to find overlaps between geometries (like in this case tumor polygon and cell points), spatial indexing ensures that you’re only comparing geometries that are close to one another, making overlap checks much quicker.

Lets get back to the problem and that was checking for million of cells if they where in the tumor area yes or no. GeoPandas was crucial and it took some time to figure it out, however as always, spending hours of time, the implementation was just a few lines of code. But before we start we first have to start with loading our cells.

#from wholeslidedata.annotation.wholeslideannotation import WholeSlideAnnotation
#from wholeslidedata.annotation.callbacks import ScalingAnnotationCallback
scaler = ScalingAnnotationCallback(1 / 8)
cells = WholeSlideAnnotation(cell_detections, callbacks=(scaler,), backend='asap')

The cell detections are in the annotation format that ASAP uses, so we can load them using the WholeSlideAnnotation class. For .xml files WholeSlideAnnotation uses ASAP as default backend. The callbacks argument is used to specify any preprocessing steps that need to be applied to the annotations. In this case, we need to scale the annotations to the same resolution as the tumor mask. In my case the cell detections in the .xml file are coordinates in the highest resolution, which is 0.25μm/pix. However, the tumor mask is at 2μm/pix. Therefore, we need to scale the coordinates by a factor of 8.

Now with our cell detections loaded we can breakdown the cell density calculation.

#import geopandas as gpd
def cell_density(tumor_poly, cells, spacing):
    cell_points = [cell._geometry for cell in cells]
    a = gpd.GeoDataFrame(cell_points).set_geometry(0)
    b = gpd.GeoDataFrame([tumor_poly]).set_geometry(0)
    check = gpd.tools.sjoin(a, b, how='left')
    counts = sum(check['index_right'].notna())
    return counts / (tumor_poly.area * spacing * 10 ** -6)

Here, the function converts tumor polygons and cell annotations into GeoDataFrames. The sjoin operation checks for overlaps between the two geometries. Without GeoPandas, you’d potentially be comparing every cell point with every part of the tumor polygon—a massive computational task, especially for high-resolution images. However, with spatial indexing, only cells that have a chance to overlap with the tumor are compared, drastically reducing the number of computations. Also note the set_geometry which was needed to designate a particular column as the ‘geometry’ column. This column then becomes responsible for holding the spatial data (like points, lines, or polygons) that can be used in spatial operations. I called it 0 due to convenience and make sure to use the same name so sjoin knows which geometries to compare.

check = gpd.tools.sjoin(a, b, how='left')

Left here means if a point from a lies within a polygon b, the resultant check DataFrame will have that point’s data combined with the data from the polygon it lies within. If a point from a doesn’t lie within any polygon in b, that point will still appear in the check DataFrame, but the columns corresponding to b will be filled with NaN values.

counts = sum(check['index_right'].notna())

Therefore, when calculating the sum of cells that are not NaN, we get the number of cells that are within the tumor polygon. We get this data by selecting column ‘index_right’ that contains the answers for each point.

return counts/(tumor_poly.area * spacing * 10 ** -6)

The purpose of the last line is to calculate and return the density of the points that are contained within the tumor polygon, in units of cells per square millimeter. Therefore we multiply with 10 ** -6 to convert the pixel spacing that has square micrometers (μm²) to square millimeters (mm²), so we end up with the metric cells/mm² tumor.

Everything put together

Want to see the full code? Check out this repository where I have put together an example notebook and normal python script and also included some example data.

In this case I like having autonomy and control of hosting my own data, but I also appreciate the efficiency and scalability of cloud services. That’s why, I’ve opted to host the database needed for Umami at home on my homeserver using Proxmox. However, when it comes to the dashboard – the visual heart of Umami – I use Netlify. Hosting the Umami dashboard on Netlify not only offloads my trusty NUC from running yet another service but also made installation a breeze with Netlify’s effortless deployment process. In this article I will show you how I set it up.

Setting Up Database on Proxmox

1. Download CT Template: Begin by obtaining the turnkey-postgresql CT template. The easiest way to do this is to download the template from the Proxmox web interface. Navigate to “Templates” -> “Download” and search for “turnkey-postgresql”. Select the template and click “Download”.

2. Create a LXC container:

   

   • Assign a fixed IP address (fixed to enable port forwarding).
   • Allocate 1024MB memory and 1024MB swap.
   • Dedicate 1 CPU core.
   • Designate 16GB for disk storage.
   • Ensure “start after created” is selected.

For me these where the resources that I had available, but you can adjust these to your own needs.

3. Complete Initialization: Access the console of your started container, log in with root and the password you set up at the previous step. Complete the installation, skipping any unnecessary add-ons but making sure to apply the updates.

   

4. Database Setup: Navigate to the browser using assigned-static-IP-address:12322 or simply input the IP address. This will take you to a dashboard where you can select Adminer. Use Adminer to log in with PostgreSQL credentials. Create a table named umami.

   

5. Port forwarding Make sure you don’t forget to enable port forwarding (5432) to the IP address of your running Postgres container. Ofcourse, this completely depends on your network setup. Keep in mind that exposing a port does come with security vulnerabilities. If you have an Ubiquiti router you can read this article for more information on how to set this up in a more safe way.

Deploying Umami on Netlify

1. Fork Repository: Fork the Umami repository to your GitHub account:

   https://github.com/umami-software/umami
   

2. Netlify Setup:

   • Log into Netlify.
   • Choose “Add New Site” -> “Import Existing Site”.
   • Opt for “Deploy with GitHub”.
   • Select your forked Umami repository.

3. Environment Variable:

   

   • In the site settings of your new project, navigate to “Site configuration” -> “Environment variables”.
   • Add the DATABASE_URL variable with the value:

     postgresql://<postgres_account>:<postgress_password>@<your_db_ip>/umami
     

4. Trigger Deployment:

   

   • In the Netlify dashboard, go to “Deploys”.
   • Select “Trigger Deploy” and choose “Clear cache and deploy site”.

The example here is just with Proxmox, but you can choose any popular solutions like AWS, Azure, DigitalOcean, or Heroku— provided they support PostgreSQL. At the end you just need to update the URL in the Netlify dashboard. Choose what’s best for your needs.

Digging into the problem, I started by examining the cover.html file in the theme. This file is crucial for handling images. As I skimmed through the lines, an interesting section of the code caught my attention:

## themes/PaperMod/layouts/partials/cover.html

{{- if $cover -}}{{/* i.e it is present in page bundle */}}
    ....optimisation code here
{{- else }}{{/* For absolute urls and external links, no img processing here */}}
    .... well...do nothing

The theme was checking for images within something called a “page bundle.” If the image wasn’t within this bundle, optimization processes wouldn’t be triggered. That was my “Aha!” moment. For the themes to work their magic, my content needed to be organized into Hugo’s page bundles.

So, with newfound clarity, I set about restructuring my content into these page bundles. The results? When building Hugo locally, the images were automatically resized and optimized, all thanks to the wonders of my new page bundles!

When I transitioned to page bundles, I anticipated that my config.yml for the CMS would require some adjustments. Luckily the CMS supported Page bundles by using so called collections and I followed the official documentation making the adjustments needed.

However, with every solution came a new challenge. Now that my content was structured the right way for image optimization, my newly configured Decap CMS (previously Netlify CMS) threw a tantrum. Suddenly, it stopped listing any of my articles. This was the most timeconsuming and puzzling setback.

I scoured the internet for answers, hoping to stumble upon someone who had faced a similar issue. And then, I came across a post on millerti.me that held the missing piece to my puzzle. The article mentioned the specific lines I needed to integrate in the collections:

# Support Hugo page bundles that puts index.md and images in folders named by slug
path: "{{slug}}/index"
media_folder: ""

Adding these lines was the magic touch. Now my CMS seamlessly stored new content, and it also listed my previous articles again and everything worked smoothly. This all resulted in the following config.yml:

## admin/config.yml
backend:
  name: git-gateway
  branch: main # Branch to update (optional; defaults to master)
site_url: https://dekeukenvandael.nl
logo_url: https://www.dekeukenvandael.nl/images/static/logo.svg
publish_mode: editorial_workflow
media_folder: static/images
public_folder: /images
collections:
  - name: 'recipes'
    label: "Recipes"
    label_singular: "Recipe"
    folder: 'content/recipes'
    create: true
    # Support Hugo page bundles that puts index.md and images in folders named by slug
    path: "{{year}}-{{month}}-{{day}}-{{slug}}/index"
    # Note the usage of title here instead of slug
    preview_path: 'recipes/{{year}}-{{month}}-{{day}}-{{title}}'
    preview_path_date_field: date
    filter: {field: "ignore", value: false}
    editor:
      preview: true
    fields:
      - { label: 'Title', name: 'title', widget: 'string' }
      - { label: 'Publish Date', name: 'date', widget: 'datetime' }
      - { label: "Authors", name: "author", widget: "list", summary: "{{fields.author_name}}", field: { label: "Author name", name: "author_name", widget: "string" }}
      - { label: 'Tags', name: 'tags', widget: 'list', required: false, items: [{ type: 'string' }] }
      - { label: 'Categories', name: 'categories', widget: 'list', required: false, items: [{ type: 'string' }] }
      - { label: 'Cover Image', name: 'cover', widget: 'object', fields: [
          { label: 'Image', name: 'image', required: false, widget: 'image' }
        ]
      }
      - { label: 'Summary', name: 'summary', widget: 'text', required: false }
      - { label: 'Body', name: 'body', widget: 'markdown' }

I hope it helps to reduce the time and effort for others facing similar issues. It wasn’t a straightforward process for me, and I’d be pleased if others can benefit from the lessons I learned. Maybe with the time you saved you can write an nice article about how this helped you ;).

1. Shortcodes: For manually embedding specific repositories in markdown.
2. Partials with Python: For automatically fetching and displaying selected repositories using a Python script.

1. Embedding a GitHub Repo Using Shortcodes

Styling with CSS:

For a consistent appearance between shortcodes and partials, we’re employing a basic CSS design. Here’s a style snippet that you can add to your site’s CSS file:

.github-card {
    border: 1px solid #e1e4e8;
    padding: 15px;
    margin: 20px 0;
    border-radius: 5px;
    box-shadow: 0 1px 3px rgba(0,0,0,0.12);
    background: #f6f8fa;
}
.github-card h2 {
    font-size: 1.2em;
    margin-bottom: 8px;
    margin: auto;
}
.github-card h2 a {
    text-decoration: none;
    color: #0366d6;
}
.github-card p {
    margin-bottom: 10px;
}
.card-details {
    display: flex;
    gap: 10px;
    font-size: 0.9em;
}
.language {
    color: #586069;
}
.stars, .forks {
    color: #586069;
}

This CSS ensures a neat card-style presentation for the GitHub repositories. Both our shortcode and our partial will utilize this styling.

The Shortcode:

Now, let’s create a new shortcode. In your Hugo site’s layouts/shortcodes directory, create a file called githubRepoCard.html and paste the following:

<div class="github-repo-card">
    <a href="{{ .Get "url" }}" target="_blank">
        <h3>{{ .Get "name" }}</h3>
    </a>
    <p>{{ .Get "description" }}</p>
    <p><strong>Language:</strong> {{ .Get "language" }}</p>
    <p><strong>Stars:</strong> {{ .Get "stars" }}</p>
    <p><strong>Forks:</strong> {{ .Get "forks" }}</p>
</div>

Using the Shortcode:

In any markdown file, you can now use the shortcode like this (note the space I added between {{ to prevent hugo from rendering the shortcode on this page):

{ {< githubRepoCard url="https://github.com/username/myrepo" name="MyRepo" description="This is my awesome repo" language="JavaScript" stars="100" forks="50" >}}

2. Auto-generating a Partial with Python

The Partial:

Similair to the shortcode we now make a partial called githubRepoCard.html and place it in our Hugo site’s layouts/partials:

<div class="github-repo-card">
    <a href="{{ .url }}" target="_blank">
        <h3>{{ .name }}</h3>
    </a>
    <p>{{ .description }}</p>
    <p><strong>Language:</strong> {{ .language }}</p>
    <p><strong>Stars:</strong> {{ .stars }}</p>
    <p><strong>Forks:</strong> {{ .forks }}</p>
    <p><strong>Last Updated:</strong> {{ .lastUpdated }}</p>
</div>

The Python Script:

The Python script is designed to scrape your GitHub profile and retrieve information about specific repositories. I use the BeautifulSoup library for this, which made parsing HTML and navigating the DOM pretty easy.

1. Setting Up

Before diving into the script, ensure you have the necessary libraries installed:

pip install requests beautifulsoup4

2. Initialize Constants and Fetch Profile Page

Start by setting up the base URL and headers. The headers, especially the ‘User-Agent’, help in avoiding blocks when making requests to GitHub.

URL = "https://github.com/daangeijs?tab=repositories"
HEADERS = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'
}

Then, fetch the repos from the main profile page, I found the information for each in the div with the class col-10 col-lg-9 d-inline-block:

response = requests.get(BASE_URL, headers=HEADERS)
soup = BeautifulSoup(response.content, 'html.parser')
repos = soup.find_all('div', class_='col-10 col-lg-9 d-inline-block')

3. Filtering Repositories

Since I don’t want to list all my repos on the website I only collect a subset of repositories. To do this, I create a list of repository names that I want to include:

specified_repos = ['daangeijs.nl', 'dekeukenvandael.nl']
partials = []
nl = '\n' # Newline character since it is not possible to use newlines in formatted strings

4. Loop through Repositories

Now, for each repository on the profile page, check if its name matches any of the names in our specified_repos list. If there’s a match, extract additional information:

for repo in repos:
    name = repo.find('a', itemprop='name codeRepository').text.strip()
    
    if name in specified_repos:
        ...

5. Dive Deeper into Each Repo

For more detailed data, like the exact number of stars and forks, we need to visit each repo’s individual page, here you see how I extracted the main project language, description and last updated time:

# Extract base info  
repo_url = f"https://github.com{repo.find('a', itemprop='name codeRepository')['href']}"  
description_elem = repo.find('p', itemprop='description')  
description = description_elem.text.strip() if description_elem else "No description provided."  
language_elem = repo.find('span', itemprop='programmingLanguage')  
language = language_elem.text.strip() if language_elem else "Not specified"  
last_updated = repo.find('relative-time').attrs.get('datetime', 'Unknown Date')

6. Extract Stars, Forks, and Last Updated Time

Using these specified HTML locations we extract the number of stars and forks:

# Fetch repo-specific page to extract stars and forks  
repo_response = requests.get(repo_url, headers=HEADERS)  
repo_soup = BeautifulSoup(repo_response.content, 'html.parser')  
  
stars_elem = repo_soup.find('span', id="repo-stars-counter-star")  
forks_elem = repo_soup.find('span', id="repo-network-counter")  
  
stars = stars_elem['title'] if stars_elem and 'title' in stars_elem.attrs else '0'  
forks = forks_elem['title'] if forks_elem and 'title' in forks_elem.attrs else '0'

We then generate the partial string that we add to a list

partial = f'''{{{{ partial "githubRepoCard.html" (dict "url" "{repo_url}" "name" "{name}" "description" "{description}" "language" "{language}" "stars" "{stars}" "forks" "{forks}" "lastUpdated" "{last_updated}") }}}}'''  
partials.append(partial)

7. Save Extracted Data as Hugo Partials

Once all the desired data is scraped and the partials are generated we save them in a file and we add a little bit of html to create a div and title around the partials:

wrapped_content = f'''
<div class="latest-publications">
    <h2>My GitHub repositories</h2>
    {"\n".join(partials)}
</div>
'''

with open("partials_generated.html", "w") as file:
    file.write(wrapped_content)

Using the Generated HTML:

Once you run the Python script, it’ll create an HTML file named partials_generated.html in your directory. This file contains wrapped partial calls, which you can include in your Hugo layouts or content. I store mine here: layouts/partials/github.html

To add the partial to a page simply use this in your html template: {{ partial "github.html" }}

So that is it, two ways I implemented partials and shortcodes to add my GitHub repo’s to this Hugo generated website. Here the final result together with the already existing gist shortcode from Hugo. Stacked on each other it looks great! For the latest version have a look at the source of my website. Or you can copy the code for the python script below:

In this article I wrote down a step-by-step walkthrough on creating an isolated environment for hosting services, using a VLAN setup. By using a VLAN and setting up a firewall we can isolate the virtual server from your primary private network, layering an additional shield of security to your setup. As an example we’ll use a virtual machine (VM) hosting a database service as our primary example.

Ubiquiti VLAN Configuration

1. Create a VLAN

• Log into your UniFi Controller.
• Navigate to the “Settings” (gear icon) at the bottom left.
• Under “Networks”, click on “Create New Network”.
• Provide a name for the network, for instance, “Public VLAN”.
• Set “Purpose” to “Corporate”.
• Assign a VLAN ID of “10”.
• Define the subnet as 10.0.0.1/24.
• Configure the DHCP range if required and save these settings.

2. Firewall Rules for VLAN Traffic

• Proceed to “Routing & Firewall” within the settings.

• Select “Firewall” and then “LAN IN”.

• Set up a rule that permits only PostgreSQL traffic:

  • Name: Allow PostgreSQL to WAN
  • Action: Accept
  • Source: Public VLAN
  • Destination: Any
  • Ports: 5432 (PostgreSQL’s default port)

• Create rules that block all traffic from the VLAN to other local networks:

  • Name: Block VLAN to all LANs
  • Action: Drop
  • Source: Public VLAN
  • Destination: All other local networks/VLANs

3. Port Forwarding

• Navigate to “Routing & Firewall” and select “Port Forwarding”.

• Click on the “+ Create New Rule” or “Add New Port Forward Rule” button, which should open a new window or pane for rule creation.

• Name: Give the rule a descriptive name, e.g., “PostgreSQL Remote Access”.

• Enabled: Make sure this is toggled on.

• Rule Applied: Set to “After Predefined Rules”

• WAN Interface: Usually set to “All” unless you have multiple WANs and prefer a specific one.

• Original IP: Leave as “Any” to allow access from any external IP or specify a range/IP if you have a static IP where you’ll be connecting from.

• Original Port: Set to the PostgreSQL default port, “5432”.

• Forward IP: Enter the IP address of the machine where PostgreSQL is running, in this case, the VM’s IP, 10.0.0.2.

• Forward Port: Again, set this to “5432”.

• Protocol: PostgreSQL typically uses TCP, so set this to “TCP”. If there are any reasons to believe you need both TCP and UDP, you can set it to “Both”, but this is usually not necessary for PostgreSQL.

Proxmox VM Configuration

1. VM Creation or Modification

Now lest assign our newly created VLAN to a VM. Either initiate a new VM or select an existing one.

• Access the Proxmox web interface.
• During the setup or via the “Network” menu for an existing VM:
  • Set the Bridge, you can use your default, typically vmbr0.
  • Assign the VLAN Tag to “10”.
  • Ensure the firewall is activated.
  • IPv4/CIDR: 10.0.0.2/32
  • Gateway: 10.0.0.1
• Once the VM is started or rebooted, it should automatically acquire the assigned static IP.

Testing the Configuration

1. Verify the IP Address

• In Proxmox, access the VM’s console.
• Execute the ifconfig command to ensure that the IP address 10.0.0.2 has been correctly assigned.

2. Test Connectivity

• In the same console, check internet access by pinging an external website: ping www.daangeijs.nl.
• Subsequently, attempt to ping a device from your private network. This ping should fail, verifying that the VM is isolated from the private network.

There you go! You’ve successfully set up a VLAN and isolated a VM within it. You can now host services on this VM and access them from the internet, while keeping your private network secure.

In this article, I’ll guide you through creating a Docker-based system for backing up and restoring your /home/user directory, storing backups on a mounted HDD, and implementing a 30-day data retention policy. Later, we will discuss how to automate this backup process using crontab.

1. Set Up the Docker Environment:

Dockerfile:

We’ll start with a Docker image based on the Debian slim version, equipped with essential tools for our operations:

FROM debian:bullseye-slim

RUN apt-get update && \
    apt-get install -y tar && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /backup
COPY .. .
RUN chmod +x backup_restore.sh
ENTRYPOINT ["./backup_restore.sh"]

Here, we’re setting up a Debian environment, installing the tar tool for compression, setting a working directory, and copying our backup script.

2. Backup and Restore Script:

The core logic resides in our backup_restore.sh script, where we define the backup, restore, and cleanup functions.

#!/bin/bash

# Destination directory on the mounted HDD
BACKUP_DEST="/mnt/storage/backup"
BACKUP_TARGET_PATH="/home/daan"

function backup() {
    tar czf ${BACKUP_DEST}/homeserver_backup_$(date +%Y%m%d).tar.gz -C ${BACKUP_TARGET_PATH} .
}

function restore() {
    local tarball=$1
    local restore_path=$2
    tar xzf ${BACKUP_DEST}/${tarball} -C ${restore_path}
}

function cleanup() {
    find ${BACKUP_DEST} -name 'homeserver_backup_*.tar.gz' -mtime +30 -exec rm {} \;
}

if [ "$1" == "backup" ]; then
    backup
    cleanup
elif [ "$1" == "restore" ] && [ -n "$2" ] && [ -n "$3" ]; then
    restore $2 $3
else
    echo "Usage:"
    echo "./backup_restore.sh backup"
    echo "./backup_restore.sh restore tarball_filename /path/to/restore"
fi

Functions Explained:

• backup(): This function compresses and archives the /home/user directory into a tarball, which is saved directly to the mounted HDD.

• restore(): Allows the user to specify a backup tarball and restore its contents to a given directory.

• cleanup(): Implements the data retention policy by deleting backups older than 30 days from the mounted HDD.

3. Building and Running the Docker Container:

First, build your Docker image:

docker build -t userbackuprestore:latest .

To run the Docker container:

• For backup:

docker run --rm -v /home/user:/home/user -v /mnt/storage:/mnt/storage userbackuprestore:latest backup

• For restore:

Replace your_tarball_filename.tar.gz with the desired backup tarball’s filename and /path/to/restore with the directory where you want to restore.

docker run --rm -v /path/to/restore:/restore -v /mnt/storage:/mnt/storage userbackuprestore:latest restore your_tarball_filename.tar.gz /restore

Automating Backups Using Crontab:

After setting up the Docker-based backup system, automating the process ensures that backups are taken regularly without manual intervention. The cron job scheduler is an excellent tool for this. Here’s how you can schedule the backup task using crontab.

1. Open Crontab:

To edit the current user’s crontab entries, use:

crontab -e

This will open up the default editor, often vi or nano, depending on the system setup.

2. Add a Cron Job:

To run the backup daily at, say, 2:30 AM, add the following line:

30 2 * * * docker run --rm -v /home/user:/home/user -v /mnt/storage:/mnt/storage userbackuprestore:latest backup

The general format of a cron job is:

[min] [hour] [day of month] [month] [day of week] [command]

For this job:

• 30 is the minute (30 minutes past the hour).
• 2 is the hour (2 AM).
• * for day of month, month, and day of week indicates “every” or “any.”

Therefore, 30 2 * * * means “2:30 AM, every day.”

3. Save and Exit:

After adding the line:

• If you’re in vi, press Esc, type :wq, and press Enter.
• If you’re in nano, press CTRL + X, press Y to confirm changes, and press Enter to save.

4. Verify the Cron Job:

To ensure your cron job has been set correctly, you can display the current user’s crontab entries:

crontab -l

You should see the line you added for the backup

Problem: After bolstering the storage capacity of a Linux VM within Proxmox or another hypervisor, you’ll discover that the operating system inside the VM doesn’t automatically recognize or utilize this newly added space.

Solution: To harness this additional storage, you need to resize both the partitions and the filesystems within the Linux environment. If you’ve configured your VM with Logical Volume Management (LVM), this entails adjusting the physical volume, the logical volume, and subsequently, the filesystem.

1. Ensure the Partition Covers the New Space:

   To observe the current partitions:

   sudo fdisk -l
   

   Example Output:

   Device     Start      End  Sectors  Size Type
   /dev/sda1   2048   999423   997376  487M EFI System
   /dev/sda2 999424 20479999 19480576  9.3G Linux filesystem
   

   If, for instance, /dev/sda2 isn’t leveraging the entire disk space, you need to adjust it. Here’s how:

   Launch the partition tool for the disk:

   sudo fdisk /dev/sda
   

   • Press p to display the existing partition layout.
   • Ensure you note the start sector of the partition you intend to resize.
   • Press d to delete the desired partition, and select its number (like 2 for /dev/sda2).
   • Hit n to create a new partition. Use the exact start sector from earlier and allow the default end sector to encompass all available space.
   • Press t to modify the partition type and assign it to 8e, denoting Linux LVM.
   • To apply the changes, press w.

   Example Output:

   Command (m for help): n
   Partition number (2-4, default 2): 2
   First sector (999424-49971199, default 999424): 999424
   Last sector, +/-sectors or +/-size{K,M,G,T,P} (999424-49971199, default 49971199): 
   Created a new partition 2 of type 'Linux filesystem' and of size 23.3 GiB.
   

2. Inform the Kernel About Partition Changes:

   sudo partprobe
   

3. Expand the LVM Physical Volume:

   To view the current status:

   sudo pvdisplay
   

   Example Output:

   --- Physical volume ---
   PV Name               /dev/sda2
   VG Name               ubuntu-vg
   PV Size               9.30 GiB
   

   To resize the physical volume:

   sudo pvresize /dev/sda2
   

4. Expand the LVM Logical Volume:

   sudo lvresize -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv
   

5. Resize the Filesystem:

   sudo resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv
   

6. Verify the Changes:

   df -h
   

   Example Output:

   Filesystem                             Size  Used Avail Use% Mounted on
   /dev/mapper/ubuntu--vg-ubuntu--lv      23G  4.8G   17G  22% /
   

LVM vs. Logical Volume: What’s the Difference?

LVM (Logical Volume Management) offers a flexible and agile solution for storage management, enabling the seamless management of disk drives and similar storage mechanisms. Under the LVM umbrella, you’ll encounter:

• Physical Volumes (PV): These are your raw storage devices or partitions that store data.
• Volume Groups (VG): Collections of physical volumes, they act as one consolidated storage reservoir.
• Logical Volumes (LV): These lie within a volume group and act as block devices that sustain the filesystem.

In essence, LVM facilitates the amalgamation of several disks (or partitions) into a singular storage pool (VG). From this pool, logical subdivisions (LVs) can be extracted, upon which filesystems are created.

Thus, when we discuss “expanding LVM”, we’re essentially alluding to a series of tasks that incorporate expanding the physical volume (PV), possibly the volume group (VG), followed by the logical volume (LV). Only post these operations is the filesystem itself stretched to occupy the new space.

Endowed with these precise commands and sample outputs, you’re equipped with a clear roadmap of what to anticipate at every juncture. As a precaution, always ensure your data is backed up before initiating any significant modifications to your storage structures.

Backing up your data regularly is essential, especially for critical applications like Home Assistant running MariaDB in a Docker container. This article will guide you through creating scripts that automate the backup and restoration processes, designed to be flexible by accepting arguments for the password and paths.

Prerequisites

• Docker installed and running.
• MariaDB container running with your data.

Backup Script with Arguments

To make our backup process versatile and reusable, we’ll create a shell script that accepts two arguments:

1. MariaDB password
2. Backup folder path

Script Creation

1. Create and open a new script:

nano /path_to_scripts_folder/backup_mariadb.sh

2. Copy and paste the following content:

#!/bin/bash

# Check if the right number of arguments are provided
if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <password> <backup_folder_path>"
    exit 1
fi

PASSWORD=$1
BACKUP_PATH=$2

docker exec mariadb /usr/bin/mysqldump -u homeassistant --password=$PASSWORD --all-databases | gzip > "$BACKUP_PATH/database_backup_$(date +\%F).sql.gz"

3. Save and exit the editor.

4. Make the script executable:

chmod +x /path_to_scripts_folder/backup_mariadb.sh

Now, run the script, passing the password and backup folder path as arguments:

/path_to_scripts_folder/backup_mariadb.sh your_password /path_to_backup_folder/

Scheduling Backups with Crontab

To automate the backup process daily:

1. Open the crontab:

crontab -e

2. Add the following line to run the script every day at 3 am:

0 3 * * * /path_to_scripts_folder/backup_mariadb.sh your_password /path_to_backup_folder/

Restoring the Database from Backup

Restoration Script with Arguments

To simplify the restoration process, we’ll create a separate shell script that accepts two arguments:

1. MariaDB password
2. Path to the backup file you want to restore

Script Creation

1. Create and open a new script:

nano /path_to_scripts_folder/restore_mariadb.sh

2. Copy and paste the following content:

#!/bin/bash

# Check if the right number of arguments are provided
if [ "$#" -ne 2 ]; then
    echo "Usage: $0 <password> <backup_file_path>"
    exit 1
fi

PASSWORD=$1
BACKUP_FILE=$2

gunzip < "$BACKUP_FILE" | docker exec -i mariadb /usr/bin/mysql -u homeassistant --password=$PASSWORD

3. Save and exit the editor.

4. Make the script executable:

chmod +x /path_to_scripts_folder/restore_mariadb.sh

Now, run the script, passing the password and the path to the backup file as arguments:

/path_to_scripts_folder/restore_mariadb.sh your_password /path_to_backup_folder/database_backup_YOUR_DATE.sql.gz

Notes on Restoration

1. Backup before restore: Always take a fresh backup before starting the restoration process. This ensures you have a fallback if the restore doesn’t go as planned.
2. Check Compatibility: Ensure the MariaDB version you are restoring to is compatible with the version from which you took the backup.
3. Downtime Considerations: Depending on the size of your database and the restoration environment’s performance, the restoration process might take some time. Plan accordingly.

This post will show you how I added the publications on my website, that is generated by Hugo. I will try to provide step-by-step instructions on exporting your .bib file from Google Scholar and converting it into TOML-formatted markdown files using Python.

Step 1: Exporting .bib from Google Scholar

1. Login to Google Scholar: Head over to Google Scholar and sign in using your credentials.
2. Access “My Library”: Here you will find all your saved articles and citations.
3. Select Articles: Choose the articles you wish to export by ticking the checkboxes next to them.
4. Click the Export Button: This button, usually represented by quotation marks, will give you different export options.
5. Choose BibTeX: Select the BibTeX option to export your articles in the .bib format.
6. Save the File: The BibTeX formatted content will be displayed in a new window. Copy this content and save it in a .bib file, for example, references.bib.

Step 2: Converting .bib to TOML-formatted Markdown with Python

Now that we have our .bib file, let’s use Python to parse the file and create individual markdown files with TOML front matter.

a. Setting Up the Environment

Before delving into the code, make sure you have the required Python libraries:

pip install pybtex pylatexenc

b. Code Breakdown

i. Key Imports:

• os & pathlib.Path: Used for directory and file path operations.
• bibtex: To parse the .bib file.
• latex2text: From pylatexenc, used to convert LaTeX to Unicode.

import os
from pybtex.database.input import bibtex
from pathlib import Path
from pylatexenc.latex2text import LatexNodes2Text

ii. LaTeX to Unicode Conversion:

We’ll use the pylatexenc library to easily convert LaTeX-specific text into Unicode. This is especially useful for author names, titles, or sources that might use LaTeX-style formatting.

def latex_to_unicode(text):
    return LatexNodes2Text().latex_to_text(text)

iii. Generating Markdown:

The create_or_update_md function will be responsible for converting each .bib entry into its markdown equivalent with TOML front matter. By default it will set the publication to hidden=true. When hidden the publication will not be listed as an article, but will be listed as non-clickable entry in the partial.

def create_or_update_md(entry):
    # Extracting common fields
    key = entry.key
    title = entry.fields.get('title', '')
    year = entry.fields.get('year', '')
    journal = entry.fields.get('journal', entry.fields.get('booktitle', ''))
    volume = entry.fields.get('volume', '')
    pages = entry.fields.get('pages', '')
    publisher = entry.fields.get('publisher', '')
    url = entry.fields.get('url', '')

    # Extracting the author list and convert it to a list of authors
    authors = [latex_to_unicode(str(author)) for author in entry.persons.get('author', [])]

    # Define the path
    folder_path = Path(f"content/publications/{key}")
    file_path = folder_path / "index.md"

    # Check if file exists
    if not file_path.exists():
        # If file doesn't exist, create a new markdown file with TOML front matter
        folder_path.mkdir(parents=True, exist_ok=True)
        with open(file_path, 'w') as f:
            f.write("+++\n")
            f.write(f'title = "{title}"\n')
            f.write('hidden = true\n')
            f.write(f'authors  = {authors}\n')
            f.write(f'date = {year}-01-01\n')
            f.write(f'journal = "{journal}"\n')
            f.write(f'volume = "{volume}"\n')
            f.write(f'pages = "{pages}"\n')
            f.write(f'publisher = "{publisher}"\n')
            f.write(f'url = "{url}"\n')
            f.write("+++\n\n")
            f.write(f"Summary about {title}.")

iv. Parsing & Execution:

Here, we parse the .bib file and process each entry:

def main():
    parser = bibtex.Parser()
    bib_data = parser.parse_file("resources/references.bib")

    for entry in bib_data.entries.values():
        create_or_update_md(entry)

if __name__ == "__main__":
    main()

Step 3: Displaying in Hugo

You will need font-awesome package, you can add this head to your partial or in a included template, but just make sure you include it somewhere.

<head>
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/css/all.min.css">
</head>

After running the Python script, you’ll have a collection of markdown files ready to be displayed in Hugo. Here’s how you can showcase them using the provided Hugo partials.

{{- $pages := where .Site.RegularPages "Section" "publications"}}
{{- $sortedPages := $pages.ByDate.Reverse }}
{{- $latestPublications := $sortedPages | first 5 }}

<div class="latest-publications">
    <h2>Latest Publications</h2>

    <ul class="">
        {{- range $latestPublications }}
        <li>
            {{ $currentPublication := . }}

            {{ if ne .Params.hidden true }}
            <a href="{{ .Permalink }}">{{ .Title }} <i class="fas fa-file-alt"></i></a> ({{ .Date.Format "2006" }})
            {{ else }}
            {{ .Title }} ({{ .Date.Format "2006" }})
            {{ end }}
            <p>
                {{ range $i, $author := .Params.authors }}
                {{ if lt $i 1 }}{{ $author }}{{ if lt $i (sub (len $currentPublication.Params.authors) 1) }} and {{ end
                }}{{ end }}
                {{ if eq $i 1 }}{{ $author }} et al.{{ end }}
                {{ end }}
            </p>
        </li>
        {{- end }}
    </ul>

</div>

Simply embed this partial into your desired Hugo template, and your publications will be presented.

The Complete Python Script:

For those who want to dive straight in, here’s the full Python script:

import os
from pybtex.database.input import bibtex
from pathlib import Path
from pylatexenc.latex2text import LatexNodes2Text
import re


def latex_to_unicode(text):
    return LatexNodes2Text().latex_to_text(text)


def create_or_update_md(entry):
    # Extracting common fields
    key = entry.key
    title = entry.fields.get('title', '')
    year = entry.fields.get('year', '')
    journal = entry.fields.get('journal', entry.fields.get('booktitle', ''))
    volume = entry.fields.get('volume', '')
    pages = entry.fields.get('pages', '')
    publisher = entry.fields.get('publisher', '')
    url = entry.fields.get('url', '')

    # Extracting the author list and convert it to a list of authors
    authors = [latex_to_unicode(str(author)) for author in entry.persons.get('author', [])]

    # Define the path
    folder_path = Path(f"content/publications/{key}")
    file_path = folder_path / "index.md"

    # Check if file exists
    if not file_path.exists():
        # If file doesn't exist, create a new markdown file with TOML front matter
        folder_path.mkdir(parents=True, exist_ok=True)
        with open(file_path, 'w') as f:
            f.write("+++\n")
            f.write(f'title = "{title}"\n')
            f.write('hidden = true\n')
            f.write(f'authors  = {authors}\n')
            f.write(f'date = {year}-01-01\n')
            f.write(f'journal = "{journal}"\n')
            f.write(f'volume = "{volume}"\n')
            f.write(f'pages = "{pages}"\n')
            f.write(f'publisher = "{publisher}"\n')
            f.write(f'url = "{url}"\n')
            f.write("+++\n\n")
            f.write(f"Summary about {title}.")


def get_author_list(entry):
    if 'author' in entry.persons:
        authors = entry.persons['author']
        author_str = ' and '.join(str(author) for author in authors)
        return latex_to_unicode(author_str)
    return ''


def update_field(content, field, value):
    # If the field exists, update it. If not, just return the content as is.
    if f"{field}:" in content:
        content = re.sub(f'{field}:.*', f'{field}: "{value}"', content)
    return content


def main():
    parser = bibtex.Parser()
    bib_data = parser.parse_file("resources/references.bib")

    for entry in bib_data.entries.values():
        create_or_update_md(entry)


if __name__ == "__main__":
    main()