flurischt.ch

Encrypted Duplicity Backup to European Amazon S3 Bucket

2017-11-05T00:00:00+01:00

Introduction

duplicity supports backups to AWS S3. There are a lot of howtos out there but most assume that you want to backup to the (default) US region. It gets more complicated for other regions and your mostly left on your own with strange errors from boto, the library that talks to AWS. Below I document how to use duplicity to do GPG encrypted backups to S3 in the EU (Ireland) (eu-west-1) region.

Setting up S3 and IAM

We'll create an S3-bucket and an IAM user with full access to that bucket. Make sure to use EU (Ireland) region so that the backup-script below works.

Go to the S3 console and create a new bucket in region EU (Ireland). Do NOT use dots in the bucket-name (#2836). Let's say we use MY-S3-BUCKET-NAME.
Next go to Identity and Access Management (IAM), create a new user with programmatic access and write down the access key and secret access key.
Still in IAM create a new policy with the content below and attach it to the user. Make sure to replace <MY-S3-BUCKET-NAME> with the actual bucket name.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<MY-S3-BUCKET-NAME>",
                "arn:aws:s3:::<MY-S3-BUCKET-NAME>/*"
            ]
        }
    ]
}

Server Setup

Assuming Ubuntu we'll install the latest duplicity version from the ppa:

root@server:~# add-apt-repository ppa:duplicity-team/ppa
root@server:~# apt clean && apt update && apt install -y duplicity python-boto

Next we'll need a gpg key to encrypt the backup stored in the cloud:

user@server:~# sudo -i # we'll run the backup as root. so make the gpg key using user root too
root@server:~# gpg --gen-key # use kind "RSA and RSA (default)", name "duplicity" and set a passphrase
root@server:~# gpg --list-keys
/root/.gnupg/pubring.gpg
------------------------
pub   4096R/<The-GPG-Key-ID> 2017-12-05
uid                  duplicity (...) <....>
sub   4096R/....... 2017-12-05
# export the gpg keys and keep them at a secure location:
root@server:~# gpg -a --export duplicity
root@server:~# gpg -a --export-secret-key duplicity

AGAIN: Save the gpg key at a secure location and not only on the server. If the server goes down you will not be able to restore your backups without that key!

Backup Script

On the server we'll create two files: /root/.duplicity_secrets with the S3 access keys and /root/backup_to_s3.sh which will be the backup script.

/root/.duplicity_secrets

This file will hold the S3 access keys and the passphrase to the gpg key:

export PASSPHRASE="<THE-PASSPHRASE-TO-YOUR-GPG-KEY>"
export AWS_ACCESS_KEY_ID="<YOUR-IAM-USERS-ACCESS-KEY>"
export AWS_SECRET_ACCESS_KEY="<YOUR-IAM-USERS-SECRET-ACCESS-KEY>"

/root/backup_to_s3.sh

The backup script will source the .duplicity_secrets file and then backup to S3. Below is an example how to backup /etc into a subdirectory server-name/etc in the S3-bucket. Make sure to configure <The-GPG-Key-ID> and <MY-S3-BUCKET-NAME> in the following script:

source /root/.duplicity_secrets

# duplicity args
# 
s3_bucket="s3://s3.eu-west-1.amazonaws.com/<MY-S3-BUCKET-NAME>/${HOSTNAME}"
encrypt_key="<The-GPG-Key-ID>"
aws_args="--s3-european-buckets --s3-use-new-style"
# experimental setting that may speed up the backup: --asynchronous-upload

# backup /etc to <MY-S3-BUCKET-NAME>/hostname/etc
/usr/bin/duplicity $aws_args --encrypt-key $encrypt_key -v 4 \
    incr --full-if-older-than 14D \
    /etc ${s3_bucket}/etc

unset PASSPHRASE
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY

Some Duplicity commands

Checkout: https://help.ubuntu.com/community/DuplicityBackupHowto

Some examples:

root@server:~# source /root/.duplicity_secrets 
root@server:~# export S3_BUCKET="s3://s3.eu-west-1.amazonaws.com/<MY-S3-BUCKET-NAME>/<SET-YOUR-SERVER-HOSTNAME-HERE>"
# list files in the backup
root@server:~# duplicity list-current-files $S3_BUCKET/etc
# restore /etc/fstab to /tmp/restored_fstab
root@server:~# duplicity --file-to-restore fstab $S3_BUCKET/etc /tmp/restored_fstab

Troubleshooting

We're connecting duplicity directly to s3.eu-west-1.amazonaws.com. This is EU (Ireland) region and will therefore only work if you setup the bucket in the correct region. If you're using another region you may have to change the URL and/or play with $aws_args in the backup script. If you're getting ACCESS DENIED errors make sure to check the IAM policy and that it is attached to the user.

Credits

A lot of this post is based on Igor Cicimov's Duplicity encrypted backups to Amazon S3 post. I followed the steps described there but ran into S3 upload problems with the URL-schema used there. That's why I'm using eu-west-1 and directly connect to that endpoint.

Sharir and Pnueli's TWO APPROACHES TO INTERPROCEDURAL DATA FLOW ANALYSIS

2017-08-18T00:00:00+02:00

This paper is really hard to come by and all scans are either incomplete or include the empty backpages. Here's a cleaned up version: Download

Spellcheck in Latex Files

2017-08-10T00:00:00+02:00

I use the following snippet to spellcheck latex files:

#!/bin/bash
hunspell -d en_GB -t $1

Put it into a spellcheck.sh file. Then ./spellcheck.sh myfile.tex.

Manually starting ssh-agent to use it in a (remote) terminal session

2017-06-22T00:00:00+02:00

ssh-agent can be used to unlock an ssh private key and therefore prevent having to enter the password multiple times. Usually your desktop environment does this for you. But if you're for example sshing into your workstation and then use ssh (e.g. for git or hg) your desktop environment won't help you. Here's how to start ssh-agent and unlock your private key:

> eval `ssh-agent -s`
> ssh-add

Now you can use the unlocked key to make ssh connections. And after your work is done here's how to kill ssh-agent and unlock the key again:

> ssh-agent -k

Python networkx (graph library) example

2017-06-06T00:00:00+02:00

This is just for me to look it up again later:

#!/usr/bin/env python3

"""
A small example that uses networkx and pyplot to create a DiGraph, 
draw it and compute floyd-warshall on it.
"""

import networkx as nx
import matplotlib.pyplot as plt

#https://www.debian.org/vote/2013/vote_001
# The beats matrix:
#
#           1   2   3   4
#Option 1       88  62  319
#Option 2   252     141 355
#Option 3   279 210     353
#Option 4   51  24  30   
BEATS_MATRIX = [
    [0, 88, 62, 319],
    [252, 0, 141, 355],
    [279, 210, 0, 353],
    [51, 24, 30, 0],
]
NUM_OPTIONS = 4

# create a graph with candidates that beat each other
G=nx.DiGraph()

# create nodes for all options
for option in range(NUM_OPTIONS):
    G.add_node(option + 1)

# connect winner to loser options
for o1 in range(NUM_OPTIONS):
    for o2 in range(o1+1, NUM_OPTIONS):
        if BEATS_MATRIX[o1][o2] > BEATS_MATRIX[o2][o1]:
            u = o1+1
            v = o2+1
            c = BEATS_MATRIX[o1][o2] - BEATS_MATRIX[o2][o1]
        else:
            u = o2+1
            v = o1+1
            c = BEATS_MATRIX[o2][o1] - BEATS_MATRIX[o1][o2]
        print("option {} wins over {} by {} votes".format(u, v, c))
        G.add_edge(u, v, weight=c)


# all pairs shortest path example
distance = nx.floyd_warshall(G)
for u in range(1, NUM_OPTIONS+1):
    for v in range(1, NUM_OPTIONS+1):
        print("{} ".format(distance[u][v]), end="")
    print()

#drawing example
pos=nx.spectral_layout(G) # positions for all nodes
nx.draw_networkx_nodes(G, pos, node_size=700)
nx.draw_networkx_edges(G, pos, edgelist=G.edges(), width=1)
nx.draw_networkx_labels(G, pos, font_size=15, font_family='sans-serif')
edge_labels=dict([((u,v,),d['weight'])
             for u,v,d in G.edges(data=True)])
nx.draw_networkx_edge_labels(G, pos, edge_labels=edge_labels,\
    font_size=15, font_family='sans-serif')

plt.axis('off')
plt.show()

Music for working / learning (updated)

2017-05-27T00:00:00+02:00

Some nice music on youtube and soundcloud:

IntelliJ Idea application menu entry for xubuntu

2017-05-25T00:00:00+02:00

Assuming intellij has been installed to /opt/ here's how to create an application menu entry (on xubuntu). Create a file ~/.local/share/applications/intellij.desktop with the following content:

[Desktop Entry]
Version=1.0
Type=Application
Name=IntelliJ
Icon=/opt/jetbrains/idea/bin/idea.png
Exec=/opt/jetbrains/idea/bin/idea.sh
NoDisplay=false
Categories=Development;
StartupNotify=false
Terminal=false

Extend a VirtualBox guest harddisk

2017-04-18T00:00:00+02:00

I use VirtualBox for the rare cases when I need a Windows OS. The latest Win10 updates failed to install because there's not enough disk space. Apparently 30GB is not enough for a whole OS and office... ;-(

Luckily that's easy to fix:

Shutdown the guest
Make sure there are no snapshots of the virtual machine (I read somewhere that it doesnt work with snapshots. Didn't try it myself)
Resize the disk: VBoxManage modifyhd VIRTUAL_MACHINE_NAME.vdi --resize 40960
Start the guest
Extend the C: partition (Right click on the partition in windows disk management)

The above steps resize the disk to 40GB. Don't forget the last step. Otherwise you'll have a 40GB disk but Windows won't be using the whole drive.

DD-WRT (kong) on Netgear R7000 and Fiber7 IPv6 configuration

2017-01-21T00:00:00+01:00

I'm a fan of Netgear's R7000 hardware. Not so much of the software. Since I bought this router I was not able to get IPv6 working with my Fiber7 internet connection. The Netgear firmware does not support DHCPv6-PD which is what my provider requires. Anyway, not having IPv6 was not a big problem (yet). But the latest security vulnerability debacle (see CVE-2016-6277 and CVE-2017-5521) was too much. Let's switch to DD-WRT.

Performance side remarks

According to the DD-WRT wiki Netgear uses proprietary hardware acceleration for IPv4 NAT. If you use a free firmware you'll lose this hardware acceleration and the throughput will be lower. So with a fast internet connection >= 500 Mbit/s you'll notice this. I have a fast internet connection but don't care about it. We don't get the full speed over W-Lan anyway and I'd rather get rid of the Netgear firmware than have a higher throughput, no IPv6 and all the security udpates...

Which version?

Apparently there are two different versions of DD-WRT for the R7000. There's Kong's mod and then there are the builds on the DD-WRT website. I decided to use Kong's mod because they seem to be the dev who brought R7000 support to DD-WRT and their firmware has some additional features (like for example a cli update program).

DD-WRT Installation

I use dd-wrt.K3_R7000.chk (dated 18th of January 2017) from Kong's website. This assumes that currently the official netgear firmware is installed.

Reset router to factory defaults using the admin interface
Use the admin interface to upload the new firmware
There might be a warning about installing the same or an older firmware. Just accept it.
As soon as the router is available visit http://192.168.1.1 and set a DD-WRT admin username and passwort
Now head to http://192.168.1.1/Factory_Defaults.asp and Restore Factory Defaults.
Router will reboot and again you need to set username and password on http://192.168.1.1.
Since it's not clear (at least for me) whether Restore Factory Defaults also deletes the nvram I did this manually:
- telnet 192.168.1.1
- username: root
- password: THE_ADMIN_PASSWORD_YOU_SET_BEFORE
- nvram erase
- reboot
Now for the last time set a username and password on http://192.168.1.1

DD-WRT is now installed and you can start configuring it. I mainly set the Wireless Regulatory Domain to Switzerland, configured wireless and port forwarding and enabled IPV6.

Fiber7 IPv6 settings

To get IPv6 with my provider working I enabled it here: http://192.168.1.1/IPV6.asp

IPv6: enable
IPv6 Type: DHCPv6 with Prefix Delegation
Prefix Length: 48
Static DNS 1: leave empty
Static DNS 2: leave empty
Remaining settings: default

After Save, Apply Settings and a Router reboot IPv6 was working on router and clients.

Further links

Cups printers do not show up in Firefox (Arch)

2017-01-02T00:00:00+01:00

If your CUPS printers do not show up in the Firefox printing dialog (but they do for example in Google Chrome) you may be missing the following package: gtk3-print-backends. Installing this solved that very annoying problem for me.

USB Keyboard not working at boot (between GRUB and disk decryption)

2016-12-18T00:00:00+01:00

On my thinkpad I have an encrypted Arch installation setup similarly to my chromebook setup. Since I bought my a new wireless usb keyboard this thing never worked for decrypting the system. It perfectly works once Arch has started up, but for entering the password at boot I had to use the notebook keyboard.

Add missing modules to initramsfs

Turns out my initramfs was missing the necessary modules.

Checkout the Arch wiki
Find the relevant keyboard modules. In my case it was hid_generic and usbhid
Edit /etc/mkinitcpio.conf and set MODULES="hid_generic usbhid" (or whatever your modules are)
mkinitcpio -p linux
Reboot

Gitlab CE setup

2016-11-02T00:00:00+01:00

The latest addition to my server at home is a Gitlab Community Edition instance. I use the docker image provided by Gitlab behind a nginx reverse proxy. Below are some notes for setting up, configuring and troubleshooting the installation.

Docker command

#!/bin/bash

sudo docker run --detach \
    --publish 11180:80 --publish 11122:22 \
    --name gitlab \
    --restart always \
    --volume /var/docker/volumes/gitlab/config:/etc/gitlab:Z \
    --volume /var/docker/volumes/gitlab/logs:/var/log/gitlab:Z \
    --volume /var/docker/volumes/gitlab/data:/var/opt/gitlab:Z \
    gitlab/gitlab-ce:latest

See run.sh

Gitlab config

Open a shell in the docker container (sudo docker exec -it gitlab /bin/bash) then edit /etc/gitlab/gitlab.rb. The following subsections show the parts I changed or added. After changing the config gitlab-ctl reconfigure can be run (still in the docker container) to restart gitlab.

external_url and relative root

The goal was to run Gitlab over SSL/TLS behind a reverse proxy. Instead of running Gitlab under / it should be accessible under /gitlab. This requires some extra effort:

external_url 'https://SOMEDOMAIN/gitlab' # default: http://hostname

(note httpS and /gitlab). Also setting RAILS_RELATIVE_URL_ROOT was necessary:

gitlab_rails['env'] = {
'RAILS_RELATIVE_URL_ROOT' => "/gitlab"
}

Email Configuration

gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "**.**.ch"
gitlab_rails['smtp_port'] = 465
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_ssl'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['smtp_openssl_verify_mode'] = 'peer'
gitlab_rails['smtp_user_name'] = "**@**.ch"
gitlab_rails['smtp_password'] = **
gitlab_rails['smtp_domain'] = **
gitlab_rails['gitlab_email_from'] = 'gitlab@**.ch'
gitlab_rails['gitlab_email_reply_to'] = 'noreply@**.ch'

Settings for running behind a reverse proxy

SSL is terminated at the reverse proxy. The communication between nginx and Gitlab will be HTTP.

nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on"
}

NGINX Reverse proxy configuration

The following snippet proxies requests to /gitlab on nginx back to the locally running Gitlab container.

server {
    .....

    location ^~ /gitlab/ {
      client_max_body_size 0;
      gzip off;
      ## https://github.com/gitlabhq/gitlabhq/issues/694
      ## Some requests take more than 30 seconds.
      proxy_read_timeout      300;
      proxy_connect_timeout   300;
      proxy_redirect          off;
      proxy_http_version 1.1;
      proxy_set_header    Host                $http_host;
      proxy_set_header    X-Real-IP           $remote_addr;
      proxy_set_header    X-Forwarded-Ssl     on;
      proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
      proxy_set_header    X-Forwarded-Proto   $scheme;
      proxy_pass http://127.0.0.1:11180;
    }
}

Assets not found

Since Gitlab is run under /gitlab it's possible that some fonts, icons and images cannot be found. Running the following commands inside the docker container should fix this:

gitlab-ctl reconfigure
NO_PRIVILEGE_DROP=true USE_DB=false gitlab-rake assets:clean assets:precompile
gitlab-ctl restart

Source

Ugly fonts in IntelliJ IDEA on Arch Linux

2016-10-11T00:00:00+02:00

UPDATE

intellij-jdk is now marked orphaned in AUR. I therefore stopped using this package and switched to a manual install of IntelliJ IDEA. This way the bundled jdk by Jetbrains is used automatically.

Old Post

Java fonts on Linux have been ugly for a while. Especially in an IDE it's very annoying. It seems Jetbrains started to ship a patched JDK with IntelliJ 2016. Make sure to use this JDK on Arch to run IntelliJ!

Install

Install intellij-jdk from AUR.

Configuration

Start IntelliJ
CTRL+SHIFT+A to enter action
enter "switch JDK"
chose "Switch IDE boot JDK"
point IntelliJ to /opt/intellij-jdk and restart

Notes

I only use this JDK for running the IDE. For development I'm still using the opendjk packages provided by Arch. Besides installing this new JDK I also force my java application to use anti-aliasing. To do so add the following to /etc/environment:

_JAVA_OPTIONS='-Dawt.useSystemAAFontSettings=on'

Atom editor notes

2016-08-11T00:00:00+02:00

Some notes on my atom editor setup. Will be expanded throughout time.

Plugins

apm install atom-runner
apm install linter
apm install linter-flake8
apm install linter-gcc

flake8

linter-flake8 needs a flake8 binary. We'll install this into a virtualenv used for atom.

mkvirtualenv atom
# make sure you're in the new virtualenv. then run:
pip install flake8
pip install flake8-docstrings

Modify Atom's env.PATH

Open Atom and go to Edit -> Init Script... Add the following line and make sure '/home/USERNAME/.virtualenvs/atom/bin' points to your atom virtualenv directory:

process.env.PATH = ['/home/USERNAME/.virtualenvs/atom/bin', process.env.PATH].join(':')

Settings

Go to Editor -> Preferences * enable Soft Tabs * set tab length to 4 spaces

Shortcuts

ALT+r run file using atom-runner
CTR+SHIFT+m show rendered markdown

Encrypted Ubuntu 14.04 LTS Installation (RAID, LLVM, LUKS)

2016-04-07T00:00:00+02:00

I meant to document the setup procedure of my home server for a long time here. Since I never got to write up an article (and probably neverl will) I'll just share my google drive notes here. Maybe you'll find something useful below or on the direct link.

Guided Static Analysis slides

2015-10-22T00:00:00+02:00

It took so much effort to create these slides so why keep them private? Here's my Research Topics in Software Engineering Talk about Guided Static Analysis by Denis Gopan and Thomas Reps:

The paper is available here.

Downloading a website for reading offline

2015-07-28T00:00:00+02:00

I needed a way to download all papers and exercises of one of my lectures. Here's a good wget snippet to mirror the website:

wget --mirror --convert-links --adjust-extension --page-requisites -np -nH -N --cut-dirs=1 http://www.ita.inf.ethz.ch/alscpr15/

Source: Tweet by @doublec (tweets are protected now)

Prevent Firefox from reloading pinned tabs at startup

2015-07-08T00:00:00+02:00

Firefox has a setting "Don't load tabs until selected". But enabling this will not prevent Firefox from reloading every pinned tab at startup. Fortunately there seems to be a specific preference for this one too:

open about:config
set browser.sessionstore.restore_pinned_tabs_on_demand to true

Arch Linux on Toshiba CB30-102 (Chromebook)

2015-06-21T00:00:00+02:00

The Beginners guide in the wiki is straightforward. Here are just some additional notes on setting up Arch Linux on my chromebook.

Planned setup

My device has a 16GB ssd. I'm planning to wipe ChromeOS and partition the disk into two partitions: 2GB /boot (ext4) and a 14GB encrypted partition (dm-crypt + LUKS). A 2GB swapfile will be created inside the encrypted partition.

Preparations

Check the Arch wiki for chrome device specific preparations. My device has an alternative boot loader so I just need to press CTR+L at the splash screen to boot from usb-stick or (later) GRUB.

Installation steps

Follow the beginners guide. After creating the two partitions and making the /boot partition bootable, proceed to setup the encrypted partions.

# assuming /dev/sda2 is the 14GB partition that will be encrypted
cryptsetup -v --cipher twofish-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random --verify-passphrase luksFormat /dev/sda2
# now open the LUKS partition
cryptsetup open --type luks /dev/sda2 root
# the decrypted partition is now available at /dev/mapper/root
mkfs.ext4 /dev/mapper/root
mount /dev/mapper/root /mnt
# proceed with installation according to the beginners guide

Check Device_encryption for encryptions settings etc.

Before the first reboot

Assuming GRUB is used as a boot loader we'll need to make sure that it finds the root device and that the initramfs contains everything necessary to decrypt the partition.

Edit /etc/default/grub and configure cryptdevice in GRUB_CMDLINE_LINUX_DEFAULT:

GRUB_CMDLINE_LINUX_DEFAULT="quiet cryptdevice=UUID=6364123f-3052-4745-bb79-ee60416b4309:root:allow-discards"

The UUID should in my case point to /dev/sda2. :root: will make the decrypted partition available at /dev/mapper/root and allow-discards is needed to make the discard mount-option work through disk-encryption. Note: allow-discards may be a security risk. I'm using it anyway. After configuring update the grub configuration with grub-mkconfig -o /boot/grub/grub.cfg

At last make sure to add encrypt to /etc/mkinitcpio.conf before the filesystems hook:

HOOKS="base udev autodetect modconf block encrypt filesystems keyboard fsck"

and update the initramfs:

mkinitcpio -k 4.0.5-1-ARCH  -g /boot/initramfs-linux.img

(the -k 4.0.5-1-ARCH argument was only needed in my case, because the installed kernel was newer than the one running on the usb-stick. Try it without -k and if it doesn't work, check /lib/modules/ for the correct kernel-version) You're now ready to exit chroot and reboot (do not forget to install all necessary modules and firmware for the network to work... check the wiki!).

Configuration

After the first reboot we should now have a working arch installation. I set up x11, gdm and cinnamon following the wiki instructions. GDM can be enabled at boot with systemctl enable gdm.service

ChromeOS Buttons

I used sxhkd to make the ChromeOS Buttons work.

~/.config/sxhkd/sxhkdrc

# Web browser Back/Forward shortcuts
{@F1,@F2}
  xte 'keydown Alt_L' 'key {Left,Right}' 'keyup Alt_L'

# Web browser Refresh shortcut
@F3
  xte 'keydown Control_L' 'key r' 'keyup Control_L'

# Awesome WM maximize current window
# Adjust as necessary for different window managers
#@F4
#  xte 'keydown Super_L' 'key m' 'keyup Super_L'

# Awesome WM move one desktop right
@F5
  xte 'keydown Super_L' 'key Right' 'keyup Super_L'

{F6,F7}
  xbacklight -{dec,inc} 10

F8
  amixer set Master toggle

{F9,F10}
  amixer set Master 5%{-,+} unmute

Then add /usr/bin/sxhkd to the autostart of your desktop env. For cinnamon I used gnome-session-properties (installable from AUR). You'll need alsa-utils, xorg-backlight, xautomation for the config above to work. The config above is a modified version from the wiki (% for F9, F10). Check Chrome_OS_devices#Sxhkd_configuration.

Swapfile

I use a swapfile inside the encrypted partition. Create a 2GB swapfile with:

dd if=/dev/zero of=/swapfile bs=1M count=2000
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

and enable it in /etc/fstab:

cat >> /etc/fstab
/swapfile   swap    swap    defaults    0 0

CTRL+D

Hibernation

For hibernate to work, the resume= and resume_offset= (only when a swapfile is used) kernel params must be set and the resume hook in initramfs must be enabled.

add

resume=/dev/mapper/root resume_offset=OFFSET_NO

to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub and update the grub config. OFFSET_NO is the first row of physical_offset in filefrag -v /swapfile.

Then add resume to HOOKS in /etc/mkinitcpio.conf after the encrypt hook (otherwise it would try to resume before decrypting the partition) and update initramfs with mkinitcpio -p linux

Hibernation should now work with systemctl hibernate.

AUR packages

Checkout yaourt. Quite useful for installing AUR packages.

OS X: Reinstall OS stuck at "2 minutes left"

2015-03-21T00:00:00+01:00

You probably have used homebrew. Haven't you? In the last step of the reinstallation the installer moves / checks files in /usr/local. If you have a lot of files over there (for example because you used homebrew), this takes forever.

Do not abort the installation. Press cmd+L to see the logs.

Using git for SVN repos

2015-03-21T00:00:00+01:00

We're forced to use SVN for some projects at the university. It's OK but when working on the go (Train) and offline it's a pain. Here are the steps for using git.

# initialize repo
git svn clone https://svn-server-address/repo/path

Now you can work and commit to your local git branches.

git checkout -b myFeature
# do some stuff
git commit -m "whatever"

To publish my new feature to svn I use the following steps:

# fetch the latest svn updates to my master branch
git checkout master
git svn rebase

# rebase my new feature to master
git checkout myFeature 
git rebase master

# merge my feature and force git to always make a merge commit
git checkout master
git merge myFeature --no-ff

# commit changes to svn
git svn dcommit

Yes, its dcommit and not commit.

x86 assembly on OSX

2015-03-21T00:00:00+01:00

During last terms compiler design course I learned x86 assembly "the hard way"... Here's some stuff that I "found out" while developing x86 assembly on OSX 10.10.

Read this first

I'm writing this post a couple of months after the course. I did not lookup the instructions and I'm justing writing what I can remember. It's therefore possible that some instructions below are incorrect. Better look them up yourself instead of copying from here...

Tools

I used gcc (compiler/assembler) and gdb (debugger). Place your assembly code in a .s file and assemble it with gcc -m32 -ggdb yoursourcefile.s -o programname. gcc is useful to have a look at generated assembly code too. Just write a c programm and then compile it with the -S flag to generate assembly output.

Stack alignment on OSX

There's this sentence

The stack is 16-byte aligned at the point of function calls

in the IA-32 Function Calling Conventions that gave us in the beginning so much pain. What does it mean? Before calling anything make sure that the stack is aligned to 16 bytes. The annoying thing is, that inside each function there are already four bytes on the stack (the return-address). After the normal function prologue (enter / push %ebp) there will be 8 bytes on the stack.

Example

.globl _main

_main:
    enter $8, $0                     
    # stack is now 16 byte aligned. 
    # 4 + 4 + 8
    # return-address + old %ebp value + 8 bytes because of "enter $8.."

    leave ret

So, make sure to always increase the stack size by a multiple of 16. The hard part was, that the alignment is only enforced when using systemcalls. In the beginning our programs jumped around a couple of times with a misaligned stack without any problems and then just blew up when printing or reading something. Quite hard to debug but very educational... (learn the hard way indeed).

Some tricks

Below are just some tricks I learned. If you've programmed assembly before you'll be bored.

; inside your assembly func you should backup the ebp
; either do
push %ebp 
movl %esp, %ebp

; or use the enter instruction:
enter $0, $0

; after the steps above there will be 8 bytes on the stack.
; align it with
subl $8, %esp
; or use enter $8,$0 

; because we set %ebp as the first step, we can now use it to 
; address stuff in our stack frame. 
; some examples:
movl $4, (%ebp)    ; write 4 into the stack address where %ebp points to 
movl $5, -4(%ebp)  ; write 5 into the address four bytes below
movl 8(%ebp), -8(%ebp) ; move the value of a parameter (higher stack address) into a local variable (lower stack address)

; to create local variables, just make space by reducing the esp
subl $4, %esp     ; create a 4 byte "variable" on the stack 
movl $42, 4(%esp) ; write something to this variable
addl $4, %esp     ; free/forget the variable

; instead of subl %esp you can also: 
pushl $42         ; decrease %esp by 4
popl %eax         ; read the previously pushed $42 to %eax and increase the stack pointer again

; set eax to zero (eax is used as the return value when exiting the program)
xor %eax, %eax

; no example but a tipp: checkout the leal instruction! it is useful for example when using scanf and writing to the stack

; there are instructions for leaving your function
leave
ret

Run IntelliJ IDEA 14 on OSX with Java 8

2015-02-21T00:00:00+01:00

Starting with Yosemite Apple finally stopped shipping Java 6 with OSX. It's recommended to install the latest Java version directly from Oracle. Unfortunately IntelliJ IDEA needs Java 6 (!) to start. Here is the recommended way to run IntelliJ IDEA with Java 8.

# first, install IntelliJ to /Applications
mkdir ~/Library/Preferences/IntelliJIdea14/
cp /Applications/IntelliJ\ IDEA\ 14.app/Contents/bin/idea.properties ~/Library/Preferences/IntelliJIdea14/
# now modify ~/Library/Preferences/IntelliJIdea14/idea.properties and change the JVMVersion from 1.6* to 1.6+

The steps for WebStorm and PyCharm are similar and you can find the directory names here.

In my case the directories were:

~/Library/Preferences/WebStorm9
~/Library/Preferences/PyCharm40

Move/Copy mails between different IMAP accounts

2014-11-15T00:00:00+01:00

I decided to switch back to Gmail/Inbox by Gmail. Copying thousands of mails in Thunderbird is a pain and after a couple of unsuccessful tries I started to search a better working alternative. mutt to the rescue!

# login to your old imap account:
mutt -f imaps://username@oldmailserver/INBOX/folder
# mark all messages in that folder (otherwise use "t" to select single messages)
enter "T" and then "~A" 
# either copy (;C) or move (;s) the messages. ;s will mark them as deleted on the old account!
enter ";C"
# as a target directory choose your gmail account:
imaps://gmailusername@imap.googlemail.com/[Gmail]/folder

You may want to start mutt in a "screen" session and then let it run for a while. On linux systems mutt should already be installed. On OSX you can find it in homebrew.

Manually installing Boost 1.55 on OSX

2014-10-19T00:00:00+02:00

Boost 1.56.0 which is currently installed by homebrew is broken. Just try to compile the following C++ program. It will fail.

#include <boost/graph/adjacency_matrix.hpp>

int main() {
}

Here are the steps to manually install Boost 1.55:

Download Boost 1.55 here
Extract the source and cd into the directory
Compile: ./bootstrap.sh && ./b2
Create a directory where Boost will be installed: mkdir /usr/share/boost
Install: ./b2 install --prefix=/usr/share/boost/
Link the include files to /usr/local/include: ln -s /usr/share/boost/include/boost /usr/local/include

You might need administrator privileges for the last three steps.

Enable homebrew for non admin users

2014-10-18T00:00:00+02:00

Even on OSX you should not be working with an admin account. Here are the steps to make homebrew usable for non admin users.

Assumptions

I assume brew has already been installed by an administrator and it's login is admin. Change admin below to the name of your admin account.

Steps

Under System Preferences create a new group "brew" and add your user to it. Then open the terminal and:

su - admin
sudo chgrp -R brew /usr/local
sudo chgrp -R brew /Library/Caches/Homebrew
sudo chmod -R g+w /usr/local
sudo chmod -R g+w /Library/Caches/Homebrew

# and if you plan to use caskroom too:
sudo mkdir -p /opt/homebrew-cask/Caskroom
sudo chgrp -R brew /opt/homebrew-cask
sudo chmod -R g+w /opt/homebrew-cask

(this logs in as the admin user and allows the members of the "brew" group to write to /usr/local and the Caches directory)

With your user account you can now run brew doctor to check if everything is working as it should.

Credits: Visit this for some more detailed instructions.

OSX 10.10 (yosemite): configure: error: C compiler cannot create executables

2014-10-18T00:00:00+02:00

Problem: brew fails with "configure: error: C compiler cannot create executables"

==> Upgrading 1 outdated package, with result:
python3 3.4.2_1
==> Upgrading python3
==> Downloading https://www.python.org/ftp/python/3.4.2/Python-3.4.2.tar.xz
Already downloaded: /Library/Caches/Homebrew/python3-3.4.2.tar.xz
==> ./configure --prefix=/usr/local/Cellar/python3/3.4.2_1 --enable-ipv6 --datar
checking for gcc... clang
checking whether the C compiler works... no
configure: error: in /private/tmp/python3-av3R7L/Python-3.4.2:
configure: error: C compiler cannot create executables
See config.log for more details

Solution: Download and install Xcode 6.1 and the Command line tools (CLT) for OSX 10.10. With the release of Yosemite Xcode 6.1 should show up in the App Store sooner or later. Then a manual download will be unnecessary.

Hello

2014-07-17T00:00:00+02:00

Welcome to the new website. I've created the new page mainly as an excuse for not learning during the current exam preparations... So you should not expect any updates soon. There'll hopefully be some updated stuff here after the exam session in august. Have a nice summer and come back later! ;-)