So a little while back, I switched all my email over to G Suite instead of hosting my own solution (I was paying for a VPS to do this as well, which was more costly). Eventually, you do get to a point where you realise that someone else can do something better than you and your time is more valuably spent doing other things. I was also routing a large amount of my email via Gmail for public usage anyway as it was easier to explain over the phone than my domain email address (possibly just a bad choice on my part for this one xD).

G Suite has a few advantages that replicating myself would be quite costly and/or time consuming:

• More Highly available: Getting a multi-master setup to be reliable with Dovecot over the internet was a bit of a time investment and I was never convinced it worked properly. I would also have to replicate the various frontends.
• Tagging & Search: This was quite a big one. It’s so much easier to tag and search in Gmail than set up the infrastructure required to do it myself and make it highly available and performant.
• Unmanaged: I would no longer have to debug and look after the infrastructure myself. Email can be fussy at times just trying to avoid spam filters. = Calendar & Contacts Support: Whilst I had this working with CalDAV and CardDAV via Davical, integrating this every time with various plugins was a bit tedious and didn’t work greatly when it came to tying it into an interface. The HA side of things comes into play here as I never set it up to be HA.

Some of the other nice features that made it easy to use were the following:

• Unlimited Domains: This was quite nice as I could add as many domains as I wanted and redirect emails to one user with many aliases, much as I had been doing with the Open Source stack I was using. I retained delimiter support in emails, so migrating across was pretty much seamless.
• Storage: With G Suite Business (see my previous article for more details), I have pretty much unlimited storage on Drive now.

To be honest, it’s just pretty easy to use, which also makes it quite appealing.

In the end, the main reason I chose it would be the custom domain support and ability to retain + delimiter support in emails.

Background

The gist of this is that over time, I’ve used various backup/mirroring tools to take copies of my data and store them in the Cloud. Over time, I’ve used a few storage providers such as:

• Google Drive
• OVH Dedicated Servers
• VPS solutions
• OneDrive
• Amazon Cloud Drive

As I’m an avid photographer, this data is actually growing at a pretty decent rate due to shooting in RAW. When an image can be 50MB+, the storage tends to add up pretty quickly!

Amazon Cloud Drive and the rclone fiasco

For a while, I decided that making use of the unlimited photo storage that Amazon Cloud Drive offers as part of its’ Amazon Prime membership seemed like a good idea and I started to make use of this with the rclone tool which I’ve been using with good success for a while now. All was well and good.

At some point in 2017 (I can’t remember when), Amazon decided to revoke the keys that rclone was using for access. This became a major problem for me as I back up from a NAS box running Gentoo Linux (with the data stored in RAID1). A workaround was soon developed using an OAUTH proxy run on Google’s App Engine and things worked briefly. Eventually I ran into rate-limiting issues which I haven’t been able to resolve and thus, ended up stuck.

I even tried running a backup tool (SyncBack SE Pro) from my desktop, to copy the files on the NAS to ACD. It decided however that on every single run, files had changed. I didn’t really fancy syncing 700GB~ of photo data every single time I wanted to take a copy and this started to seem like more and more effort for something that should really be a painless process.

Finding something better

At this point, I’d pretty much had enough and decided that if I wanted something painless and that would tick along, I’d have to find another solution to the problem and this might involve some form of monthly spending fee. I won’t go into huge amounts of detail but here are the thoughts on solutions I came across.

Microsoft Onedrive (as part of an Office 365 subscription)

Disclaimer: I already use this for a lot of things such as: music storage, encrypted backups of smaller data, document storage and sharing etc.

I’m actually a big fan and it works pretty well for me. I’ve even got a weird setup where it syncs my Lightroom Catalog files so that I can edit the previews on the move (it’s got a few quirks but works pretty much seamlessly!).

Pros

• £46.50 for the year for an Office 365 sub from Amazon (£3.90~ per month).
• 1TB storage per user.
• Desktop synchronization client.
• API support (works with rclone, duplicity etc).
• Office Desktop & mobile license is included and use of all the web applications as well.
• £68 per year (£5.70~ per month) for the 5 user subscription from Amazon. Apparently, each user can have 1TB of OneDrive storage.

Cons

• You can’t really have more than 1TB of storage per user as a consumer, which rules it out for using as photo storage as well at this point.
• Additional setup required to use multiple accounts. I also don’t know how easy it is to move from the 1 to 5 user subscription and and pre-requisites that might have.

Google Drive (Consumer Edition)

So, I’ve used this before and it was pretty good but as listed below, cost was a bit much. A second Office 365 subscription for backups would be half the cost (at the expense of being a bit of a pain to scale).

Pros

• API Support (same as OneDrive)
• Transfer speeds are good
• Variable storage limits depending on how much you need

Cons

• The desktop applications are undergoing a bit of a transition which I haven’t quite got to grips with yet. They removed the original sync app and replaced it with two others. As I haven’t used that in a while, it might be fine, but something to be aware of.
• £7.99 a month per TB (over twice the cost of an Office 365 subscription. The limit is going up to 2TB soon though, so it will be more competitive if you’re just looking for storage.)

Dropbox Plus

I won’t go into too much detail on this one but basically the 1TB is the same price as Google Drive. This pretty much has the same pros/cons as Google Drive (Consumer Edition) apart from the fact that when I have used it in the past, the Desktop Application was pretty slick compared to other providers I’ve used. If you are swayed by the previous but want better desktop support, Dropbox is definitely a contender!

AWS S3/Glacier

Given that this is run by an entirely different Amazon division and is Production grade storage, it’s worth including to see how this pans out. That said, retrieval cost pretty much rules out this option for consumer usage.

Pros

• Production grade, various policies for availability and durability, all of them being 99% or higher.
• Glacier costs for primarily storage are reasonable per GB ($0.004-$0.0045 per GB).
• S3 API support is an industry standard for a lot of tools, so lots of support.

Cons

• Costs vary per region per GB stored.
• API calls and bandwidth are calculated as separate costs.
• Retrieval from Glacier (which is the only price-competitive option), can be very costly.

Backblaze B2

I very much almost went for this, as it was quite an appealing option. Note, this is the professional, not the consumer option which just backs up everything on your PC.

Pros

• Good API support, with a multitude of clients available (including rclone).
• Price per GB ($0.005) is good. Not as good as Glacier but you don’t have a myriad of costs. You pay a small amount for putting data in and then just pay for the data at rest. Deletes are mostly free unless you do a huge amount of transactions (and even the transaction cost is reasonable).
• Dedicated subreddit with responses from the people who run Backblaze
• If you need to restore a lot, you can pay to get a drive with all your data shipped to you, rather than trying to download it all.

Cons

• Limited amount of data centers (I think 2 at time of writing?)
• Daytime outage when attempting to provision support for more clients. With the amount of data they hold, I was expecting somewhat better management when I read about the recent incident, with regards to scaling.
• There’s a restore cost for getting data out, varying by method.

Google Drive for Business (as part of G Suite)

So, this is an interesting consideration, depending on number of users (< 5 or 5+), you get slightly different packages. Also, there’s a difference between G Suite Basic (£3.30 pcm/user) vs G Suite Business (£6.60 pcm/user).

Here, we’ll evaluate the Business edition, as the Basic doesn’t give you more than 30GB of storage.

Pros

• 1TB per user (<5 users) or unlimited (5+ users). The big pro here is that, according to reddit users, the 1TB limit isn’t enforced actively. This means you effectively can have unlimited storage with less than 5 users on the account.
• Plenty of API support for Google Drive (see above)

Cons

• If you can consider this a con, there’s a 750GB upload limit daily and there’s a multi terabyte download limit daily.
• At any point, if you’re using with less than 5 users, in theory the 1TB limit of storage could be enforced.
• Same issue as Google Drive for consumers with desktop applications

Overall Solution

Winner: Google Drive for Business (G Suite Business)

To start with, I already use OneDrive as part of the Office 365 subscription due to the price and storage space and the fact I get Office licenses and web apps all together. As much as Google pushes its’ own productivity suite, the Microsoft suite is a major preference for me.

I was also a G Suite Basic subscriber, therefore I have all my email etc hosted with Google already and paying £3.30 per month for the privilege of high availability and all the other integration features. This made my choice a no-brainer really.

• For the £3.30 per month upgrade to G Suite Business, I could use the Google Drive storage. It gives me 1TB (although in practice, it’s unlimited, so it could scale forever).
• The speeds are good.
• I don’t need to create yet another user account to manage.

If in future I hit the storage limit and it becomes enforced, I might look back into the Office 365 5 User Family subscription and start looking at spreading my data across multiple accounts for cost efficiency. Until then though, this was a fairly easy solution to put in place (a few button clicks to upgrade my subscription and an OAUTH signin with rclone and I was good to go and start syncing!).

Installation

Gentoo

• Enable sambakr5passwd and samba USE flags for OpenLDAP

• Enable ldap USE flag for Samba

  1. USE="smbkrb5passwd samba" emerge openldap
  2. USE="ldap" emerge samba
  3. emerge net-nds/smbldap-tools

Configuration

1. Edit /etc/openldap/slapd.conf to include the following:

    include /etc/openldap/schema/samba.schema
   
    access to attrs=userPassword,sambaLMPassword,sambaNTPassword,shadowLastChange,sambaPwdLastSet,sambaPwdMustChange
    	by dn="cn=Samba Admin,ou=SystemAccounts,dc=canet,dc=me,dc=uk" write
   
    access to *
    	by dn="cn=Samba Admin,ou=SystemAccounts,dc=canet,dc=me,dc=uk" write  This grants the `Samba Admin` account full access to LDAP once authenticated. 
   

2. We need to add an OU for the Samba required entries as well as the Samba Admin account. To do this, add to the LDIF data the following (use slapcat > myldif.ldif to generate an LDIF file including all the current data):

    dn: ou=SystemAccounts,dc=canet,dc=me,dc=uk
    objectClass: organizationalUnit
    objectClass: top
    objectClass: domainRelatedObject
    associatedDomain: canet.co.uk
    ou: SystemAccounts
   
    dn: cn=Samba Admin,ou=SystemAccounts,dc=canet,dc=me,dc=uk
    givenName: Samba
    sn: Admin
    uid: smbadmin
    cn: Samba Admin
    userPassword: {SSHA}eXKqB5iVZnu+sValdI3Omqbyvt1el6c5
    objectClass: inetOrgPerson
    objectClass: top
   

3. Load the LDIF changes into LDAP with the following (you are free to add the data into LDAP any other way you wish as long as it gets there!)

    rm -rf /var/lib/openldap*
    slapadd -l myldif.ldif
    chown -R ldap:ldap /var/lib/openldap/
   

4. Edit the Samba config to reflect the following so it knows how to connect to the LDAP server (/etc/samba/smb.conf):

    # LDAP server location
    passdb backend = ldapsam:ldap://127.0.0.1:389/
   	
    # The Base DN
    ldap suffix = dc=canet,dc=me,dc=uk
   	
    # The LDAP path to computer, user and group account OUs
    ldap machine suffix = ou=Computers
    ldap user suffix = ou=People
    ldap group suffix = ou=Groups
   	
    # Samba Admin account in LDAP to use for Administrative tasks
    ldap admin dn = cn=Samba Admin,ou=SystemAccounts,dc=canet,dc=me,dc=uk
   	
    # Not sure what this is for
    ldap delete dn = no
      
    # This is for password synchronisation between the Unix and the Samba password in LDAP. 
    # So if you change your Samba password over smbpasswd or Windows this option changes your Unix/Linux password too.
    ldap password sync = yes
   

5. Run smbpasswd -W to update local secret for accessing the Samba Admin account.

6. Run smbldap-config
   • Set . where required to generate blank fields.
   • Set password validation time to 99999
7. Run smbldap-populate -e server.ldif. This generates an LDIF file with all the required LDAP users and groups.
   1. Remove entries that already exist and remove the root and nobody accounts. Make sure your Groups OU is set correctly!
   2. Make sure it picks up the correct domain. It picked up my workgroup instead (as it was previously something else). If you run net getlocalsid it will give you the domain in capitals.

8. Load the Samba accounts with ldapadd -x -D "cn=Manager,dc=canet,dc=me,dc=uk" -f server.ldif -W (this uses the LDAP Admin account to update from the LDIF file)

9. Once done, we need to update existing ldap entries with SAM entries so need to run:

    smbldap-usermod -a christakis  WARNING: If the ACLs in `slapd.conf` are too restrictive, this WON'T work
   

10. smbpasswd christakis should then update both LDAP and Samba passwords as we’ve set Samba to sync them with ldap password sync = yes in the configuration file earlier.
    • At this point, if we update the LDAP password without smbpasswd, it won’t update the Samba password!!

Enable smbk5pwd Overlay

This overlay enables the updating of Kerberos and Samba passwords via an LDAP overlay whenever the LDAP password is changed via LDAP. Following the instructions for initial installation, the relevant library should be installed so it should only need configuration

1. Edit /etc/openldap/slapd.conf:

    moduleload /usr/lib64/openldap/openldap/smbk5pwd.so
   	
    #
    # More configuration…
    #
    # Database definition…
    #
   	
    overlay smbk5pwd
    smbk5pwd-enable samba
    smbk5pwd-must-change 0
   

   This enables the overlay in LDAP to propegate the changes to LDAP

2. Edit /etc/samba/smb.conf:

    ldap password sync = only
   

   This tells Samba to only update the LDAP password and let the LDAP server do the rest (so use the new overlay!)

Google Drive

It was at this point when Google Drive had recently announced 1TB of storage space for $9.99 a month making it a very cheap storage solution so I decided to do a bit more research into it to find out the benefits and drawbacks.

Benefits

• Cheap at $9.99 per TB (with UK tax at 20% and the conversion rate, it is about £7 a month)
• Quick to access with apps for Windows, OSX, Android and iOS and a brilliant webapp
• Has a couple of music player addons to stream music from your browser as well as dedicated Android apps for music streaming with Drive as a storage backend
• Should be as redundant as Google’s storage for other services with nearly no downtime

Drawbacks

• Mainly API based access for 3rd parties
• No Linux server clients for syncing (which is a bit of a problem given my Gentoo-based storage server)
• Music playback apps leave something to be desired regarding interface and usability

gdrsync

This looked like a good option for me except for the lack of Linux support so I did a bit of digging around and came across a tool called gdrsync. This attempts to emulate the rsync tool with the explicit intention of use against Google Drive. The only problem with the original version of this was that it didn’t support downloading the data back down. Luckily I found a fork here which does support it. Using the following with it preconfigured in the instructions in the README, I can sync any set of files to a given path in Google Drive from my Linux server.

gdrsync -rdvs /path/to/files/ gdrive:/path/to/files/

It’s worth noting you need to set up a development account and acquire a key and ID to go with this from Google but it’s a fairly straightforward process and doesn’t cost anything. It will work with the default configuration from the original developer but if more and more people are using, it will start running into issues with API calls so I suggest getting your own set up.

dsync is primarily worth using because it uses dovecot’s log files to replicate as mail is received so filesystem corruption isn’t replicated and it also avoids having to script and control synchronization between the two instances. Because it uses the log files, it can also run master-master and whilst the official documentation recommends a user always goes to the same host, it also states that the worse case scenario is a user has to redownload mails again. dsync will also sync your sieve configurations if you have sieve set up!

It is worth noting that this is a bit of a pain to set up, so be prepared! This guide was also written based on the Gentoo installation configurations.

Dovecot Services Used

• doveadm - Front facing TCP listening service providing access to aggregator and replicator I believe
• aggregator - I believe this is used to provided notifications to the replicator process when there’s changes to synchronize
• replicator - Triggers/handles synchronization

Pre-requisites

• doveadm user '*' should return a list of all the users dovecot hosts. I found this needed to be a complete list in email format for it to replicate properly (this is possibly in part due to me organizing my mail folders with folders for domain followed by folders for users underneath)
• Optional - SSL Configured as per the Dovecot Documentation on SSL. This is only required if you wish to use synchronization over SSL (and using Gentoo, you need to have dovecot compiled with USE="ssl".
• Optional (probably…) Mailbox format in mdbox or sdbox. I’m not sure where I read this but with a large amount of mail, the maildir format is expensive to sync in comparison and may have more issues. If I find the documentation on this, I’ll add it in.

Configuration (On both hosts)

1. /etc/dovecot/conf.d/10-mail.conf

    # Enable globally the notify and replication plugins
    	# This will then apply to all protocols dovecot supports
    mail_plugins = notify replication
   

• /etc/dovecot/conf.d/30-dsync.conf - You’ll need to create this manually

    # This sets globally the port needed to connect
    # to the doveadm service
    doveadm_port = 1234
      
    # Set the password globally to use for connecting to 
    # and requiring for the doveadm service 
    doveadm_password = password
      
    # Set the directory for validating certificates
    # provided when running with SSL enabled
    # Should be the standard OpenSSL certificate directory by default
    ssl_client_ca_dir = /etc/ssl/certs
      
    # Configure the doveadm service to listen
    service doveadm {
    	inet_listener {
        	# port to listen on
        	port = 1234
            # enable SSL
            ssl = yes
        }
    }
      
    # Configure the aggregator service for notifications
    service aggregator {
    	fifo_listener replication-notify-fifo {
        	# Your mail user that's managing files generally is used here
        	user = mail
        }
        unix_listener replication-notify {
        	user = mail
        }
    }
      
    # Configure the replicator service
    service replicator {
    	process_min_avail = 1
        unix_listener replicator-doveadm {
        	mode = 0666
        }
    }
      
    # Configure target hosts for replication
    # tcps can be tcp if you don't want to connect with SSL
    # :port can be omitted if it's the default set globally for doveadm
    plugin {
    	mail_replica = tcps:fqdn:port
    }
  

• Restart dovecot

Notes on SSL

• By default, doveadm will use the certificates set globally for ssl_key and ssl_cert if you tell it to listen on SSL. It does not support SNI like the imap and pop3 listeners nor does dovecot when it attempts to connect to the other instance.
• Unlike the imap and pop3 listeners, doveadm does not supply all certificates in the chain, even if they are in the certificate file. This means that verifying is left down to the certificates in /etc/ssl/certs. I found that the intermediate certificate for StartCom was missing from there and needed adding. After adding, you need to run c_rehash to update hashes in the directory that are used by OpenSSL for lookup.

Useful Commands

• doveadm replicator replicate <email> - Replicate a given email account manually
• doveadm replicator replicate -f <email> - Replicate a given email account manually IN FULL
• doveadm replicator status <email> - Check replication status. Also works without the email parameter

mdbox stores messages even after they are completely deleted and expunged. This is done by decremented a variable called refcount on the message and when there is finally no more references for it, the value should be set to 0. Even after this though, the message is still stored in the file. When syncing however over dsync, these messages are not replicated as it is specific to the file format.

Apparently, during quiet periods of the day, it is recommended to schedule a maintenance command via cron for purging these messages completely. This can be done as follows:

# To purge a single user's mails where refcount=0
doveadm purge -u <user>

# To purge all users' mails where refcount=0
doveadm purge -A

For the latter to succeed, doveadm user '*' must successfully return all the correct usernames for deletion. This of course is now running on both hosts as a nightly cron job to keep things tidy.

My initial issues manifested whilst trying to play videos on my PS3 via streaming, something that usually isn’t a problem. Having upgraded PS3 Media Server recently and using a video I wasn’t sure about, I gave up trying to fix it at the time. What worried me is that when I was trying to access files via a Samba share on the same server, it would start to play briefly then die, even on videos that used to work fine!

So, I take to the shell to try and diagnose the issues!

• Nothing in the Samba logs to indicate what’s going on.
• ifconfig decides to tell me that there’s no packet loss on the Gigabit interface at all

All I get is this slightly obscure message in syslog from the kernel

Jan 27 00:07:56 rinoa kernel: [  584.495023] r8169 0000:01:06.0 eth1: link down
Jan 27 00:07:59 rinoa kernel: [  586.852168] r8169 0000:01:06.0 eth1: link up
Jan	27 00:08:22 rinoa kernel: [  609.994358] r8169 0000:01:06.0 eth1: link down
Jan 27 00:08:24 rinoa kernel: [  612.328398] r8169 0000:01:06.0 eth1: link up
Jan 27 00:08:44 rinoa kernel: [  632.046656] r8169 0000:01:06.0 eth1: link down
Jan 27 00:08:46 rinoa kernel: [  634.587853] r8169 0000:01:06.0 eth1: link up

Why is the interface going up and down like this? Google didn’t really have a good answer for me. Investigation led me to replacing the ethernet cable, which just seemed to make the situation worse. In vain effort, I also tried updating my Linux kernel to the latest stable to see if that resolved the issue. I settled in the end for reverting to the 100mbit internal NIC to see if that fared any better. Despite the downgrade in speed, it’s been 100% reliable.

So, if you see this and you’ve tested your switch and ethernet cable, your NIC is pretty much FUBAR. Just wish this info was available more easily!

Ingredients

Chicken

• seasoned flour, for dusting
• 1 egg, beaten
• 100g natural dried breadcrumbs (if you can get the Japanese panko breadcrumbs then use those!)
• 4 large skinless, boneless chicken thighs

Curry Sauce

• 1 onion, chopped
• 2 garlic cloves, crushed
• a small chunk ginger, peeled and finely chopped
• a pinch chilli flakes (optional)
• oil, for frying
• 1 tbsp plain flour, mixed with 1 tbsp mild curry powder
• 350ml chicken stock
• 1 tsp brown sugar (any kind will do)

Rice

• 100g basmati rice
• 75g frozen peas
• sesame oil
• soy sauce

Method

1. Put the flour, egg and breadcrumbs in separate dishes. Dust the chicken in flour, then the egg, then breadcrumbs. Chill for 20 minutes.

2. To make the curry sauce cook the onion, garlic and ginger (and chilli flakes if using) in a little splash of oil until softened. Add the flour and curry powder mix, and cook for 2-3 minutes, stirring to stop it catching. Slowly add the chicken stock then stir in the sugar and simmer for 15 minutes. Blend the sauce to smooth using a stick blender or food processor.

3. Heat the oven to 180C/fan 160C/gas 4. Heat 2 tbsp oil in a non-stick frying pan. Fry the chicken until golden on both sides then transfer to a rack and finish cooking in the oven for 20 minutes.

4. Boil the rice in lots of salted water, adding the peas for the last 2 minutes of cooking. Drain well then toss with a couple of drops of sesame oil and soy.

5. Slice the chicken and arrange on top of the rice then serve with a drizzle of the reheated sauce.