Category: projects

Categories
projects robots

Project Hexapod Part 3: No plan survives contact with the enemy

Design change 1:

Originally, I was going to control the hexapod from an Arduino pro mini.  Partly because that’s how I originally started the project.  But also, because I’d heard ropey things about raspberry pi’s and Python controlling servo’s in real time.  I was however planning to use a raspberry pi zero-w as a master.  The control system would connect to the Bluetooth adapter in the PI, some of the more complex calculations could be done on the PI, and then the PI would send messages to the Arduino which was in charge of moving the legs.

When I’ve seen people on YouTube do this (Tom Stanton, James Bruton), they often take this approach.  But that’s partly because they seem to have a remote-controlled background and are using devices like RC car controllers or drone controllers (and the associated electronics on the robot end).  I don’t have those devices and don’t have that experience. 

After playing with the Arduino code a bit, I’m starting to think that idea is a pain in the ass.  There’s the physical weight of the Arduino, but also the additional power draw it will take.  The calibration of the servo’s and the interfacing between the pi and the Arduino seems like extra work.  And configuring the servo’s end stops on the Arduino seems like a massive kludge. 

Most of my coding experience is with mobile apps.  My thought process was to write a smart phone app to connect to the device over Bluetooth.  Therefore, I’m not going to be going down the physical handheld controller route.

In addition, I have more experience with UNIX based systems.  So once the remote is walking, I’m more interested in doing client / server / web stuff where the robot is interacting with both it’s environment but also the web.  Whilst I have very little experience in Python, I have more experience in that kind of UNIX based scripting language than the C based Arduino language.  I’m not a natural developer so I need to minimise the number of language’s I’m learning.  I expect I’ll have more usage out of Python from a professional and personal projects perspective

For all of that, I’m now planning on ditching the Arduino and seeing just how difficult it is to get the pi zero-w to control the servo’s using the PCA985 PWM controllers over I2C.

Categories
projects robots Uncategorized

Project Hexapod Part 2: More walking design thinking

Each leg on this robot has 3 motors; a shoulder, an elbow, and an ankle

To make things easier I labelled them up like this:

There are 18 separate motors to control:

BoardServoBoardServo
LeftFrontShoulderPwm18RightFrontShoulderPwm17
LeftFrontElbowPwm110RightFrontElbowPwm15
LeftFrontAnklePwm113RightFrontAnklePwm12
  
LeftMiddleShoulderPwm19RightMiddleShoulder136
LeftMiddleElbowPwm114RightMiddleElbow141
LeftMiddleAnklePwm115RightMiddleAnkle150
  
LeftBackShoulderPwm28RightBackShoulderPwm27
LeftBackElbowPwm29RightBackElbowPwm26
LeftBackAnklePwm210RightBackAnklePwm25

Design Thoughts

First thought was to create a function that controls each limb.  At the least every limb should move in the same way.  However, whilst limbs move together from the gait perspective (Figure 2) I don’t think we can do that from an Arduino perspective.  I don’t think I can create a function that says move limbs L1, R2, L3 at the same time.  I think I’ll need a function that does that.  But then separately what it’s doing is moving “Left Front Shoulder, Right Middle Shoulder, Left       Back Shoulder” together, then “Left Front Elbow, Right Middle Elbow, Left Back Elbow”, etc.  until that function is complete.

I think I’ll have a multidimensional array with degrees in it for each joint.  What I might do though is write a routine which sets each joint to 0.  Then unscrew and reattach the joints so I know where zero is on each motor.  Either way, I probably need a few lists to define:

the start and end position of each joint for a forward motion

the start and end position of each joint for a circular motion

the start and end position of each joint for being sat still.

That currently looks like this:

NameBoardServoservo_minservo_max
LeftFrontShoulderPwm18500500
LeftFrontElbowPwm110  
LeftFrontAnklePwm113  
LeftMiddleShoulderPwm19  
LeftMiddleElbowPwm114  
LeftMiddleAnklePwm115  
LeftBackShoulderPwm28  
LeftBackElbowPwm29  
LeftBackAnklePwm210  
RightFrontShoulderPwm17  
RightFrontElbowPwm15  
RightFrontAnklePwm12  
RightMiddleShoulder136  
RightMiddleElbow141  
RightMiddleAnkle150  
RightBackShoulderPwm27  
RightBackElbowPwm26  
RightBackAnklePwm25  

Maybe a piece of code that takes serial input and resets the device to zero’s.

Control System:

Due to its built in Bluetooth, I’m thinking about having a raspberry pi zero-w connected to an Arduino nano over USB Serial:

https://maker.pro/raspberry-pi/tutorial/how-to-connect-and-interface-raspberry-pi-with-arduino

ser=serial.Serial(“dev/ttyACM0”,9600)

Then I can both ssh into the robot to tell it what to do.  And I can also write a phone app to act like a remote control, connecting over Bluetooth.  The v5 of the robot car did that.

Categories
robots

Project Hexapod Part 1: Walking Design

The design of the hexapod looks like this.  Each “limb” comprises of a shoulder, elbow and a foot.  The shoulder moves in a horizontal plane, left and right.  The elbow and feet move in vertical plans, up and down.

To control the hexapod, I need to consider stability. My current plan is to move three legs at a time. Something like this

This is an example gait I’ve found online.  It resembles how an ant walks.  So either side always has at least one foot on the ground and the robot should be balanced.  And this should be quicker (and easier) than moving each foot at a time. 

#Problem – how do I make walking asynchronous?  If I have a single function to move a limb, how do I move two limbs at once?

This is an example gait I’ve found online.  It resembles how an ant walks.  So either side always has at least one foot on the ground and the robot should be balanced.  And this should be quicker (and easier) than moving each foot at a time. 

#Problem – how do I make walking asynchronous?  If I have a single function to move a limb, how do I move two limbs at once?

#Answer


/* Sweep
 by BARRAGAN <http://barraganstudio.com>
 This example code is in the public domain.

 modified 8 Nov 2013
 by Scott Fitzgerald
 http://www.arduino.cc/en/Tutorial/Sweep
*/

#include <Servo.h>
 
Servo servo1;  // create servo object to control a servo
Servo servo2;  // create servo object to control a servo
                // twelve servo objects can be created on most boards
 
int pos = 90;    // variable to store the servo position
 
void setup()
{
  servo1.attach(8);  // attaches the servo on pin 9 to the servo object
  servo2.attach(7);  // attaches the servo on pin 9 to the servo object
}
 
void loop()
{
  for(pos = 90; pos >= 19; pos --){
  servo1.write(pos);
  servo2.write(180-pos);
  }
  //delay(5000);
  for(pos = 19; pos <= 90; pos ++)     // goes from 19 degrees to 90 degrees
  {                               
    servo1.write(pos);              // tell servo to go to position in variable 'pos'
    servo2.write(180-pos);              // tell servo to go to position in variable 'pos'
    delay(15);                       // waits 15ms for the servo to reach the position
  }
}

Note that this is my design process as I try and plan how I’m going to make this work, and part of that is trying to understand what’s possible. So this isn’t my code, but an example of how to control the servos I found here .

Categories
projects robots

Project Hexapod: Introduction

In 2018 I was building lots of Arduino projects. One of which was a robot car which sense walls using ultrasound to sense when it was about to drive into a wall. In the video below I connect via bluetooth, then enter some commands into the Arduino serial console

To build on this I decided to make a hexapod – a walking spider.

https://www.instagram.com/p/B-UYLZ4gD9uStzXugmaBnCfFzCwmCFCYKgSnoE0/

The parts are still available here, although there isn’t a codebase to use the robot – you need to make that yourself. And that’s what this new project is about.

Categories
IoT projects

Raspberry PI TFT Screen / USB Bitcoin Miner

Earlier this year I built a Bitcoin mining project with my old Raspberry PI.  Even though it’s the older model, the USB ASIC offloads all of the CPU processing so it’s an ideal project for the older PI (there’s a whole separate thread around whether we really need the extra CPU horsepower in the newer PI’s.  Although a benefit is that the old model B’s will probably get a lot cheaper on ebay)

 IMG_20160307_085113090

Buyer Beware – I did this about 4 weeks ago and had loads of fiddling to make it work.  I’ve gone back through bash histories to document everything I did.  If I’ve missed anything then that’s my excuse.  But feel free to drop me a line if you’ve tried stuff and it doesn’t work.

I installed a clean version of Jessie to start the project.  But I had to do a fair amount of fiddling to get the screen up and running.   Most of the effort in getting this project working was around getting the screen to a) work, and b) stay on.  As such I thought I’d document it here to see if it helps anyone.

The screen I picked up is from Amazon – described as a “Makibes® 3.5 inch Touch Screen TFT LCD (A) 320*480 Designed for Raspberry Pi RPi/Raspberry Pi 2 Model B”.  The back of the screen says “3.5inch RPi LCD (A) V3 WaveShare SpotPear”.

screen

I think the Makibes thing is a rebrand as most of the google search results for the errors I was getting were brought up the WaveShare screen.  As per the comments in the Amazon page, this link many found helpful.  It got me to the point where I could manually load the modules.  But it didn’t stay persistent over the reboot.  As per the page I just linked to (and I’m just copying and pasting his work here, check out the link for the full info), I could get the screen working with a modprobe

modprobe flexfb nobacklight regwidth=16 init=-1,0xb0,0x0,-1,0x11,-2,250,-1,0x3A,0x55,-1,0xC2,0x44,-1,0xC5,0x00,0x00,0x00,0x00,-1,0xE0,0x0F,0x1F,0x1C,0x0C,0x0F,0x08,0x48,0x98,0x37,0x0A,0x13,0x04,0x11,0x0D,0x00,-1,0xE1,0x0F,0x32,0x2E,0x0B,0x0D,0x05,0x47,0x75,0x37,0x06,0x10,0x03,0x24,0x20,0x00,-1,0xE2,0x0F,0x32,0x2E,0x0B,0x0D,0x05,0x47,0x75,0x37,0x06,0x10,0x03,0x24,0x20,0x00,-1,0x36,0x28,-1,0x11,-1,0x29,-3 width=480 height=320
modprobe fbtft_device name=flexfb speed=16000000 gpios=reset:25,dc:24

If your screen matches the description above and that works, then happy days.  Here’s what I ended up doing to make it persistent post reboot:

First off, if it isn’t already enable SPI in the raspi-config tool

in  “/boot/config.txt” I’ve appended the following lines

Enable audio (loads snd_bcm2835)
dtparam=spi=on
# dtoverlay=ads7846,cs=1,penirq=17,penirq_pull=2,speed=1000000,keep_vref_on=1,swapxy=0,pmax=255,xohms=60,xmin=200,xmax=3900,ymin=200,ymax=3900
dtoverlay=ads7846,speed=500000,penirq=17,swapxy=1
dtparam=i2c_arm=on
dtoverlay=pcf2127-rtc
# dtoverlay=w1-gpio-pullup,gpiopin=4,extpullup=1
device_tree=on

/boot/cmdline.txt passes parameters to the bootloader.  I’ve appended a couple of lines to make the console appear on the SPI TFT screen instead of the default HDMI.  Also, the console blanking is disabled

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait fbcon=map:1 fbcon=font:ProFont6x11 logo.nologo consoleblank=0

/etc/modules now looks like this:

snd-bcm2835
i2c-bcm2708
i2c-dev

Something I didn’t pick up from other forum posts and blogs is the config required to auto load modules on bootup.  So I created /etc/modules-load.d/fbtft.conf to effectively do what modprobe is doing from the command line

spi_bcm2708
flexfb
fbtft_device

Console Blanking is apparently a bit busted in Jessie, so /etc/kbd/config needs the following settings (they aren’t next to each other in the file so you’ll need to search through it to make both edits:

BLANK_TIME=0
POWERDOWN_TIME=0

And /etc/init.d/kbd needs to look like this (search for screensaver stuff in the file – it’s quite long)

# screensaver stuff
setterm_args=""
if [ "$BLANK_TIME" ]; then
setterm_args="$setterm_args -blank $BLANK_TIME"
fi
if [ "$BLANK_DPMS" ]; then
setterm_args="$setterm_args -powersave $BLANK_DPMS"
fi
if [ "$POWERDOWN_TIME" ]; then
setterm_args="$setterm_args -powerdown $POWERDOWN_TIME"
fi
if [ "$setterm_args" ]; then
# setterm $setterm_args
TERM=linux setterm > /dev/tty1 $setterm_args
fi

That should get you to the point where your Raspberry PI will reboot and then always use the TFT screen as a display output.

To complete the project I used this USB ASIC to do my bitcoin mining.  Amazon is out of stock at the time of writing.  However it will give you what you need to search ebay for, etc.  This instructables is complete enough that there’s little point me replicating it here.  However, there were a couple of additions that I needed to do before it worked and to complete the project.

dongle

First off I needed to install some additional packages:

sudo apt-get install autoconf autogen libtool uthash-dev libjansson-dev libcurl4-openssl-dev libusb-dev libncurses-dev git-core

Download the zip file, build and install the code

wget http://luke.dashjr.org/programs/bitcoin/files/bfgminer/3.1.4/bfgminer-3.1.4.zip
unzip bfgminer-3.1.4.zip
cd bfgminer-3.1.4
make clean
./autogen.sh
./configure
make
sudo make install

The last part of the project is a total hack as I ran out of steam with my enthusiasm.  It’s pretty insecure and absolutely not best practise, etc. etc.  But I got lazy, and it works.  I’m sure you can make something better given a few more brain cells.  First off, make sure the pi boots into console mode and not x windows.(option number 3 in raspi-config).  Then choose option

B2 Console Autologin Text console, automatically logged in as 'pi' user

Then I created  the following script – /home/pi/login.sh

#!/bin/bash
#
/usr/local/bfgminer-3.1.4/bfgminer -o stratum.bitcoin.cz:3333 -O username.password -S all

Every time it boots, the Raspberry PI automatically logs in as the ‘pi’ user.  And everytime the pi user logs in, that script is run.  Totally insecure.  And I’ll probably go back and fix it when I’ve got a spare 30 minutes.  But it works for now

Categories
IoT projects

Raspberry PI RTC X100

Here is how to get the Real Time Clock (RTC) for the X100 Raspberry PI expansion board working in Jessie.  Much of the information I found was out of date or for older versions of Raspbian.  To check out which version of the expansion board you have, there’s a decent write up on Sainsmart.  My X100 Board on my original Model B looks like this

 

X100 sidebyside

 

There are instructions on that SainSmart link for getting the different components of the expansion board up and running, including the real time clock.  However they’re horrendously out of date, including the suggestion to install version 3.0 of the Linux Kernel.  A bit of detective work shows that the rtc on the X100 expansion board is actually the RasClock.  (you can tell because the Sainsmart page has copied AfterThought’s instructions for setting up “Setting up on old version of Raspbian”

It wasn’t massively clear from their instructions what I needed to to do to get the clock working.  A bit fiddling got me the following steps:

1.  Check which overlay you have:

# ls/boot/overlays/

2. Follow the Afterthought Instructions (quoted here for convenience)

If you see the file /boot/overlays/i2c-rtc-overlay.dtb. To enable this add the following line to /boot/config.txt:

dtoverlay=i2c-rtc,pcf2127

If you see the file /boot/overlays/pcf2127-rtc-overlay.dtb. To enable this add the following line to /boot/config.txt:

dtoverlay=pcf2127-rtc

Edit the file /lib/udev/hwclock-set

  • Comment out or remove the three lines
if [ -e /run/systemd/system ] ; then
    exit 0
fi
  • Comment out or remove the two lines which contain --systz

 Now reboot your PI

3. Getting and Setting the time

The first time you use the clock you will need to set the time. To copy the system time into the clock module:

sudo hwclock -w

To read the time from the clock module:

sudo hwclock -r

To copy the time from the clock module to the system:

sudo hwclock -s

On my system, it looked like this:

hwclock

I did also enable the I2C and SPI bus’s on the PI during this process.  I don’t think that actually affected the RTC but maybe something to have a look at if yours isn’t working.

Categories
projects

How To Set Up an Internet Email Server

There’s a number of howto’s on the web about how to configure an open-source email server.  What I struggled to find is something that told me WHAT  it was I was trying to configure, as opposed to HOW I should configure it.

If you have something working, and you break it – it’s easy to undo your change, work out what part of the change broke it, and then decide what you need to do to fix it.  However what if I’ve never had something working in the first place? Have I misconfigured the software?  Or is the software configured correctly but I’m expecting the software to do something it isn’t supposed to do?  Both end up with it not working, but the solutions are drastically different.

In my view it’s important to understand not only the nuts and bolts of how to configure something, but be clear about what you are configuring it to do.  That makes the whole process a lot easier

Assumptions here are that you have your own server, on the public internet, and you’re trying to set it up so you can send and receive email from it.  I’m also assuming you own and manage your own DNS records.  My server is running Ubuntu Linux, so I’ll refer to their instructions.  If you’ve chosen something else then there’s actually an awful lot of configuration guides out there.  This article is to try and help you work out what you’re trying to configure, not how to do it.

To start with then, here’s a baffling diagram illustrating how the different components interact:

overview-email-setup

The 10,000 mile high view:

Email comes in to the server via SMTP (Postfix) and stored on a filesystem.  That email can be accessed by the user with a client talking to the IMAP server (Dovecot) which looks in the folders Postfix is delivering emails to.  At a really high level that’s how it works.

Before delving into more detail, lets take a look at what the system needs to do

What before how

Lost of other places will show you the configuration options to setup the software components I’m going to talk about here.  But before I talk about HOW to do something (and even then I’m only going to talk at a high level and leave configuration for the other HOWTO’s out there) – I want to talk about the WHAT.  What was it I wanted to create?  What would I have at the end of the exercise?

I wanted to replicate (well replace actually) my Gmail experience. To do that I needed to:

  1. Have a domain name people can email me at,
  2. Install Email Server Software that can receive email,
  3. Install Email Server Software that can receive email,
  4. Install Web Server Software that can show me my email through a web interface,,
  5. Configure the system so I can only see my email and not email for other users
  6. Configure the system so I can have pretend users (that’s kind of complicated and confused me for a while.  Take it as read for the minute and I’ll explain it later),
  7. Make the system secure.  I wanted the system to use encryption where possible so my information can’t be easily be read over a network connection.  I also wanted the system locked down so it can only be used for the purpose it was designed and not as a SPAM relay.

Some of that requirement is iterative.  Specifically point 6 wasn’t something I realised I wanted until I started learning about how to set the system up. I did however get completely bogged down in the configuration of the point 2 and point 3 – mainly because I hadn’t properly thought out what I was trying to setup.  The errors in the logs didn’t make sense.  I just dived straight into a “how to configure Dovecot / Postfix / SASL” HOWTO without thinking about what the end thing was going to look like.

Your requirements may differ (well probably will differ tbh).  If some of this helps someone then great.  The biggest thing I think will help is reminding people to have a plan before they start configuring.

SMTP – Sending Email

1. Have a domain name people can email me at

Email is delivered using the SMTP protocol. First off you need your DNS configuring. Every domain name has a number of attributes;

  • A Records – the names of real servers
  • CNAME’s – aliases to A Records
  • NS Records – the DNS server that has the details about your domain
  • MX records – Mail eXchanger.  This is the email server for your domain

MX is what we are interested here.  For PressedonTech a DNS search will tell you the MX record for pressedontech.com is mail.pressedontech.com.

2. Install Email Server Software that can receive email

I selected Postfix for this tool (partly because it was already installed, partly because there was a lot of related documentation, partly because I hate SendMail).

smtp1

Postfix is a Mail Transfer Agent (MTA) and it’s doing two jobs.

  1. Inbound email;
  • Postfix listens on TCP Port 25, using the SMTP protocol, to receive email ,
  • Postfix is checking it’s the right server for the email it’s being asked to process (i.e. it is configured to process email for pressedontech.com)
  • Postfix is checking it is configured to send the email to the place it’s being asked to?  For pressedontech.com it’s storing email on the server filesystem.  Other configurations will tell Postfix to pass on the email to someone else.  If that’s not what you want then make sure you aren’t telling Postfix to do that.  Potentially you’re allowing your server to be a spam relay.
  • After it’s passed these checks it puts the email where it’s told to (or errors if the checks fail 🙂 )

These things are important because spam is an issue on the internet.  Postfix is making sure the SMTP client isn’t trying to use my installation to send email pressedontech.com isn’t authorised to .

2. Outbound email

  • Postfix is sending email on your behalf to MTA’s running on other computers using SMTP.
  • Postfix is looking for outbox messages, for your user, (on the unix filesystem in my case) and then trying to deliver them

To recap – Postfix is the Mail Transfer Agent that controls the delivering of email in and out of my server.  The next step is something that can show that email to a user.  Read on to find out more…..

IMAP – Reading Email

3. Install Email Server Software that can receive email

Email has been delivered to you server using SMTP.  How do you read it?

IMAP1

IMAP is the Internet Message Access Protocol.  IMAP client software talk to IMAP server software to access messages in mailboxes on the server.

You can use the POP3 – the Post Office Protocol if you like.  For a massive oversimplification – IMAP is better for accessing email on a server. POP3 is better for downloading email to a computer.  For my use case I can imagine multiple IMAP clients talking to my server so it makes more sense for me to use IMAP and not POP3.

From the search I did, there were two options for a Linux / Open Source based IMAP Server.  Cyrus or Dovecot.  Cyrus is a much older and more mature project.  However Dovecot seemed to have much more recent documentation available, and more howto’s and forum threads that seemed related to what I was trying to do.  So I picked Dovecot.

Next is second part of reading email, the web interface….

IMAP – Reading Email – part 2

4. Install Web Server Software that can show me my email through a web interface

For my use case, I expected to want to read email on my phone, and read email in a web browser interface from my PC.  I could have just set up another desktop IMAP client but webmail seemed to be the way to go (I suspect this is Gmail brainwashing but it was something else to learn so I went with it.  For this I selected Roundcube.

IMAP2

Roundcube is a PHP application.  In your web browser you log in to the Roundube application.  Roundcube then talks to Dovecot and shows you your emails. Roundcube stores it’s own configuration in MySQL (hence the database requirement).

But how does Dovecot know how to show Roundcube just my email and no-one elses?  Follow through to the next page to find out…..

Each User has their own email

5. Configure the system so I can only see my email and not email for other users

Dovecot is acting as the authentication process for everything.

  • When Postfix receives an email to process it asks Dovecot “does user@pressedontech.com” exist?  If it has a valid user it lets Postfix know.  It took me ages to get this bit right.  If enough people want it I can post my final config but there’s plenty of technical documentation to help you sort this out.
  • When a user on their smartphone connects through the mail app, it’s providing a username and password.  Dovecot checks and provides access to the appropriate mailbox, assuming that exists

If you look at the various DoveCot / Postfix forum posts, it’s the integration between these two software components (specifically getting Postfix to authenticate users using Dovecot’s authentication process) that seems to be the one thing that gives people the most headache.  First off I setup Dovecot to authenticate against real users

authentication

Dovecot can check against real system users (i.e. a UNIX user who can ssh onto a server, a separate file with email only users or data in a database such as MySQL).  This is where I came unstuck as well because many of the HOWTO’s talk about virtual users – description which makes sense in hindsight but I didn’t properly wrap my head around when trying to troubleshoot why Dovecot wasn’t talking to Postfix properly.  By configuring Dovecot against real users first, I got the basic system working before trying to do something clever.

A virtual email user means that I don’t need a real UNIX user to send email to someone.  Why is that important?  If I want a secure system, I don’t want the username’s and passwords of users that can log in via ssh spread across the internet.  By configuring virtual users I can send email to tom@pressedontech.com, dick@pressedontech.com or harry@pressedontech.com without needing those real users on the system. This can be implemented using encrypted flat files, or with a database backend.  I chose to use MySQL and the final configuration looked like this:

authentication 2

Next we learn about making the system secure

Security Security Security

7. Make the system secure.

There’s a few best practises to consider here.

The first is to expose as few services to the internet as possible.  Therefore

  • I don’t run a POP3 Server because I don’t see an immediate need to use POP3.  This is a key principle of only configuring the minimum set of services required. If you configure a service then you’d better make sure it’s configured properly.  If you aren’t going to use a service, then why waste time configuring it when you could just disable it and focus your efforts on doing a better job with the services you are going to use
  • I do use a firewall.  Belt and braces here.  I think I’ve disabled services I don’t need.  With a firewall configured then I’m making extra sure that I’m only exposing the services to the internet that I intend to.
  • As many of my services as possible are routed through the localhost / loopback / 127.0.0.1 IP address.  I’m not providing database services to the internet so why would my MySQL server need to be contacted by an internet IP address?  By using either local UNIX ports or non-routable IP Addresses you are limiting the amount of traffic you are sharing outside of the service and reducing the attack vectors for your server
  • Encrypt as much as possible.  For the services I do expose to the internet, then these are configured using TLS.  At the minute I’m self signed, mainly because it’s just me using it. But it gives me the confidence that I’m not passing my username and password details around in plaintext for all to see.

My secondary concern was around not turning my system into an open spam relay.  I don’t want other people using my SMTP server to send emails pretending to be from my domain because

a) because that’s not cool.

b) because that will get my domain and IP blacklisted so my own email won’t work

c) I’ll probably get moaned at and potentially switched off by my ISP.  and I don’t want that either

As part of configuring the system I spent a fair amount of time on this topic, making sure I understood how to lock the system down, how to prevent unauthorised users and how to only relay for my domain.

External checking

There were a number of tools I used once I had my system up and running.  The first was a portscan to check I was only exposing the services I thought I was exposing.  Secondly I used a number of different websites to check my server wasn’t acting as an open spam relay.  Notice here I said I used more than one service.  I did this for both the port scanning and the SMTP relay check.  Using more than one tool gives you the extra confidence in your results.  It’s one thing to think you’ve done it correctly.  It’s another to have someone else check your homework.  This was one of the key points in making sure I was happy with what I’d setup.

It took me several days of trial and error to get my system up and running.  During this period I made a point of switching things off whilst I wasn’t configuring them.  I would make some progress, hit a roadblock and leave the system to come back another day and try and make more progress.  It would be silly to leave a half configured or misconfigured service live on the internet so when I wasn’t configuring, the services were disabled.

Finally make sure you’re patched and up to date.  If you’re using a Linux system with a package manager, it’s a 5 second job.

I don’t think any of these make my server hack proof.  Nothing is invulnerable.  However by taking these options your making it less vulnerable and reducing any options potentially hackers may have.

In Summary

Hopefully that gives a good overview of how an internet email service can be setup.

overview-email-setup

There are lots of other websites out there giving instructions on how to configure the email software, which options to pick, how to generate self signed certificates, etc. What I wanted to do here was give a high level view of what you were trying to configure and not duplicate the other places that tell you how to configure it.

As per usual, any feedback or clarification on what I’ve written here is always welcome. I hope some of that info is useful to people

Categories
projects

What to consider when replacing free services like Google, Microsoft, etc

As per my previous posts around removing myself from Google’s clutches, I’ve setup my email server.  And it was hard.  Something that’s very common in open source howto’s are very detailed explanations of how to configure things.  What’s not very common are high level overviews of how technologies interlink.  Which makes troubleshooting things difficult.  Did I choose the right option in my IMAP server?  Have I misconfigured my authentication modules?  Or is it broken because what I’m trying to do is fundamentally wrong because I haven’t properly understood what I’m doing?

Running a service on the public internet is a dangerous thing.  Part of the logic behind moving my email to a personal server instead of using one of the free public services Google or Microsoft provide is for enhanced privacy and control over my own data.  As such setting up an email server full of security holes isn’t going to achieve that goal.

The upshot therefore is

  • Choose the right software.  Take a look at the product history.  Is it still frequently updated?  Is it well documented?  Does it have a history of being insecure or is it well patched?
  • Make sure you update that software.  No point running “secure” products if you’re running an old version of that software
  • Have a think about how you’re configuring that software.  For different server components that need to communicate (such as Apache talking to MySQL), configuring them to talk over localhost (127.0.0.1) instead of the internet facing IP address will limit the servers exposure to the internet.

That’s just a few ideas.  It’s not an exhaustive list but they illustrate that once you’ve worked out what software you want, and you’ve worked out how it should work you’ve then got an initial task of making sure the product is configured properly and securely and then an ongoing job making sure it’s still secure.  By taking on the job yourself you’re actually taking on a level of responsibility for your own security.  Something you currently take for granted.

I’ve written an email server overview here.  It’s not designed to be a configuration HOWTO.  There’s plenty of those out there written to a high quality. What I have tried to do is describe the different email server components so you can understand what you want to configure, before trying to delve into how you’re going to configure it.

It’s easy to see why getting all this stuff for free is so tempting

Categories
privacy projects

Bye Bye GMail, see you later OneDrive

Here’s my project to extricate myself from free services.  I’ve got an Android phone.  I use gmail.  And Hangouts.  And Google Maps.  And Google News

Google Now watches where I go to frequently and it thinks it knows where I work.  It sees the purchases I make and gmail tells it when they are being delivered.  It sees the places I go and makes suggestions on where to go.  It’s a little creepy

As such I feel motivated to do something about it.

Following in the footsteps of Bryan at Bad Voltage, I’m taking steps to extricate the “free” services from my technology life.  Starting today I’m moving away from GMail.  It’ll be a slow transition because it’s not a 5 minute job.  iPhone email and webmail are working.  But I think I want a more complicated configuration; multiple email domains, virtual users, etc.  That’s obviously part of the hook with Gmail and it’s Software as a Service model.  All that hard work is done for you.

(The irony that most of my mobile technology posts are pro-Android but from a security and privacy perspective, Apple’s mobile OS is probably much better for you – well that’s not lost on me.  I’m just sulking about it)

This is the second step I’ve taken.  The first was to give Firefox yet another go.  I’ve got a post brewing for a while about why Firefox is really the only browser to trust and the steps you can take to make it even more secure and private.  Back to the “too many eggs in one basket” theme of this post however – the summary is that nothing is free, there is a cost to everything.

But is it worth it????

Categories
cloud projects

Why I bought a server instead of using the Cloud

Everything is cloud computing, Amazon AWS, etc. etc. these days.  And yet for this website I kept it old school and purchased a Linux web server I had full root access to.  Why was I such a Luddite?  A couple of reasons really.

Curiosity killed the techie

Part of this is a learning experience.  A hundred years ago I used to be a sys-admin.  I still play with Linux periodically, on laptops, raspberry pi’s and virtual machines.  But I wanted to build a thing from the ground up.  To choose my webserver (apache or lighttpd), choose my blogging software, etc.  I wanted a platform where I could do a little coding and be free to install what I wanted.  And then to secure the whole platform correctly.  In summary this website isn’t just a tool for me to share (or vent) my thoughts on a number of technology subjects.  It’s also a learning exercise

But why so old school?

One of the supposed break-through’s that Amazon AWS provides is the ability to be charged for what you use.  From a processor  / memory perspective but also from a bandwidth perspective.  Which for a small blog like this should be more cost effective.  But what happens when my massive genius delivers the post that makes me famous.  The popular web-based source code repository Github hosts their sites on Amazon’s AWS cloud platform.  When they got DDoS ‘d earlier this year (supposedly by The Great Firewall of China), through no fault of their own and completely out of their control, Github started incurring a $30,000-a-day Amazon bill.  Which is more than my little hobby can stand.

I’ve obviously picked an extreme example to make a point. But whilst paying for a dedicated virtual server is certainly not the cheapest website hosting option, it’s a known cost every month and something I can control.

The elephant in the cloud computing room

The pay per use / consumption model stands out to me as one of the major unspoken issues with cloud computing.  The implication is that we have all bought far too many servers for what we really need. If we just moved to a usage model where we just pay for what we use then that’s bound to be so much cheaper.  This concept is based on the assumption that IT (or non-IT people for that matter) have a really good idea on how specific applications are applying load to existing infrastructure.  Or they have mature processes for projecting the infrastructure requirements for future applications currently in development.

Oink Flap Oink Flap Flap

Ahem

A consumption based solution like cloud has an associated requirement that product revenue is closely tied to infrastructure utilisation, i.e. if my hosting costs increase then that’s fine because that means my product is generating more revenue in line with the increased cost.  But in reality how many people are re-architecting their applications this way.  Sometimes that’s not even possible.  Re-designing an application to tie it’s infrastructure utilisation to the revenue it will create is more than just a technology problem. It’s a business process and sales problem as well.

Summary

As a technology I’m a big fan of Infrastructure as a Service and Platform as a Service cloud technologies.  I think the problem is when we are promoting them based on business reasons and not technology reasons.  And often either the business isn’t engaged in that conversation, is just too far removed to understand the problem or even be in a position to make the required changes to adopt those benefits.

I’d share a picture of a Emperor but nobody needs to see that