- hpcloud
- :
- HP Scaling the Cloud Blog
- :
- Setting Up a Single Node OpenStack Storage Server ...
Setting Up a Single Node OpenStack Storage Server - Part 1
We’re pleased to publish this guide to walk you through the process of installing OpenStack Object Storage (aka Swift to the cool kids). While a lot of people are comfortable using public cloud for testing and development, some folks like setting up a simple environment on their own local servers, first. Most of the OpenStack documentation is pretty good, but it’s always good to have a more detailed, step by step guide. At the end of this tutorial, you should have a stable, local OpenStack Storage server where you can create users and store and retrieve files. Your local server will have the same API interface as HP Cloud Services, and you can use this interface for API testing.
What Is OpenStack Storage
OpenStack Storage, also known as Swift, is a python-based, open source cloud storage platform. It's generally set up with several front end proxy nodes talking to a collection of back end storage nodes, though for this article we'll be installing both on the same machine. You access the storage system through HTTP calls to the proxy server, which then talks to the storage nodes and figures out where your files are stored and returns them to you.
Swift has a concept of zones, which are groups of servers which store copies of data for fault tolerance. Swift by default stores copies of each file in three different zones, so you'll have as much capacity as 1/3rd of the drive space you allocate. If you have a 1.5 TB drive for Swift, you'll be able to store 500 Gigabytes of data. If you have a 30 Gigabyte VM drive, you'll be able to store 10 Gigabytes. We'll be setting up four zones on our server so you can see the system balance itself.
This article shows how to install the Diablo-4 version of Swift, aka Swift 1.4.3, and setting it up with swauth. There are new versions of Swift coming out all the time, but this is the last of the Diablo releases and the code is mature and stable.
If you'd like to learn more about how Swift works, check out the Swift Architectural Overview on the Swift Wiki. This article is heavily indebted to it's great instructions for setting up a Swift All In One (SAIO) development environment. If you're interested in building the latest trunk code yourself, be sure to check it out.
What You'll Need
We're going to walk through setting up a 2 disk server, one drive for the OS and Swift software, and the other drive dedicated to Swift's data storage. If you're setting up a physical server we'll need two hard drives, if you're setting up a virtual machine in VirtualBox or VMware Workstation you'll need to create a second virtual disk.
You can setup a server with both OS and swift storage on the same drive, just make sure you leave free space for your second partition when you install Ubuntu. In that case, you can replace 'sdb1' in these instructions with 'sda2'. If you have a more complex partition scheme, well, you can probably figure out your partition mapping yourself.
You'll need a copy of Ubuntu 11.04 Server for this, you can get the ISOs from Ubuntu:
http://releases.ubuntu.com/natty/
If you're creating a VM I'd suggest allocating at least 1 Gig of RAM, and an 8 gig boot drive. For the second disk I'd suggest at least 30 gig, since you'll probably want to be able to test storing large files. Creating a dynamically allocated disk will use less space but will be slightly slower in use.
For a VM you'll want to set the network to bridged mode so you can access the server from your computer and other VMs or machines on your network. You want to configure the server with a static IP address so you can tell swift it's address (the authentication system is 2-step, first you authenticate and get a token and a reference to the storage server, then you use your token to talk to the storage server).
System Installation
Install Ubuntu 11.04 Server on your first drive and configure the network. I'd suggest installing OpenSSH during setup so you can remotely administer your server. At this point you should be able to SSH into your server from another machine on the network. If you can do that, we should be ready to go.
First, let's update our package lists and make sure we have any available security updates installed. All of the following should be done as root, so:
sudo bash
And then:
apt-get update apt-get upgrade -y
We'll also install memcached and xfsprogs now, as well as curl to download files with, python-setuptools to install the swauth code with and python-software-properties to allow us to talk to PPA servers.
apt-get install -y curl memcached xfsprogs python-setuptools python-software-properties
We'll also add the PPA (python package archive) server for swift and install it. In this tutorial we're using the ppa: openstack-release/2011.3 PPA, which should have the latest version of the Diablo milestone. This will also create a swift user for us.
add-apt-repository ppa:openstack-release/2011.3 apt-get update apt-get install -y swift swift-account swift-container swift-object swift-proxy
And then set our user info for later commands. If you want to change this to some other user, set it here.
export SWIFT_USER=swift export SWIFT_GROUP=swift
Since we're using the swauth authentication scheme, we need a static IP address for the swift server. If you're only going to try out swift within the VM itself you can set this to 127.0.0.1, otherwise you should set it to the machine's public IP address!!! If you need to figure out your public IP address, ifconfig is your friend.
export SWIFT_PUBLIC_IP=127.0.0.1
Create a single partition on your second drive. (The routine here is probably 'n', 'p', '1', enter, enter, 'w'.)
NOTE: If you're installing everything on one drive, you'll need to setup the second partition on your drive here, instead, and you'll want to replace sdb1 with sda2 in all the following commands.
fdisk /dev/sdb
Format the partition as an XFS partition. XFS, or the X File System supports storing metadata on files in the file system, a feature which swift uses.
mkfs.xfs -i size=1024 /dev/sdb1
Next we modify /etc/fstab, so the file system will be mounted on boot.
cat >>/etc/fstab << EOF /dev/sdb1 /mnt/sdb1 xfs noatime,nodiratime,nobarrier,logbufs=8 0 0 EOF
Make the mount directory for our file system and mount it. Once it's mounted, make the four directories where we store our four different zones, and change their ownership to our swift user. Next we create a /srv directory and soft-link the directories we just created to it, for easy reference.
mkdir /mnt/sdb1
mount /mnt/sdb1
mkdir /mnt/sdb1/1 /mnt/sdb1/2 /mnt/sdb1/3 /mnt/sdb1/4
chown $SWIFT_USER:$SWIFT_GROUP /mnt/sdb1/*
mkdir -p /srv
for x in {1..4}; do ln -s /mnt/sdb1/$x /srv/$x; done
Now we make the directories to store our config files and zone data in, and change the ownership to our Swift user.
mkdir -p /etc/swift/object-server /etc/swift/container-server /etc/swift/account-server /srv/1/node/sdb1 /srv/2/node/sdb2/srv/3/node/sdb3 /srv/4/node/sdb4 /var/run/swift chown -R $SWIFT_USER:$SWIFT_GROUP /etc/swift /srv/[1-4]/ /var/run/swift
Next we'll create an rsyncd config file. We use rsync to keep our storage servers directories in sync.
cat >/etc/rsyncd.conf <<EOF uid = $SWIFT_USER gid = $SWIFT_GROUP log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid address = 127.0.0.1 [account6012] max connections = 25 path = /srv/1/node/ read only = false lock file = /var/lock/account6012.lock [account6022] max connections = 25 path = /srv/2/node/ read only = false lock file = /var/lock/account6022.lock [account6032] max connections = 25 path = /srv/3/node/ read only = false lock file = /var/lock/account6032.lock [account6042] max connections = 25 path = /srv/4/node/ read only = false lock file = /var/lock/account6042.lock [container6011] max connections = 25 path = /srv/1/node/ read only = false lock file = /var/lock/container6011.lock [container6021] max connections = 25 path = /srv/2/node/ read only = false lock file = /var/lock/container6021.lock [container6031] max connections = 25 path = /srv/3/node/ read only = false lock file = /var/lock/container6031.lock [container6041] max connections = 25 path = /srv/4/node/ read only = false lock file = /var/lock/container6041.lock [object6010] max connections = 25 path = /srv/1/node/ read only = false lock file = /var/lock/object6010.lock [object6020] max connections = 25 path = /srv/2/node/ read only = false lock file = /var/lock/object6020.lock [object6030] max connections = 25 path = /srv/3/node/ read only = false lock file = /var/lock/object6030.lock [object6040] max connections = 25 path = /srv/4/node/ read only = false lock file = /var/lock/object6040.lock EOF
Next, we'll enable rsync:
perl -i.bak -p -e's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/g' /etc/default/rsync
And restart it:
service rsync restart
To use the swauth authentication module, you'll need to download it from GitHub, as it's been removed from the core swift project. It lives at:
https://github.com/gholt/swauth
curl https://nodeload.github.com/gholt/swauth/tarball/master > swauth.tgz tar fxvz swauth.tgz
You'll end up with a version-tagged directory containing the swauth codebase. It'll be something like 'gholt-swauth-4755a33'. Lets rename that and install it.
mv gholt-swauth-* swauth cd swauth python setup.py install
Swift Configuration
Next we'll create the proxy server configuration. This configuration doesn't use SSL and has a default super_admin_key of 'swauthkey'. If you'd like to secure your swift install a bit more, change that to something else.
cat >/etc/swift/proxy-server.conf <<EOF [DEFAULT] bind_port = 8080 user = $SWIFT_USER log_facility = LOG_LOCAL1 [pipeline:main] pipeline = healthcheck cache swauth proxy-server [app:proxy-server] use = egg:swift#proxy allow_account_management = true account_autocreate = true [filter:swauth] use = egg:swauth#swauth set log_name = swauth super_admin_key = swauthkey default_swift_cluster = local#http://$SWIFT_PUBLIC_IP:8080/v1 [filter:healthcheck] use = egg:swift#healthcheck [filter:cache] use = egg:swift#memcache EOF
Next create /etc/swift/swift.conf, this file includes your unique hash string, if you lose it you'll lose all the data in your swift server.
cat >/etc/swift/swift.conf <<EOF [swift-hash] # random unique string that can never change (DO NOT LOSE) swift_hash_path_suffix = changeme EOF
Next create the 4 account server, container server and object server config files:
cat >/etc/swift/account-server/1.conf <<EOF [DEFAULT] devices = /srv/1/node mount_check = false bind_port = 6012 user = $SWIFT_USER log_facility = LOG_LOCAL2 [pipeline:main] pipeline = account-server [app:account-server] use = egg:swift#account [account-replicator] vm_test_mode = yes [account-auditor] [account-reaper] EOF cat >/etc/swift/account-server/2.conf <<EOF [DEFAULT] devices = /srv/2/node mount_check = false bind_port = 6022 user = $SWIFT_USER log_facility = LOG_LOCAL3 [pipeline:main] pipeline = account-server [app:account-server] use = egg:swift#account [account-replicator] vm_test_mode = yes [account-auditor] [account-reaper] EOF cat >/etc/swift/account-server/3.conf <<EOF [DEFAULT] devices = /srv/3/node mount_check = false bind_port = 6032 user = $SWIFT_USER log_facility = LOG_LOCAL4 [pipeline:main] pipeline = account-server [app:account-server] use = egg:swift#account [account-replicator] vm_test_mode = yes [account-auditor] [account-reaper] EOF cat >/etc/swift/account-server/4.conf <<EOF [DEFAULT] devices = /srv/4/node mount_check = false bind_port = 6042 user = $SWIFT_USER log_facility = LOG_LOCAL5 [pipeline:main] pipeline = account-server [app:account-server] use = egg:swift#account [account-replicator] vm_test_mode = yes [account-auditor] [account-reaper] EOF cat >/etc/swift/container-server/1.conf <<EOF [DEFAULT] devices = /srv/1/node mount_check = false bind_port = 6011 user = $SWIFT_USER log_facility = LOG_LOCAL2 [pipeline:main] pipeline = container-server [app:container-server] use = egg:swift#container [container-replicator] vm_test_mode = yes [container-updater] [container-auditor] [container-sync] EOF cat >/etc/swift/container-server/2.conf <<EOF [DEFAULT] devices = /srv/2/node mount_check = false bind_port = 6021 user = $SWIFT_USER log_facility = LOG_LOCAL3 [pipeline:main] pipeline = container-server [app:container-server] use = egg:swift#container [container-replicator] vm_test_mode = yes [container-updater] [container-auditor] [container-sync] EOF cat >/etc/swift/container-server/3.conf <<EOF [DEFAULT] devices = /srv/3/node mount_check = false bind_port = 6031 user = $SWIFT_USER log_facility = LOG_LOCAL4 [pipeline:main] pipeline = container-server [app:container-server] use = egg:swift#container [container-replicator] vm_test_mode = yes [container-updater] [container-auditor] [container-sync] EOF cat >/etc/swift/container-server/4.conf <<EOF [DEFAULT] devices = /srv/4/node mount_check = false bind_port = 6041 user = $SWIFT_USER log_facility = LOG_LOCAL5 [pipeline:main] pipeline = container-server [app:container-server] use = egg:swift#container [container-replicator] vm_test_mode = yes [container-updater] [container-auditor] [container-sync] EOF cat >/etc/swift/object-server/1.conf <<EOF [DEFAULT] devices = /srv/1/node mount_check = false bind_port = 6010 user = $SWIFT_USER log_facility = LOG_LOCAL2 [pipeline:main] pipeline = object-server [app:object-server] use = egg:swift#object [object-replicator] vm_test_mode = yes [object-updater] [object-auditor] EOF cat >/etc/swift/object-server/2.conf <<EOF [DEFAULT] devices = /srv/2/node mount_check = false bind_port = 6020 user = $SWIFT_USER log_facility = LOG_LOCAL3 [pipeline:main] pipeline = object-server [app:object-server] use = egg:swift#object [object-replicator] vm_test_mode = yes [object-updater] [object-auditor] EOF cat >/etc/swift/object-server/3.conf <<EOF [DEFAULT] devices = /srv/3/node mount_check = false bind_port = 6030 user = $SWIFT_USER log_facility = LOG_LOCAL4 [pipeline:main] pipeline = object-server [app:object-server] use = egg:swift#object [object-replicator] vm_test_mode = yes [object-updater] [object-auditor] EOF cat >/etc/swift/object-server/4.conf <<EOF [DEFAULT] devices = /srv/4/node mount_check = false bind_port = 6040 user = $SWIFT_USER log_facility = LOG_LOCAL5 [pipeline:main] pipeline = object-server [app:object-server] use = egg:swift#object [object-replicator] vm_test_mode = yes [object-updater] [object-auditor] EOF
Now that all our files are in place, we're going to make our ring files. Rings store the mapping to where files are actually stored. When the servers need to figure out who has what, they check the ring files. There are separate rings for accounts, containers and objects. When you add a new server to a Swift cluster or take a server out, you have to update your ring files, also known as rebalancing.
cd /etc/swift swift-ring-builder object.builder create 18 3 1 swift-ring-builder object.builder add z1-127.0.0.1:6010/sdb1 1 swift-ring-builder object.builder add z2-127.0.0.1:6020/sdb2 1 swift-ring-builder object.builder add z3-127.0.0.1:6030/sdb3 1 swift-ring-builder object.builder add z4-127.0.0.1:6040/sdb4 1 swift-ring-builder object.builder rebalance swift-ring-builder container.builder create 18 3 1 swift-ring-builder container.builder add z1-127.0.0.1:6011/sdb1 1 swift-ring-builder container.builder add z2-127.0.0.1:6021/sdb2 1 swift-ring-builder container.builder add z3-127.0.0.1:6031/sdb3 1 swift-ring-builder container.builder add z4-127.0.0.1:6041/sdb4 1 swift-ring-builder container.builder rebalance swift-ring-builder account.builder create 18 3 1 swift-ring-builder account.builder add z1-127.0.0.1:6012/sdb1 1 swift-ring-builder account.builder add z2-127.0.0.1:6022/sdb2 1 swift-ring-builder account.builder add z3-127.0.0.1:6032/sdb3 1 swift-ring-builder account.builder add z4-127.0.0.1:6042/sdb4 1 swift-ring-builder account.builder rebalance
And then start up Swift:
swift-init restart all
Swift should now be running. You should see 75 or so Swift processes if you run:
ps ax | grep swift
And there should be messages from replicators in /var/log/syslog:
tail /var/log/syslog
We should now have Swift installed and running. In Part 2 we'll cover creating accounts, configuring SSL and some suggestions about reverse proxies and troubleshooting.
Continue to Part 2




