mongodb replica set using docker compose

In this post I am going to show how to setup a mongodb replica set using docker-compose for a development environment. This will allow us to build and test applications that will be using a replica set in production. Using docker compose, we can setup three docker containers each running a mongodb database, on a single host. We do not have to provision separate servers for us to be able to test our application. This keeps our development environment costs low.

Note: See Part II for mongodb replica set using docker, private DNS in AWS VPC.

Procedure

First make sure you have docker and docker compose installed on your server. If this is your first time working with docker or docker compose, follow the procedure listed here for their installation.

Docker Community Edition Installation

Docker Compose Installation

I will be running my procedure as root user. However, if you do not wish to run as root user, you can run docker as a non root user by following instructions from here.

Create a work folder call compose to create your docker-compose configuration file. I am doing this in /home/ubuntu/compose/docker-compose.yml and paste the configuration listed below.


version: "3"
services:
  mongo1:
    hostname: mongo1
    container_name: mongo1
    image: mongo:4.0.4
    volumes:
      - ./data1/db:/data/db
      - ./data1/configdb:/data/configdb
    networks:
      - mongo-dev-net
    expose:
      - 27017
    ports:
      - 30001:27017
    restart: always
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "devrs" ]
  mongo2:
    hostname: mongo2
    container_name: mongo2
    image: mongo:4.0.4
    volumes:
      - ./data2/db:/data/db
      - ./data2/configdb:/data/configdb
    networks:
      - mongo-dev-net
    expose:
      - 27017
    ports:
      - 30002:27017
    restart: always
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "devrs" ]
  mongo3:
    hostname: mongo3
    container_name: mongo3
    image: mongo:4.0.4
    volumes:
      - ./data3/db:/data/db
      - ./data3/configdb:/data/configdb
    networks:
      - mongo-dev-net
    expose:
      - 27017
    ports:
      - 30003:27017
    restart: always
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "devrs" ]

networks:
  mongo-dev-net:


Configuration Review

Let us do a quick review of the file we just created. There are three containers defined in this file. Let us examine one of them, the other two are identical except for the ports and volumes.


  mongo1:
    hostname: mongo1
    container_name: mongo1
    image: mongo:4.0.4
    volumes:
      - ./data1/db:/data/db
      - ./data1/configdb:/data/configdb
    networks:
      - mongo-dev-net
    expose:
      - 27017
    ports:
      - 30001:27017
    restart: always
    entrypoint: [ "/usr/bin/mongod", "--bind_ip_all", "--replSet", "devrs" ]

I am setting the container name to be the same as the hostname. This is not a requirement but something I just decided. As long as the names for the three containers are unique, they will work for us. These names will be referenced by us when we setup the replica set.

Next lists the docker image I want to use to setup the database. I am locking it down to a particular version in this test by specifying


image:mongo:4.0.4

In this setup, I am going to use local folders and map them into my container. I could instead use docker volumes as well. Both these method will ensure that the data will persist.

Note: If you use docker volumes, make sure you do not run docker-compose down until you are done with you testing. This will remove the docker volumes that were created.


    volumes:
      - ./data1/db:/data/db
      - ./data1/configdb:/data/configdb

I am mapping local folder /home/ubuntu/compose/data1/db to /data/db and /home/ubuntu/compose/data1/configdb to /data/configdb inside the container. When mongodb starts it will end up writing to the local folders I have setup. The data1, data2 and data3 folders do not have to exists, they will be created for you if they do not exist on first run.


    networks:
      - mongo-dev-net
    expose:
      - 27017
    ports:
      - 30001:27017

mongo-dev-net is the name of the docker network that will be setup when docker-compose is run. I am setting the localhost port 30001 to map to 27107 for the first container. Similarly the other two containers use the next consecutive port number. This will allow us to connect to our mongdb database from the host server on which we are setting up our containers.

The name of the replica set I choose is devrs and the final lines define the network (mongo-dev-net) to be setup.

Startup


docker-compose up -d
Creating network "compose_mongo-dev-net" with the default driver
Pulling mongo1 (mongo:4.0.4)...
4.0.4: Pulling from library/mongo
7b8b6451c85f: Pull complete
ab4d1096d9ba: Pull complete
e6797d1788ac: Pull complete
e25c5c290bde: Pull complete
45aa1a4d5e06: Pull complete
b7e29f184242: Pull complete
ad78e42605af: Pull complete
1f4ac0b92a65: Pull complete
55880275f9fb: Pull complete
bd0396c9dcef: Pull complete
28bf9db38c03: Pull complete
3e954d14ae9b: Pull complete
cd245aa9c426: Pull complete
Digest: sha256:1b29fbe615ce2f0a91e8973a1aa6fca59b4aaa21bc5d6c8311e6a55cc6ff6b18
Status: Downloaded newer image for mongo:4.0.4
Creating mongo2 ... done
Creating mongo1 ... done
Creating mongo3 ... done

As you can see above, I ran this for the first time and since I did not have that image locally, docker pulled the image and setup my network and containers. A docker network was setup with the name mongo-dev-net.


docker container ps
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                      NAMES
7045092b85da        mongo:4.0.4         "/usr/bin/mongod --b…"   About a minute ago   Up About a minute   0.0.0.0:30001->27017/tcp   mongo1
236101f30e8c        mongo:4.0.4         "/usr/bin/mongod --b…"   About a minute ago   Up About a minute   0.0.0.0:30002->27017/tcp   mongo2
9442de2b68ec        mongo:4.0.4         "/usr/bin/mongod --b…"   About a minute ago   Up About a minute   0.0.0.0:30003->27017/tcp   mongo3

The above shows me my three containers: mongo1. mongo2 and mongo3 running and each one is mapped to a unique host port from 30001 to 30003.

Replica Set

Let us connect to one of our mongodb databases and form the replica set.


mongo --port 30001
MongoDB shell version v4.0.10
rs.initiate()
{
        "info2" : "no configuration specified. Using a default configuration for the set",
        "me" : "mongo1:27017",
        "ok" : 1,
        "operationTime" : Timestamp(1559332438, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1559332438, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
devrs:SECONDARY>
devrs:PRIMARY>
devrs:PRIMARY> rs.add('mongo2')
{
        "ok" : 1,
        "operationTime" : Timestamp(1559332464, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1559332464, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
devrs:PRIMARY> rs.add('mongo3')
{
        "ok" : 1,
        "operationTime" : Timestamp(1559332467, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1559332467, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}


I logged into and ran rs.initiate() and then added in the other two mongo databases to form the replica set. I used the names defined in my configuration file.

Let us check our replica set.


devrs:PRIMARY> rs.printSlaveReplicationInfo()
source: mongo2:27017
        syncedTo: Fri May 31 2019 19:56:00 GMT+0000 (UTC)
        0 secs (0 hrs) behind the primary
source: mongo3:27017
        syncedTo: Fri May 31 2019 19:56:00 GMT+0000 (UTC)
        0 secs (0 hrs) behind the primary

The above shows our replica set is synced.

A find from our host folder shows that mongodb is correctly using our local folders.


find data1
data1
data1/db
data1/db/index-1-8152178721862226256.wt
data1/db/collection-15-8152178721862226256.wt
data1/db/WiredTiger.turtle
data1/db/index-5-8152178721862226256.wt
data1/db/index-3-8152178721862226256.wt
data1/db/index-10-8152178721862226256.wt
data1/db/collection-8-8152178721862226256.wt
data1/db/collection-9-8152178721862226256.wt
data1/db/collection-6-8152178721862226256.wt
data1/db/index-12-8152178721862226256.wt
data1/db/index-14-8152178721862226256.wt
data1/db/index-16-8152178721862226256.wt
data1/db/collection-0-8152178721862226256.wt

Now let us stop and start our containers and verify our db is intact.


root compose]#docker-compose stop
Stopping mongo1 ... done
Stopping mongo2 ... done
Stopping mongo3 ... done
root compose]#docker-compose start
Starting mongo1 ... done
Starting mongo2 ... done
Starting mongo3 ... done

Let us login back to any one of our mongodb container and see the status.


devrs:SECONDARY> db.runCommand("ismaster")
{
        "hosts" : [
                "mongo1:27017",
                "mongo2:27017",
                "mongo3:27017"
        ],
        "setName" : "devrs",
        "setVersion" : 3,
        "ismaster" : false,
        "secondary" : true,
        "primary" : "mongo2:27017",
        "me" : "mongo1:27017",
        ----- other info ------
        ----- other info ------
}

As you can see from above, our replica set is still there with its three members. The replica set name is devrs what we defined in our configuration file.

In this case, I can even run docker-compose down and still retain my replica set on next startup as all the mongodb information was written to local folders, which were mapped to the containers.

For connecting to this mongodb replica set, I could use the host ip and any of the three ports form 30001 to 30003 to test. Now in my application, I could setup my connection string to use all three host:ip strings.

Conclusion

This is an inexpensive way to setup a mongodb replica set for testing. It does not take much time to setup the databases for doing some tests quickly.

The data folders could be moved to a different server if needed and the containers could be started on a different host. This setup is meant for a non production environment only.

Further Reading

8 COMMENTS

  1. Interessting aticle! I was possible get the setup working. But i am not able to connect to the DB. Would you be so kind to share the connectionstring used to connect to this setup?

  2. Thanks for reading the post!

    On the same host you could connect like this
    mongo –port 30001

    or from another host
    mongo –port 30001 –host ip of your server

    This is a good way to test a replica set, for connecting from applications, I suggest to have DNS. See the part – ii of this this article where I also set DNS for the replica set.

  3. For people who want to connect directly from host. Edit your /etc/hosts and add:

    127.0.0.1 mongo1
    127.0.0.1 mongo2
    127.0.0.1 mongo3

    then connect with:

    mongo “mongodb://mongo1:27017,mongo2:27018,mongo3:27019/test?replicaSet=devrs”

  4. I’m being able to connect to each single mongo from my host machine, but not able to connect to them like a replicaSet either from robo3t or nodejs using
    “mongodb://mongo1:30001,mongo2:30002,mongo3:30003/test?replicaSet=devrs”
    I tried with localhost, to each server and also using 27017.

    My host os is Windows, if that matters.

Leave a Reply