ODK Sync Endpoint

ODK Sync Endpoint is an implementation of ODK Cloud Endpoints. It runs a server inside a Docker container that implements the ODK-X REST Protocol.

It communicates with your ODK-X Android applications to synchronize your data and application files.

Authentication

ODK Sync Endpoint does not store user information in its own database, instead it integrates with an LDAP directory or an Active Directory. That directory is then used to authenticate users and obtain user roles.

Note

As a consequence of the integration, Basic Authentication is the only supported authentication method.

ODK Sync Endpoint prerequisites

You must have Docker 17.06.1 or newer, and be running in Swarm Mode. Follow these links for detailed instructions on installing Docker and enabling Swarm Mode.

ODK Sync Endpoint Setup

ODK Sync Endpoint requires a database and a LDAP directory, you could follow the instructions and deploy all three components together or supply your own database and/or LDAP directory.

Note

All of the following commands should be run on your server.

If you are using git on Windows, make sure git is configured with "core.autocrlf=false" - otherwise it will convert line endings with LF to CRLF, which will cause problems with the .sh-files when used in the Docker containers, thus preventing odk/sync-endpoint from starting and instead just returning with an ":invalid argument"-error.

Setup instructions:

  1. Choose a directory to store you endpoint in. In that directory, run:
$ git clone https://github.com/opendatakit/sync-endpoint-default-setup
  1. Navigate into the the "sync-endpoint-default-setup" directory
  2. Checkout the sync-endpoint code by running:
$ git clone https://github.com/opendatakit/sync-endpoint
  1. Navigate into the sync-endpoint directory. Most likely
$ cd sync-endpoint
  1. Build sync endpoint by running the following: (NOTE: you will need Apache Maven installed >= 3.3.3)
$ mvn clean install
  1. Navigate back to the parent "sync-endpoint-default-setup" directory.
  2. In the "sync-endpoint-default-setup" directory run:
$ docker build --pull -t odk/sync-web-ui https://github.com/opendatakit/sync-endpoint-web-ui.git
  1. In the "sync-endpoint-default-setup" cloned repository run:
$ docker build --pull -t odk/db-bootstrap db-bootstrap
  1. In the "sync-endpoint-default-setup" cloned repository run:
$ docker build --pull -t odk/openldap openldap
  1. In the "sync-endpoint-default-setup" cloned repository run:
$ docker build --pull -t odk/phpldapadmin phpldapadmin
  1. Enter your hostname in the security.server.hostname field in the security.properties file.
  2. If you're not using the standard ports (80 for HTTP and 443 for HTTPS) enter the ports you're using in the security.server.port and security.server.securePort fields in the security.properties. Then edit the ports section under the sync section in docker-compose.yml to be YOUR_PORT:8080.

Note

It is important that the right side of the colon stays as 8080. This is the internal port that the web server is looking for.

  1. If you're using your own LDAP directory or database, continue with the instructions:
  1. In the cloned repository:
$ docker stack deploy -c docker-compose.yml syncldap
  1. The server takes about 30s to start, then it will be running at http://127.0.0.1.
  2. See the LDAP section for instructions on configuring users and groups.

Custom database

  1. If you haven't followed the common instructions, start with those.
  2. Remove the db and db-bootstrap sections in docker-compose.yml.
  3. Modify jdbc.properties to match your database. Supported database systems are PostgreSQL, MySQL and Microsoft SQL Server. Sample config for each type of database can be found on Github.
  4. Modify sync.env to match your database
  5. In the cloned repository,
$ docker stack deploy -c docker-compose.yml syncldap
  1. The server takes about 30s to start, then it will be running at http://127.0.0.1.

Custom LDAP directory

  1. If you haven't followed the common instructions, start with those.
  2. OPTIONAL: If your LDAP directory uses a certificate that was signed by a self-signed CA,
  1. Make the public key of the CA available to ODK Sync Endpoint with this command.
$ docker config create org.opendatakit.sync.ldapcert PATH_TO_CERT
  1. Uncomment the relevant lines in the configs section in docker-compose.yml and the configs section under the sync section in docker-compose.yml.
  1. Remove the ldap-service and phpldapadmin sections in docker-compose.yml.
  2. Modify the relevant sections in security.properties to match your LDAP directory. Further instructions are in the file.

Note

The default configuration does not use ldaps or StartTLS because the LDAP directory communicates with the ODK Sync Endpoint over a secure overlay network. You should use ldaps or StartTLS to communicate with your LDAP directory.

  1. In the cloned repository:
$ docker stack deploy -c docker-compose.yml syncldap
  1. The server takes about 30s to start, then it will be running at http://127.0.0.1.

Stopping ODK Sync Endpoint

  1. Run:
$ docker stack rm syncldap
  1. OPTIONAL: If you want to remove the volumes as well,
  • Linux/macOS:
$ docker volume rm $(docker volume ls -f "label=com.docker.stack.namespace=syncldap" -q)
  • Windows:
$ docker volume rm (docker volume ls -f "label=com.docker.stack.namespace=syncldap" -q)

LDAP

  • The default admin account is cn=admin,dc=example,dc=org.
  • The default password is admin - it can be changed with the LDAP_ADMIN_PASSWORD environment variable in ldap.env
  • The default readonly account is cn=readonly,dc=example,dc=org.
  • The default password is readonly - it can be changed with the LDAP_READONLY_USER_PASSWORD environment variable in ldap.env. This account is used by the Sync Endpoint to retrieve user information.

The LDAP directory that you deployed with the instructions above is an OpenLDAP server. In addition to the directory, a phpLDAPadmin server is also deployed to help you configure the directory.

If you'd prefer to use the OpenLDAP command line utilities, they're installed in the OpenLDAP container. These tools are accessible with this command:

  • Linux/macOS:
$ docker exec $(docker ps -f "label=com.docker.swarm.service.name=syncldap_ldap-service" --format '{{.ID}}') LDAPTOOL ARGS
  • Windows:
$ docker exec (docker ps -f "label=com.docker.swarm.service.name=syncldap_ldap-service" --format '{{.ID}}') LDAPTOOL ARGS

Note

The phpLDAPadmin server listens on port 40000, it is important that you do not expose this port to the internet.

The following guides assume that you're using phpLDAPadmin. In order to perform the following operation, please go to https://127.0.0.1:40000 in your browser.

Creating users

  1. Click: login on the left and login as admin.
  2. Expand the tree view on the left until you see ou=people.
  3. Click on ou=people and choose Create a child entry.
  4. Choose the Generic: User Account template.
  5. Fill out the form and click Create Object.
  6. Assign users to groups with these instructions.

Creating groups

  1. Click: login on the left and login as admin.
  2. Expand the tree view on the left until you see ou=groups.
  3. Click on ou=default_prefix and choose Create a child entry.
  4. Choose the Generic: Posix Group template.
  5. Fill out the form and click Create Object.

Note

The group name must start with the group prefix, in this case the group prefix is default_prefix so for example: default_prefix my-new-group

  1. Assign users to groups with these instructions.

Assigning users to groups

  1. Click: login on the right and login as admin.

  2. Expand the tree view on the right until you see ou=default_prefix, then expand ou=default_prefix.

  3. This list is all the groups under ou=default_prefix.

  4. Click on the group that you want to assign users to.

  5. A few groups are created when the LDAP server is brought up, refer to Data Permission Filters for descriptions of these groups.

  6. If the memberUid section is not present:

    1. Choose Add new attribute.
    2. Choose memberUid from the dropdown, then enter uid of the user you want to assign.
    3. Click Update Object at the bottom to update.
  7. If the memberUid section is present,

  1. Navigate to the memberUid section.
  2. Click modify group members to manage members.

HTTPS

  1. Store your certificate public key in a Docker config with this command:
$ docker config create example.com.fullchain.pem PATH_TO_PUBLIC_KEY
  1. Store your certificate private key in a Docker secret with this command:
$ docker secret create examepl.com.privkey.pem PATH_TO_PRIVATE_KEY
  1. Modify the configs section and secrets section in docker-compose.yml to include name of the Docker config and Docker secret created above.
  2. Uncomment the relevant lines in the nginx section in docker-compose.yml.