Commit 99cbd169 authored by Anastasios Lisgaras's avatar Anastasios Lisgaras

Merge branch 'devel' into 'gitlab-mirror'

# Conflicts:
#   README.md
parents c7607ec2 7660ff68
*~
.DS_Store
setup.sh
.*.sw?
roles/has_certificate/files/*.key
......@@ -12,115 +12,43 @@ The administrator of the ARGO product being deployed via these Ansible playbooks
- `host_vars/{inventory_hostname}`
Per ARGO product more details on prerequisites and variables are given in the following subsections.
## Sysprep the VM
- Disable Selinux: Vi /etc/sycoconfig/selinux change SELINUX=enforcing to SELINUX=disabled
- Allocate interfaces to zones: e.g. firewall-cmd --zone=internal --change-interface=eth2 --permanent
- Upload Public keys inclunding GRNET_CI
- Reboot
## WebAPI deployment
## Run or Develop Ansible Playbooks
Contains Ansible playbook for the deployment of the ARGO datastore and API service. The play is split into four (4) roles:
- repos (includes tasks for the installation of the required repository definitions)
- has_certificate (task for uploading the certificate file onto the host under the appropriate path)
- mongodb (installation and configuration of mongodb datastore)
- webapi (installation and bootstrap of ARGO api service)
- In order to run an ansible playbook, you need to make sure that you are using `ansible 2.6`.
### Things to do before deployment
- In order to develop new playbooks you will also need to have `docker` and `molecule`.
- Obtain a key/certificate pair from a trusted CA and after place them both under roles/has_certificate/files with names `{{inventory_hostname}}.key` and `{{inventory_hostname}}.pem` respectively. As `{{inventory_hostname}}` use the exact name used within the `inventory` file.
- Edit inventory and replace `webapi.node` with the hostname that you intend to deploy the API onto.
To make the set up process easier, you will have to create a virtual environment executing the following steps:
### Prerequisites
- Make sure you have `python2.7` installed
- Deploy against CentOS 6.x node
- Make sure `libselinux-python` is installed on the target node
- Ansible version used is `1.7.2`
- Update `pip`
### How to deploy
`pip install --upgrade pip`
```bash
$ ansible-playbook -v webapi.yml
```
- Install the virtualenv package
`pip install virtualenv`
- Create the new virtual environment
## Web UI deployment
`virtualenv --python=/usr/bin/python2.7 ./argo-ansible-env`
Contains Ansible playbook for the deployment of the ARGO Web UI service. The play is split into four (4) roles:
- firewall (configures iptables firewall rules)
- repos (includes tasks for the installation of the required repository definitions)
- has_certificate (task for uploading the certificate file onto the host under the appropriate path)
- webui (installation and bootstrap of ARGO Web UI service)
- Navigate inside the virtual environment and activate it
### Things to do before deployment
`cd argo-ansible-env && source ./bin/activate`
- Obtain a key/certificate pair from a trusted CA and after place them both under roles/has_certificate/files with names `{{inventory_hostname}}.key` and `{{inventory_hostname}}.pem` respectively. As `{{inventory_hostname}}` use the exact name used within the `inventory` file.
- Edit inventory and replace `webui.node` with the hostname that you intend to deploy the Web UI onto.
- Edit `roles/webui/vars/main.yml` file and change the values of the `certificate_password` and `keystore_password` variables to a stronger value.
- Clone the repo and install the appropriate packages
- Note that by default the EGI based web UI will be deployed on your target node. To change this behaviour use the `argo_web` and `branch_name` variables within the `roles/webui/vars/main.yml` file to point to another upstream lavoisier repository.
After cloning the repo,navigate inside it, and issue the command
### Prerequisites
`pip install -r requirements.txt`
- Deploy against CentOS 7.x node
- Ansible version used is `1.9.2`
- After setting up your environment, you will also need some pre-defined roles that our playbooks are using.To get these roles, issue the command:
### How to deploy
```bash
$ ansible-playbook -v webui.yml
```
## POEM deployment
Contains Ansible playbook for the deployment of the ARGO POEM service. The play is split into four (4) roles:
- firewall (configures iptables firewall rules)
- repos (includes tasks for the installation of the required repository definitions)
- has_certificate (task for uploading the certificate file onto the host under the appropriate path)
- poem (installs and bootstraps poem service)
### Things to do before deployment
- Obtain a key/certificate pair from a trusted CA and after place them both under roles/has_certificate/files with names `{{inventory_hostname}}.key` and `{{inventory_hostname}}.pem` respectively. As `{{inventory_hostname}}` use the exact name used within the `inventory` file.
- Edit inventory and replace `poem.node` with the hostname that you intend to deploy the POEM service onto.
- Create a `host_vars/{{inventory_hostname}}` file and place therein the variables found within the `roles/poem/defaults/main.yml` file in order to overwrite them.
- In order to generate a uuid to be used in the place of the `poem_secret` variable you may use the `uuidgen` linux cli utility.
### Prerequisites
- Deploy against CentOS 6.x node
- Make sure `libselinux-python` is installed on the target node
- Ansible version used is `1.9.2`
### How to deploy
```bash
$ ansible-playbook -v poem.yml
```
## Full standalone deployment
Contains Ansible playbook for the deployment of all ARGO components. The play is split into six (6) roles:
- repos (includes tasks for the installation of the required repository definitions)
- ca_bundle (includes a task for the installation of the egi-ca-policy-core bundle)
- has_certificate (task for uploading the certificate file onto the host under the appropriate path)
- consumer (includes tasks for the installation of the ARGO consumer and feed components)
- mongodb (installation and configuration of mongodb datastore)
- webapi (installation and bootstrap of ARGO api service)
### Things to do before deployment
- Obtain a key/certificate pair from a trusted CA and after place them both under roles/has_certificate/files with names `{{inventory_hostname}}.key` and `{{inventory_hostname}}.pem` respectively. As `{{inventory_hostname}}` use the exact name used within the `inventory` file.
- Edit inventory and replace `standalone.node` with the hostname that you intend to deploy the complete ARGO stack onto.
### Prerequisites
- Deploy against CentOS 6.x node
- Make sure `libselinux-python` is installed on the target node
- Ansible version used is `1.7.2`
### How to deploy
```bash
$ ansible-playbook -v standalone.yml
```
## Monitoring your services
In case you are using Nagios or Icinga for health monitoring purposes a minimal `is_monitored` role is included in the repo. The puspose of this role is to install and configure the nrpe service on your target machines. Modify the remote host variable within the `roles/is_monitored/defaults/main.yml` file and include it in your playbooks.
`ansible-galaxy install -r requirements.yml`
#
# config file for ansible
# https://raw.githubusercontent.com/ansible/ansible/devel/examples/ansible.cfg
#
[defaults]
remote_user = root
hostfile = inventory
\ No newline at end of file
---
# Variable enabled_argo_repo specifies which RPM repository to use.
# To use the development repository set its value to argo-devel
enabled_argo_repo: argo-prod
cert_dir: /etc/grid-security
---
cert_path: /etc/pki/tls/certs/localhost.crt
key_path: /etc/pki/tls/private/localhost.key
ca_path: /etc/pki/tls/certs/ca-bundle.crt
iptables_rules:
input:
- { dport: "80", proto: "tcp", policy: "accept"}
- { dport: "443", proto: "tcp", policy: "accept"}
nagios_plugins:
- { name: nagios-plugins-tcp , repo: "" }
- { name: nagios-plugins-disk , repo: "" }
- { name: nagios-plugins-http , repo: "" }
- { name: nagios-plugins , repo: "" }
- { name: nagios-plugins-dummy , repo: "" }
- { name: nagios-plugins-procs , repo: "" }
- { name: nagios-plugins-ping , repo: "" }
\ No newline at end of file
---
iptables_rules:
input:
- { dport: "443", proto: "tcp", policy: "accept"}
---
mongo_bind_interfaces: 127.0.0.1
cert_path: /etc/grid-security/hostcert.pem
key_path: /etc/grid-security/hostkey.pem
iptables_rules:
input:
- { dport: "443", proto: "tcp", policy: "accept"}
hive_retention_in_days: 900
mongo_retention_in_days: 900
files_retention_in_days: 900
---
mongo_bind_interfaces: 0.0.0.0
cert_path: /etc/pki/tls/certs/localhost.crt
key_path: /etc/pki/tls/private/localhost.key
iptables_rules:
input:
- { dport: "443", proto: "tcp", policy: "accept"}
- { dport: "27017", proto: "tcp", policy: "accept"}
---
- hosts: all
become: yes
roles:
- { role: commons, task: timezone, tags: timezone }
- { role: commons, task: repos, tags: repos }
- { role: commons, task: basic_utils, tags: basic_utils }
- { role: commons, task: users, tags: groups_users_sshKeys }
- { role: commons, task: sshd, tags: sshd }
- { role: commons, task: firewall, tags: firewall }
- { role: commons, task: fail2ban, tags: fail2ban_conf }
- { role: commons, task: cert, tags: cert }
- { role: commons, task: rsyslog, tags: rsyslog_conf }
- { role: nickhammond.logrotate, tags: logrotate }
# - { role: commons, task: is_monitored, tags: monitored }
# - { role: commons, task: backupamsmongo, tags: rsyslog_conf }
- hosts: connectors
become: yes
roles:
- { role: consumer, task: connectors, tags: connectors }
- { role: consumer, task: cron_jobs, tags: cron_jobs }
- { role: consumer, task: delete_files, tags: delete_files }
- hosts: archiver
become: yes
roles:
- { role: archiver, task: archiver_setup, tags: archiver_setup }
- hosts: poem
become: yes
roles:
- { role: poem, tags: poem}
- { role: httpd, tags: httpd }
- hosts: haproxy
become: yes
roles:
- { role: haproxy, task: rsyslog }
- { role: haproxy, task: haproxy , tags: haproxy_install }
- hosts: ams_store
become: yes
roles:
- { role: private_hosts }
- { role: mongodb }
- { role: ams, task: init_db, tags: ams_install }
- hosts: ams
become: yes
roles:
- { role: private_hosts }
- { role: zookeeper, tags: zookeeper_install }
- { role: kafka, tags: kafka_install }
- { role: ams, task: deploy, tags: ams_install }
- { role: ams, task: deploy_metrics, tags: ams_install }
- hosts: ams_push_server
become: yes
roles:
- { role: push-server, task: push-server-setup, tags: push_install }
- hosts: authn
become: yes
roles:
- { role: mongodb }
- { role: argo-api-authn, task: authn-setup }
- { role: argo-api-authn, task: python-env-setup }
- { role: argo-api-authn, task: ams-create-users-gocdb-script }
- { role: argo-api-authn, task: ams-create-users-cloud-info-script }
- hosts: metrics
become: yes
roles:
- { role: metrics, task: ams-metrics-ui }
- { role: metrics, task: ce_comp }
- hosts: monbox
become: yes
roles:
- { role: monbox, task: deploy }
- { role: monbox, task: config }
- hosts: alerta
become: yes
roles:
- { role: mongodb }
- { role: alerta, task: deploy, tags: alerta }
- hosts: webapi
become: yes
roles:
- { role: mongodb }
- { role: webapi, task: deploy }
- { role: webapi, task: init_api }
- hosts: swagger
become: yes
roles:
- { role: httpd, tags: httpd }
- { role: swagger }
- hosts: c_cluster
become: yes
roles:
- { role: private_hosts, tags: private_hosts }
- hosts: c_gateway
become: yes
roles:
- { role: squid }
- { role: cloudera_gateway}
- hosts: c_private
become: yes
roles:
- { role: through_http_proxy }
- { role: cloudera_internal_node }
- { role: disable_ipv6, tags: disable_ipv6 }
- hosts: c_manager
become: yes
roles:
- { role: cloudera_manager }
- hosts: c_flink
become: yes
roles:
- { role: flink, tags: deploy_flink }
[webapi]
webapi.node
[standalone]
standalone.node
[poem]
poem.node
[webui]
webui.node
[monitoring_engine]
monitoring_engine.node
\ No newline at end of file
---
- hosts: monitoring_engine
sudo: true
roles:
- { role: firewall, tags: firewall }
- { role: repos, tags: repos }
- { role: ca_bundle, when: ca_bundle_install, tags: ca_bundle }
- { role: has_certificate, tags: certificate }
- { role: monitoring_engine, tags: monitoring_engine }
---
- hosts: poem
sudo: true
roles:
- { role: firewall, tags: firewall }
- { role: repos, tags: repos }
- { role: has_certificate, tags: certificate }
- { role: poem, tags: poem }
# install some needed roles from galaxy
# basic modifications for increased security
- src: dev-sec.os-hardening
# role for handling logrotate configuration
- src: nickhammond.logrotate
extends: default
rules:
braces:
max-spaces-inside: 1
level: error
brackets:
max-spaces-inside: 1
level: error
line-length: disable
# NOTE(retr0h): Templates no longer fail this lint rule.
# Uncomment if running old Molecule templates.
# truthy: disable
Role Name
=========
Role to setup an alerta node with the following services
- alerta-server
- alerta-web-ui
- alerta-amqp plugin
- alerta-mailer
- mongodb
- argo-alert-publisher service