Commit 99cbd169 authored by Anastasios Lisgaras's avatar Anastasios Lisgaras
Browse files

Merge branch 'devel' into 'gitlab-mirror'

# Conflicts:
#   README.md
parents c7607ec2 7660ff68
*~
.DS_Store
setup.sh
.*.sw?
roles/has_certificate/files/*.key
......@@ -12,115 +12,43 @@ The administrator of the ARGO product being deployed via these Ansible playbooks
- `host_vars/{inventory_hostname}`
Per ARGO product more details on prerequisites and variables are given in the following subsections.
## Sysprep the VM
- Disable Selinux: Vi /etc/sycoconfig/selinux change SELINUX=enforcing to SELINUX=disabled
- Allocate interfaces to zones: e.g. firewall-cmd --zone=internal --change-interface=eth2 --permanent
- Upload Public keys inclunding GRNET_CI
- Reboot
## WebAPI deployment
## Run or Develop Ansible Playbooks
Contains Ansible playbook for the deployment of the ARGO datastore and API service. The play is split into four (4) roles:
- repos (includes tasks for the installation of the required repository definitions)
- has_certificate (task for uploading the certificate file onto the host under the appropriate path)
- mongodb (installation and configuration of mongodb datastore)
- webapi (installation and bootstrap of ARGO api service)
- In order to run an ansible playbook, you need to make sure that you are using `ansible 2.6`.
### Things to do before deployment
- In order to develop new playbooks you will also need to have `docker` and `molecule`.
- Obtain a key/certificate pair from a trusted CA and after place them both under roles/has_certificate/files with names `{{inventory_hostname}}.key` and `{{inventory_hostname}}.pem` respectively. As `{{inventory_hostname}}` use the exact name used within the `inventory` file.
- Edit inventory and replace `webapi.node` with the hostname that you intend to deploy the API onto.
To make the set up process easier, you will have to create a virtual environment executing the following steps:
### Prerequisites
- Make sure you have `python2.7` installed
- Deploy against CentOS 6.x node
- Make sure `libselinux-python` is installed on the target node
- Ansible version used is `1.7.2`
- Update `pip`
### How to deploy
`pip install --upgrade pip`
```bash
$ ansible-playbook -v webapi.yml
```
- Install the virtualenv package
`pip install virtualenv`
- Create the new virtual environment
## Web UI deployment
`virtualenv --python=/usr/bin/python2.7 ./argo-ansible-env`
Contains Ansible playbook for the deployment of the ARGO Web UI service. The play is split into four (4) roles:
- firewall (configures iptables firewall rules)
- repos (includes tasks for the installation of the required repository definitions)
- has_certificate (task for uploading the certificate file onto the host under the appropriate path)
- webui (installation and bootstrap of ARGO Web UI service)
- Navigate inside the virtual environment and activate it
### Things to do before deployment
`cd argo-ansible-env && source ./bin/activate`
- Obtain a key/certificate pair from a trusted CA and after place them both under roles/has_certificate/files with names `{{inventory_hostname}}.key` and `{{inventory_hostname}}.pem` respectively. As `{{inventory_hostname}}` use the exact name used within the `inventory` file.
- Edit inventory and replace `webui.node` with the hostname that you intend to deploy the Web UI onto.
- Edit `roles/webui/vars/main.yml` file and change the values of the `certificate_password` and `keystore_password` variables to a stronger value.
- Clone the repo and install the appropriate packages
- Note that by default the EGI based web UI will be deployed on your target node. To change this behaviour use the `argo_web` and `branch_name` variables within the `roles/webui/vars/main.yml` file to point to another upstream lavoisier repository.
After cloning the repo,navigate inside it, and issue the command
### Prerequisites
`pip install -r requirements.txt`
- Deploy against CentOS 7.x node
- Ansible version used is `1.9.2`
- After setting up your environment, you will also need some pre-defined roles that our playbooks are using.To get these roles, issue the command:
### How to deploy
```bash
$ ansible-playbook -v webui.yml
```
## POEM deployment
Contains Ansible playbook for the deployment of the ARGO POEM service. The play is split into four (4) roles:
- firewall (configures iptables firewall rules)
- repos (includes tasks for the installation of the required repository definitions)
- has_certificate (task for uploading the certificate file onto the host under the appropriate path)
- poem (installs and bootstraps poem service)
### Things to do before deployment
- Obtain a key/certificate pair from a trusted CA and after place them both under roles/has_certificate/files with names `{{inventory_hostname}}.key` and `{{inventory_hostname}}.pem` respectively. As `{{inventory_hostname}}` use the exact name used within the `inventory` file.
- Edit inventory and replace `poem.node` with the hostname that you intend to deploy the POEM service onto.
- Create a `host_vars/{{inventory_hostname}}` file and place therein the variables found within the `roles/poem/defaults/main.yml` file in order to overwrite them.
- In order to generate a uuid to be used in the place of the `poem_secret` variable you may use the `uuidgen` linux cli utility.
### Prerequisites
- Deploy against CentOS 6.x node
- Make sure `libselinux-python` is installed on the target node
- Ansible version used is `1.9.2`
### How to deploy
```bash
$ ansible-playbook -v poem.yml
```
## Full standalone deployment
Contains Ansible playbook for the deployment of all ARGO components. The play is split into six (6) roles:
- repos (includes tasks for the installation of the required repository definitions)
- ca_bundle (includes a task for the installation of the egi-ca-policy-core bundle)
- has_certificate (task for uploading the certificate file onto the host under the appropriate path)
- consumer (includes tasks for the installation of the ARGO consumer and feed components)
- mongodb (installation and configuration of mongodb datastore)
- webapi (installation and bootstrap of ARGO api service)
### Things to do before deployment
- Obtain a key/certificate pair from a trusted CA and after place them both under roles/has_certificate/files with names `{{inventory_hostname}}.key` and `{{inventory_hostname}}.pem` respectively. As `{{inventory_hostname}}` use the exact name used within the `inventory` file.
- Edit inventory and replace `standalone.node` with the hostname that you intend to deploy the complete ARGO stack onto.
### Prerequisites
- Deploy against CentOS 6.x node
- Make sure `libselinux-python` is installed on the target node
- Ansible version used is `1.7.2`
### How to deploy
```bash
$ ansible-playbook -v standalone.yml
```
## Monitoring your services
In case you are using Nagios or Icinga for health monitoring purposes a minimal `is_monitored` role is included in the repo. The puspose of this role is to install and configure the nrpe service on your target machines. Modify the remote host variable within the `roles/is_monitored/defaults/main.yml` file and include it in your playbooks.
`ansible-galaxy install -r requirements.yml`
#
# config file for ansible
# https://raw.githubusercontent.com/ansible/ansible/devel/examples/ansible.cfg
#
[defaults]
remote_user = root
hostfile = inventory
\ No newline at end of file
---
# Variable enabled_argo_repo specifies which RPM repository to use.
# To use the development repository set its value to argo-devel
enabled_argo_repo: argo-prod
cert_dir: /etc/grid-security
---
cert_path: /etc/pki/tls/certs/localhost.crt
key_path: /etc/pki/tls/private/localhost.key
ca_path: /etc/pki/tls/certs/ca-bundle.crt
iptables_rules:
input:
- { dport: "80", proto: "tcp", policy: "accept"}
- { dport: "443", proto: "tcp", policy: "accept"}
nagios_plugins:
- { name: nagios-plugins-tcp , repo: "" }
- { name: nagios-plugins-disk , repo: "" }
- { name: nagios-plugins-http , repo: "" }
- { name: nagios-plugins , repo: "" }
- { name: nagios-plugins-dummy , repo: "" }
- { name: nagios-plugins-procs , repo: "" }
- { name: nagios-plugins-ping , repo: "" }
\ No newline at end of file
---
iptables_rules:
input:
- { dport: "443", proto: "tcp", policy: "accept"}
---
mongo_bind_interfaces: 127.0.0.1
cert_path: /etc/grid-security/hostcert.pem
key_path: /etc/grid-security/hostkey.pem
iptables_rules:
input:
- { dport: "443", proto: "tcp", policy: "accept"}
hive_retention_in_days: 900
mongo_retention_in_days: 900
files_retention_in_days: 900
---
mongo_bind_interfaces: 0.0.0.0
cert_path: /etc/pki/tls/certs/localhost.crt
key_path: /etc/pki/tls/private/localhost.key
iptables_rules:
input:
- { dport: "443", proto: "tcp", policy: "accept"}
- { dport: "27017", proto: "tcp", policy: "accept"}
---
- hosts: all
become: yes
roles:
- { role: commons, task: timezone, tags: timezone }
- { role: commons, task: repos, tags: repos }
- { role: commons, task: basic_utils, tags: basic_utils }
- { role: commons, task: users, tags: groups_users_sshKeys }
- { role: commons, task: sshd, tags: sshd }
- { role: commons, task: firewall, tags: firewall }
- { role: commons, task: fail2ban, tags: fail2ban_conf }
- { role: commons, task: cert, tags: cert }
- { role: commons, task: rsyslog, tags: rsyslog_conf }
- { role: nickhammond.logrotate, tags: logrotate }
# - { role: commons, task: is_monitored, tags: monitored }
# - { role: commons, task: backupamsmongo, tags: rsyslog_conf }
- hosts: connectors
become: yes
roles:
- { role: consumer, task: connectors, tags: connectors }
- { role: consumer, task: cron_jobs, tags: cron_jobs }
- { role: consumer, task: delete_files, tags: delete_files }
- hosts: archiver
become: yes
roles:
- { role: archiver, task: archiver_setup, tags: archiver_setup }
- hosts: poem
become: yes
roles:
- { role: poem, tags: poem}
- { role: httpd, tags: httpd }
- hosts: haproxy
become: yes
roles:
- { role: haproxy, task: rsyslog }
- { role: haproxy, task: haproxy , tags: haproxy_install }
- hosts: ams_store
become: yes
roles:
- { role: private_hosts }
- { role: mongodb }
- { role: ams, task: init_db, tags: ams_install }
- hosts: ams
become: yes
roles:
- { role: private_hosts }
- { role: zookeeper, tags: zookeeper_install }
- { role: kafka, tags: kafka_install }
- { role: ams, task: deploy, tags: ams_install }
- { role: ams, task: deploy_metrics, tags: ams_install }
- hosts: ams_push_server
become: yes
roles:
- { role: push-server, task: push-server-setup, tags: push_install }
- hosts: authn
become: yes
roles:
- { role: mongodb }
- { role: argo-api-authn, task: authn-setup }
- { role: argo-api-authn, task: python-env-setup }
- { role: argo-api-authn, task: ams-create-users-gocdb-script }
- { role: argo-api-authn, task: ams-create-users-cloud-info-script }
- hosts: metrics
become: yes
roles:
- { role: metrics, task: ams-metrics-ui }
- { role: metrics, task: ce_comp }
- hosts: monbox
become: yes
roles:
- { role: monbox, task: deploy }
- { role: monbox, task: config }
- hosts: alerta
become: yes
roles:
- { role: mongodb }
- { role: alerta, task: deploy, tags: alerta }
- hosts: webapi
become: yes
roles:
- { role: mongodb }
- { role: webapi, task: deploy }
- { role: webapi, task: init_api }
- hosts: swagger
become: yes
roles:
- { role: httpd, tags: httpd }
- { role: swagger }
- hosts: c_cluster
become: yes
roles:
- { role: private_hosts, tags: private_hosts }
- hosts: c_gateway
become: yes
roles:
- { role: squid }
- { role: cloudera_gateway}
- hosts: c_private
become: yes
roles:
- { role: through_http_proxy }
- { role: cloudera_internal_node }
- { role: disable_ipv6, tags: disable_ipv6 }
- hosts: c_manager
become: yes
roles:
- { role: cloudera_manager }
- hosts: c_flink
become: yes
roles:
- { role: flink, tags: deploy_flink }
[webapi]
webapi.node
[standalone]
standalone.node
[poem]
poem.node
[webui]
webui.node
[monitoring_engine]
monitoring_engine.node
\ No newline at end of file
---
- hosts: monitoring_engine
sudo: true
roles:
- { role: firewall, tags: firewall }
- { role: repos, tags: repos }
- { role: ca_bundle, when: ca_bundle_install, tags: ca_bundle }
- { role: has_certificate, tags: certificate }
- { role: monitoring_engine, tags: monitoring_engine }
---
- hosts: poem
sudo: true
roles:
- { role: firewall, tags: firewall }
- { role: repos, tags: repos }
- { role: has_certificate, tags: certificate }
- { role: poem, tags: poem }
# install some needed roles from galaxy
# basic modifications for increased security
- src: dev-sec.os-hardening
# role for handling logrotate configuration
- src: nickhammond.logrotate
extends: default
rules:
braces:
max-spaces-inside: 1
level: error
brackets:
max-spaces-inside: 1
level: error
line-length: disable
# NOTE(retr0h): Templates no longer fail this lint rule.
# Uncomment if running old Molecule templates.
# truthy: disable
Role Name
=========
Role to setup an alerta node with the following services
- alerta-server
- alerta-web-ui
- alerta-amqp plugin
- alerta-mailer
- mongodb
- argo-alert-publisher service
- argo-rulegen service
- supervisord
- nginx (https enabled)
- uwsgi
Requirements
------------
- OS: CentOS 7.x
- certificate already issued
- selinux disabled
Role Variables
--------------
alerta_admin_list: a list of expected alerta admin usernames
alerta_plugin_list: a list of expected alerta enabled plugins
alerta_allowed_env_list: a list of expected alerta enabled environments
alerta_server_dir: path to install alerta server and python 3 virtual env
alerta_token: alerta service token to be used for cli/api access
alerta_dashboard_dir: '/var/www/html/'
alerta_dashboard_repo: https://github.com/alerta/alerta-webui.git
alerta_dashboard_release: version of dashboard
alerta_amqp_release: version (commit hash) of amqp plugin
alerta_mailer_release: version (commit hash) of mailer intergration
alerta_secret_key: alerta secret key hash
alerta: alerta version
alerta_server_version: alerta server version
alerta_main_mail: mail mail for notifications
alerta_mail_from: mail from
alerta_smtp_host: smtp host used
alerta_smtp_port: smtp port used
ssmtp_debug: if smpt will run in debug log mode
alerta_tenants: - list of alerta tenants
eudat: - tenant name
gocdb_auth_method: 'cert' - authentication method to gocdb
gocdb_api_endpoint: gocdb endpoint to call
gocdb_notification_flag: True/False if notification flag will be used
gocdb_verify: True/False ssl verify gocdb connection
gocdb_top_request: gocdb request pattern for top level group
gocdb_sub_request: gocdb request pattern for second level group
mail_template: path to mail text template to be used
mail_template_html: path to html template to be used
mail_type: html/text
mail_debug: true
alerta_environment: environment name to be used for each alert
alert_extra_emails: a list of extra emails to be notified
alert_timeout: how much until the alert to be considered stale
alert_group: top level group name to be used in alerts
ui_endpoint: url to the argo ui endpoint that serves a/r results
Dependencies
------------
Dependent on mongo role using repo_mongo_4x: true
Example Playbook
----------------
Including an example of how to use your role (for instance, with variables
passed in as parameters) is always nice for users too:
- hosts: alerta
become: yes
roles:
- { role: mongodb }
- { role: alerta, task: deploy, tags: alerta }
License
-------
Apache 2
Author Information
------------------
GRNET
---
ssmtp_debug: YES;
alerta_admin_list: ['admin@localhost']
alerta_plugin_list: ['reject','amqp']
alerta_allowed_env_list: ['production']
alerta_dashboard_dir: '/var/www/html/'
alerta_dashboard_repo: https://github.com/alerta/alerta-webui.git
alerta_dashboard_release: 'v7.2.11'
alerta_amqp_release: '8b1b3b3'
alerta_mailer_release: '74834e6'
alerta_secret_key: 'HOf%A)w6de5oJXJ^F=Jb@rfz4YDRKui9'
alerta_server_dir: '/opt/alerta/'
alerta_token: ''
alerta_www_url: 'localhost'
alerta_www_alias: 'localhost'
alerta_main_mail: ''
alerta_mail_from: ''
alerta_smtp_host: ''
alerta_smtp_port: 587
alerta_mail_debug: true
alerta_version: 7.2.2
alerta_server_version: 7.2.11
# default: use pip to install argo-alert from github repo using devel branch
argo_alert_repo: 'git+https://github.com/ARGOeu/argo-alert@devel#egg=argoalert'
argo_alert_kafka: 'kafka.node.foo'
ca_bundle: true
cert_path: /etc/grid-security/hostcert.pem
cert_key_path: /etc/grid-security/hostkey.pem
cert_dir: /etc/grid-security/
ca_path_bundle: /etc/pki/tls/certs/ca-bundle.crt
alerta_tenants:
foo:
ui_endpoint: argo_ui_hostname
contact_api_type: 'contact.api.type'
contact_api_endpoint: 'https://contact.api.foo'
mail_template: ~/alerta-foo.tmpl
mail_template_html: ~/alerta-foo.html.tmpl
mail_type: html
mail_debug: true
mail_rule: ~/alerta-foo/mail-rules/mail-foo-rules.json
smtp_debug: true
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server {
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
location /api { try_files $uri @api; }
location @api {
include uwsgi_params;
uwsgi_pass unix:/var/run/uwsgi/uwsgi.sock;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
ssl_certificate "/etc/grid-security/hostcert.pem";
ssl_certificate_key "/etc/grid-security/hostkey.pem";
ssl on;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
location / {
root /var/www/html;
try_files $uri $uri/ /index.html;
}
}
server_names_hash_bucket_size 64;
}
\ No newline at end of file