SOAK Testing deployment

Owned by Frederic Randriamora (Unlicensed)
Last updated: 30 Sept 2018

About

This doc describes how we deploy and run the tests

Deploy the instance

  • using ansible
TIS-DEVOPS $ ansible-playbook -e NUMTHREADS=1 -e DURATION=120 -i ansible/inventory/dev ansible/soaktest.yml


  • using jenkins

https://build.tis.nhs.uk/jenkins/job/tis-soaktest/

That is all we need to get soak testing running. Below is how we set the system up.

Read the logs

  • manually

For DEV, discover what ETL instance IP is

$ cd TIS-DEVOPS
$ ./ansible/inventory/dev
$ ssh -v 10.140.0.136
$ less /tmp/Jmeter/jmeter.log


Create docker image

Currently this is a hack

The procedure: 

  • create ansible/docker-compose/templates/soaktest/Dockerfile and ansible/docker-compose/templates/soaktest/docker-compose.yml
  • deploy somewhere
  • create an image from the running instance, then push the iage to the docker registry

Dockerfile

FROM ubuntu:18.04
ENV LC_ALL C
ENV DEBIAN_FRONTEND noninteractive
ENV DEBCONF_NONINTERACTIVE_SEEN true
RUN apt-get update && apt-get install -y gnupg wget
RUN apt-get install -y openssh-server
RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
# SSH login fix. Otherwise user is kicked off after login
RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd

ENV NOTVISIBLE "in users profile"
RUN echo "export VISIBLE=now" >> /etc/profile

EXPOSE 22
RUN mkdir -p /run/sshd
RUN echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" > /etc/apt/sources.list.d/webupd8team-java.list && \
    echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main" >> /etc/apt/sources.list.d/webupd8team-java.list && \
    apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886 && \
    apt-get -y update && \
    echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \
    apt-get install -y oracle-java8-installer && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* /var/cache/oracle-jdk8-installer
ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
RUN echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" | tee /etc/apt/sources.list.d/google-chrome.list
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN apt-get update && apt-get install -y google-chrome-stable && mkdir /Jmeter && chmod 755 -R /Jmeter
RUN useradd -ms /bin/bash soaktester
#RUN chmod +x /Jmeter/bin/jmeter
#USER soaktester
#COPY /tmp/Jmeter/* /Jmeter/
ENV JMETER_HOME=/Jmeter
ENV PATH $JMETER_HOME/bin:$PATH
#VOLUME /result
#VOLUME /Jmeter
ENV DISPLAY=:1
WORKDIR /Jmeter
CMD ["/usr/sbin/sshd", "-D"]
#ENTRYPOINT ["sh", "-c", "./bin/jmeter -n -t ./SoakTestPlan.jmx -l /result/TestResult.jtl"]


docker-compose.yml

version: '2'

services:
  soaktest:
    restart: "on-failure:1"
    build: .
    logging:
      options:
        max-size: '500m'
        max-file: '3'
    cap_add:
    - SYS_ADMIN
    volumes:
      - /tmp/Jmeter:/Jmeter
      - /dev/shm:/dev/shm # Mitigates the Chromium issue described at https://code.google.com/p/chromium/issues/detail?id=519952

then we can run

$ ansible-playbook -i ansible/inventory/<ENV> ansible/soaktest.yml

then enter the docker instance and do this hack

# cp /opt/google/chrome/google-chrome /opt/google/chrome/google-chrome.real
# echo '/opt/google/chrome/google-chrome.real --headless --no-gui "$@"' > /opt/google/chrome/google-chrome
# su - soaktester

Create and upload the image to our docker repo

$ docker commit  -a "@TIS-DEVOPS" -m "SoakTest" soaktest_soaktest_1 soaktest
$ docker tag soaktest repository.tis.nhs.uk:5000/hee/soaktest:0.1.0
$ docker push repository.tis.nhs.uk:5000/hee/soaktest:0.1.0

Soak File

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="4.0" jmeter="4.0 r1823414">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="TIS-KC-Soak-Test-Plan-DEV" enabled="true">
      <stringProp name="TestPlan.comments"></stringProp>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp name="TestPlan.user_define_classpath"></stringProp>
    </TestPlan>
    <hashTree>
      <com.googlecode.jmeter.plugins.webdriver.config.ChromeDriverConfig guiclass="com.googlecode.jmeter.plugins.webdriver.config.gui.ChromeDriverConfigGui" testclass="com.googlecode.jmeter.plugins.webdriver.config.ChromeDriverConfig" testname="Chrome Driver Config" enabled="true">
        <stringProp name="WebDriverConfig.proxy_type">SYSTEM</stringProp>
        <stringProp name="WebDriverConfig.proxy_pac_url"></stringProp>
        <stringProp name="WebDriverConfig.http_host"></stringProp>
        <intProp name="WebDriverConfig.http_port">8080</intProp>
        <boolProp name="WebDriverConfig.use_http_for_all_protocols">true</boolProp>
        <stringProp name="WebDriverConfig.https_host"></stringProp>
        <intProp name="WebDriverConfig.https_port">8080</intProp>
        <stringProp name="WebDriverConfig.ftp_host"></stringProp>
        <intProp name="WebDriverConfig.ftp_port">8080</intProp>
        <stringProp name="WebDriverConfig.socks_host"></stringProp>
        <intProp name="WebDriverConfig.socks_port">8080</intProp>
        <stringProp name="WebDriverConfig.no_proxy">localhost</stringProp>
        <boolProp name="WebDriverConfig.maximize_browser">true</boolProp>
        <boolProp name="WebDriverConfig.reset_per_iteration">false</boolProp>
        <boolProp name="WebDriverConfig.dev_mode">false</boolProp>
        <stringProp name="ChromeDriverConfig.chromedriver_path">chromedriver.2.39</stringProp>
        <boolProp name="ChromeDriverConfig.android_enabled">false</boolProp>
        <boolProp name="ChromeDriverConfig.headless_enabled">false</boolProp>
      </com.googlecode.jmeter.plugins.webdriver.config.ChromeDriverConfig>
      <hashTree/>
      <CacheManager guiclass="CacheManagerGui" testclass="CacheManager" testname="HTTP Cache Manager" enabled="true">
        <boolProp name="clearEachIteration">true</boolProp>
        <boolProp name="useExpires">true</boolProp>
      </CacheManager>
      <hashTree/>
      <CookieManager guiclass="CookiePanel" testclass="CookieManager" testname="HTTP Cookie Manager" enabled="true">
        <collectionProp name="CookieManager.cookies"/>
        <boolProp name="CookieManager.clearEachIteration">true</boolProp>
      </CookieManager>
      <hashTree/>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="KC-Threads" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <intProp name="LoopController.loops">-1</intProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">${__P(num_threads,1)}</stringProp>
        <stringProp name="ThreadGroup.ramp_time">1</stringProp>
        <boolProp name="ThreadGroup.scheduler">true</boolProp>
        <stringProp name="ThreadGroup.duration">${__P(duration,120)}</stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
      </ThreadGroup>
      <hashTree>
        <com.googlecode.jmeter.plugins.webdriver.sampler.WebDriverSampler guiclass="com.googlecode.jmeter.plugins.webdriver.sampler.gui.WebDriverSamplerGui" testclass="com.googlecode.jmeter.plugins.webdriver.sampler.WebDriverSampler" testname="Open TIS" enabled="true">
          <stringProp name="WebDriverSampler.script">var pkg = JavaImporter(org.openqa.selenium)
var support_ui = JavaImporter(org.openqa.selenium.support.ui.WebDriverWait)
var wait = new support_ui.WebDriverWait(WDS.browser, 5000)
var conditions = org.openqa.selenium.support.ui.ExpectedConditions
WDS.sampleResult.sampleStart()
try {
    WDS.browser.get('${__P(myurl,https://dev-apps.tis.nhs.uk/)}')
    wait.until(conditions.presenceOfElementLocated(pkg.By.id('username')))
    WDS.sampleResult.sampleEnd()
} catch(ex) {
    WDS.sampleResult.sampleEnd()
    WDS.sampleResult.setSuccessful(false);
    WDS.sampleResult.setResponseMessage("Failed to load page, timeout occured");
}</stringProp>
          <stringProp name="WebDriverSampler.parameters"></stringProp>
          <stringProp name="WebDriverSampler.language">javascript</stringProp>
        </com.googlecode.jmeter.plugins.webdriver.sampler.WebDriverSampler>
        <hashTree>
          <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
            <collectionProp name="Asserion.test_strings">
              <stringProp name="-857427365">TRAINEE INFORMATION SYSTEM</stringProp>
            </collectionProp>
            <stringProp name="Assertion.custom_message"></stringProp>
            <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
            <boolProp name="Assertion.assume_success">false</boolProp>
            <intProp name="Assertion.test_type">2</intProp>
          </ResponseAssertion>
          <hashTree/>
        </hashTree>
        <LoopController guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">true</boolProp>
          <intProp name="LoopController.loops">-1</intProp>
        </LoopController>
        <hashTree>
          <com.googlecode.jmeter.plugins.webdriver.sampler.WebDriverSampler guiclass="com.googlecode.jmeter.plugins.webdriver.sampler.gui.WebDriverSamplerGui" testclass="com.googlecode.jmeter.plugins.webdriver.sampler.WebDriverSampler" testname="Login" enabled="true">
            <stringProp name="WebDriverSampler.script">var pkg = JavaImporter(org.openqa.selenium)
var support_ui = JavaImporter(org.openqa.selenium.support.ui.WebDriverWait)
var wait = new support_ui.WebDriverWait(WDS.browser, 5000)
var conditions = org.openqa.selenium.support.ui.ExpectedConditions
WDS.sampleResult.sampleStart()
try {
    var username = WDS.browser.findElement(pkg.By.id('username'))
    username.sendKeys('${__P(username,jamesh)}')
    var password = WDS.browser.findElement(pkg.By.id('password'))
    password.sendKeys('${__P(password,j4m3srul3z)}')
    var login = WDS.browser.findElement(pkg.By.id('kc-login'))
    login.click()
    wait.until(conditions.presenceOfElementLocated(pkg.By.id('smartsearch')))
    WDS.sampleResult.sampleEnd()
} catch(ex) {
    WDS.sampleResult.sampleEnd()
    WDS.sampleResult.setSuccessful(false);
    WDS.sampleResult.setResponseMessage("Failed to Login, timeout occured");
}</stringProp>
            <stringProp name="WebDriverSampler.parameters"></stringProp>
            <stringProp name="WebDriverSampler.language">javascript</stringProp>
          </com.googlecode.jmeter.plugins.webdriver.sampler.WebDriverSampler>
          <hashTree>
            <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
              <collectionProp name="Asserion.test_strings">
                <stringProp name="307030181"><title>TIS-ADMINS-UI</title></stringProp>
              </collectionProp>
              <stringProp name="Assertion.custom_message">Failed to Login to TIS</stringProp>
              <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
              <boolProp name="Assertion.assume_success">false</boolProp>
              <intProp name="Assertion.test_type">2</intProp>
            </ResponseAssertion>
            <hashTree/>
          </hashTree>
          <com.googlecode.jmeter.plugins.webdriver.sampler.WebDriverSampler guiclass="com.googlecode.jmeter.plugins.webdriver.sampler.gui.WebDriverSamplerGui" testclass="com.googlecode.jmeter.plugins.webdriver.sampler.WebDriverSampler" testname="Logout" enabled="true">
            <stringProp name="WebDriverSampler.script">var pkg = JavaImporter(org.openqa.selenium)
var support_ui = JavaImporter(org.openqa.selenium.support.ui.WebDriverWait)
var wait = new support_ui.WebDriverWait(WDS.browser, 5000)
var conditions = org.openqa.selenium.support.ui.ExpectedConditions
WDS.sampleResult.sampleStart()
try {
    var logout = WDS.browser.findElement(pkg.By.id('logout'))
    logout.click()
    WDS.sampleResult.sampleEnd()
} catch(ex) {
    WDS.sampleResult.sampleEnd()
    WDS.sampleResult.setSuccessful(false);
    WDS.sampleResult.setResponseMessage("Failed to Login, timeout occured");
}</stringProp>
            <stringProp name="WebDriverSampler.parameters"></stringProp>
            <stringProp name="WebDriverSampler.language">javascript</stringProp>
          </com.googlecode.jmeter.plugins.webdriver.sampler.WebDriverSampler>
          <hashTree>
            <ResponseAssertion guiclass="AssertionGui" testclass="ResponseAssertion" testname="Response Assertion" enabled="true">
              <collectionProp name="Asserion.test_strings">
                <stringProp name="-857427365">TRAINEE INFORMATION SYSTEM</stringProp>
              </collectionProp>
              <stringProp name="Assertion.custom_message">Failed to Login to TIS</stringProp>
              <stringProp name="Assertion.test_field">Assertion.response_data</stringProp>
              <boolProp name="Assertion.assume_success">false</boolProp>
              <intProp name="Assertion.test_type">2</intProp>
            </ResponseAssertion>
            <hashTree/>
          </hashTree>
          <ConstantTimer guiclass="ConstantTimerGui" testclass="ConstantTimer" testname="Delay" enabled="true">
            <stringProp name="ConstantTimer.delay">2000</stringProp>
          </ConstantTimer>
          <hashTree/>
        </hashTree>
        <ResultCollector guiclass="ViewResultsFullVisualizer" testclass="ResultCollector" testname="View Results Tree" enabled="true">
          <boolProp name="ResultCollector.error_logging">false</boolProp>
          <objProp>
            <name>saveConfig</name>
            <value class="SampleSaveConfiguration">
              <time>true</time>
              <latency>true</latency>
              <timestamp>true</timestamp>
              <success>true</success>
              <label>true</label>
              <code>true</code>
              <message>true</message>
              <threadName>true</threadName>
              <dataType>true</dataType>
              <encoding>false</encoding>
              <assertions>true</assertions>
              <subresults>true</subresults>
              <responseData>false</responseData>
              <samplerData>false</samplerData>
              <xml>false</xml>
              <fieldNames>true</fieldNames>
              <responseHeaders>false</responseHeaders>
              <requestHeaders>false</requestHeaders>
              <responseDataOnError>false</responseDataOnError>
              <saveAssertionResultsFailureMessage>true</saveAssertionResultsFailureMessage>
              <assertionsResultsToSave>0</assertionsResultsToSave>
              <bytes>true</bytes>
              <sentBytes>true</sentBytes>
              <threadCounts>true</threadCounts>
              <idleTime>true</idleTime>
              <connectTime>true</connectTime>
            </value>
          </objProp>
          <stringProp name="filename"></stringProp>
        </ResultCollector>
        <hashTree/>
        <ResultCollector guiclass="SummaryReport" testclass="ResultCollector" testname="Summary Report" enabled="true">
          <boolProp name="ResultCollector.error_logging">false</boolProp>
          <objProp>
            <name>saveConfig</name>
            <value class="SampleSaveConfiguration">
              <time>true</time>
              <latency>true</latency>
              <timestamp>true</timestamp>
              <success>true</success>
              <label>true</label>
              <code>true</code>
              <message>true</message>
              <threadName>true</threadName>
              <dataType>true</dataType>
              <encoding>false</encoding>
              <assertions>true</assertions>
              <subresults>true</subresults>
              <responseData>false</responseData>
              <samplerData>false</samplerData>
              <xml>false</xml>
              <fieldNames>true</fieldNames>
              <responseHeaders>false</responseHeaders>
              <requestHeaders>false</requestHeaders>
              <responseDataOnError>false</responseDataOnError>
              <saveAssertionResultsFailureMessage>true</saveAssertionResultsFailureMessage>
              <assertionsResultsToSave>0</assertionsResultsToSave>
              <bytes>true</bytes>
              <sentBytes>true</sentBytes>
              <threadCounts>true</threadCounts>
              <idleTime>true</idleTime>
              <connectTime>true</connectTime>
            </value>
          </objProp>
          <stringProp name="filename"></stringProp>
        </ResultCollector>
        <hashTree/>
        <kg.apc.jmeter.vizualizers.CorrectedResultCollector guiclass="kg.apc.jmeter.vizualizers.ResponseTimesOverTimeGui" testclass="kg.apc.jmeter.vizualizers.CorrectedResultCollector" testname="Response Times Over Time" enabled="true">
          <boolProp name="ResultCollector.error_logging">false</boolProp>
          <objProp>
            <name>saveConfig</name>
            <value class="SampleSaveConfiguration">
              <time>true</time>
              <latency>true</latency>
              <timestamp>true</timestamp>
              <success>true</success>
              <label>true</label>
              <code>true</code>
              <message>true</message>
              <threadName>true</threadName>
              <dataType>true</dataType>
              <encoding>false</encoding>
              <assertions>true</assertions>
              <subresults>true</subresults>
              <responseData>false</responseData>
              <samplerData>false</samplerData>
              <xml>false</xml>
              <fieldNames>true</fieldNames>
              <responseHeaders>false</responseHeaders>
              <requestHeaders>false</requestHeaders>
              <responseDataOnError>false</responseDataOnError>
              <saveAssertionResultsFailureMessage>true</saveAssertionResultsFailureMessage>
              <assertionsResultsToSave>0</assertionsResultsToSave>
              <bytes>true</bytes>
              <sentBytes>true</sentBytes>
              <threadCounts>true</threadCounts>
              <idleTime>true</idleTime>
              <connectTime>true</connectTime>
            </value>
          </objProp>
          <stringProp name="filename"></stringProp>
          <longProp name="interval_grouping">500</longProp>
          <boolProp name="graph_aggregated">false</boolProp>
          <stringProp name="include_sample_labels"></stringProp>
          <stringProp name="exclude_sample_labels"></stringProp>
          <stringProp name="start_offset"></stringProp>
          <stringProp name="end_offset"></stringProp>
          <boolProp name="include_checkbox_state">false</boolProp>
          <boolProp name="exclude_checkbox_state">false</boolProp>
        </kg.apc.jmeter.vizualizers.CorrectedResultCollector>
        <hashTree/>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

Jmeter Command

./bin/jmeter -n -t ./SoakTestPlan.jmx -l TestResult.jtl -Jnum_threads=1 -Jduration=120 -Jmyurl=https://dev-apps.tis.nhs.uk/ -Jusername=jamesh -Jpassword=XXXXXXXX

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213